Skip to content

Migrate vLLM Ray Serve Container #5463

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 63 commits into from
Nov 12, 2025
Merged

Migrate vLLM Ray Serve Container #5463

merged 63 commits into from
Nov 12, 2025

Conversation

@junpuf
Copy link
Contributor

@junpuf junpuf commented Nov 7, 2025 •
edited
Loading

a sample PR build and test workflow

@junpuf
try build
8ea932b 
Signed-off-by: Junpu Fan <[email protected]>
@aws-deep-learning-containers-ci aws-deep-learning-containers-ci bot added authorized Size:XL Determines the size of the PR labels Nov 7, 2025
junpuf added 27 commits November 7, 2025 15:06
@junpuf
fix action
50e9793 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
using long commit ref
47e7bf6 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
install/update uv only if not already installed
f3e7416 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
96d976b 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
e21334c 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix actionlint
082f67c 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
try inline cache
a82924d 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix
d82b4a1 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
use buildx
c7d65bc 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
per day cache refresh
09bfc63 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
8a21087 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix
031a0e8 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
test
df2d590 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix
2d59406 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
try artifact
75a8f1a 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update docker command
65975f7 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix command
ff4725e 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix command
3dd1a99 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix entrypoint
872029d 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update test
fadf714 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix command
557e649 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
checkout vllm
58aa567 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update workflow
b071a75 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
e362483 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix
369551b 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
try test
aeebfe8 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix typo
18f2b64 
Signed-off-by: Junpu Fan <[email protected]>
junpuf added 22 commits November 10, 2025 18:19
@junpuf
update
85cffdf 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix
12e2dc1 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update script
ccb5a73 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
enable Entrypoints Integration Test (LLM)
43c2232 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
15f0c89 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
e9fa11c 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update test
60fd04f 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
add cleanup
56d85c1 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
fix
432917d 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
e5ad9e6 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
8e7a408 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
f5e61e3 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
16d5f1e 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update workflow
c0a8c85 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
enable more test
af227b4 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update tests
6ba7e45 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
parallel tests
c3cc99c 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
remove encoder decoder test
dcb9302 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
add hf token
c7b284b 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
update
9807927 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
remove push on main
11ead3b 
Signed-off-by: Junpu Fan <[email protected]>
@junpuf
revert
92f77d9 
Signed-off-by: Junpu Fan <[email protected]>
sirutBuasai
sirutBuasai reviewed Nov 11, 2025
View reviewed changes
Copy link
Member

@sirutBuasai sirutBuasai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll most likely need cuda compat script as well. not sure how vllm has been working thus far. https://github.com/aws/deep-learning-containers/blob/eb524f7c0737b007cf06d4fd36f67de246cc8d8f/sglang/build_artifacts/start_cuda_compat.sh

@junpuf
Copy link
Contributor Author

junpuf commented Nov 11, 2025

We'll most likely need cuda compat script as well. not sure how vllm has been working thus far. https://github.com/aws/deep-learning-containers/blob/eb524f7c0737b007cf06d4fd36f67de246cc8d8f/sglang/build_artifacts/start_cuda_compat.sh

Not running any SageMaker test yet

sirutBuasai
sirutBuasai reviewed Nov 11, 2025
View reviewed changes
.github/workflows/pr-vllm-rayserve.yml
Comment on lines +94 to +114
- name: Download image URI artifact
uses: actions/download-artifact@v4
with:
name: vllm-rayserve-ec2-image-uri

- name: Resolve image URI for test
run: |
IMAGE_URI=$(cat image_uri.txt)
echo "Resolved image URI: $IMAGE_URI"
echo "IMAGE_URI=$IMAGE_URI" >> $GITHUB_ENV

- name: Pull image
run: |
docker pull $IMAGE_URI

- name: Checkout vLLM Tests
uses: actions/checkout@v5
with:
repository: vllm-project/vllm
ref: v0.10.2
path: vllm_source
Copy link
Member

@sirutBuasai sirutBuasai Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there's a way to DRY these steps. These are going to be used repeatedly across multiple stages

Copy link
Contributor Author

@junpuf junpuf Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

later we can refactor common patterns into callable workflows or other things

sirutBuasai
sirutBuasai reviewed Nov 11, 2025
View reviewed changes
scripts/telemetry/deep_learning_container.py
parser = argparse.ArgumentParser()
parser.add_argument(
"--framework",
choices=["tensorflow", "mxnet", "pytorch", "base", "vllm"],
Copy link
Member

@sirutBuasai sirutBuasai Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably fix this list to ["tensorflow", "pytorch", "base", "vllm", "sglang"]

Copy link
Contributor Author

@junpuf junpuf Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually this telemetry things doesn't work, cuz the template replacement stuff doesn't exist. will need to fix it separately

sirutBuasai
sirutBuasai reviewed Nov 11, 2025
View reviewed changes
scripts/telemetry/deep_learning_container.py
)
parser.add_argument(
"--container-type",
choices=["training", "inference", "general"],
Copy link
Member

@sirutBuasai sirutBuasai Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also side note and unrelated to this PR. Currently vllm and sglang are classified as general. We should change these to inference.

Copy link
Contributor Author

@junpuf junpuf Nov 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the overall telemetry integration needs to be fixed.

@junpuf junpuf merged commit 57003ef into main Nov 12, 2025
9 checks passed
@junpuf junpuf deleted the try-build branch November 12, 2025 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sirutBuasai sirutBuasai sirutBuasai approved these changes

Assignees

No one assigned

Labels

authorized Size:XL Determines the size of the PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@junpuf @sirutBuasai