Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 0 additions & 85 deletions deployments/AWS_EKS/2. Deploy_Dynamo_Cloud.md

This file was deleted.

131 changes: 0 additions & 131 deletions deployments/AWS_EKS/3. Deploy_LLM_Example.md

This file was deleted.

80 changes: 80 additions & 0 deletions deployments/AWS_EKS_vLLM/2. Deploy_Dynamo_Cloud.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Steps to install Dynamo Cloud from Source

## 1. Build Dynamo Base Image

Create 1 ECR repositoriy

```
aws configure
aws ecr create-repository --repository-name <ECR_REPOSITORY>
```

Build Image

```
export NAMESPACE=dynamo-cloud
export DOCKER_SERVER=<ECR_REGISTRY>
export DOCKER_USERNAME=AWS
export DOCKER_PASSWORD="$(aws ecr get-login-password --region <ECR_REGION>)"

export IMAGE_TAG=0.3.2.1
./container/build.sh
```

Push Image

```
docker tag dynamo:latest-vllm <ECR_REGISTRY>/<ECR_REPOSITORY>:$IMAGE_TAG

aws ecr get-login-password | docker login --username AWS --password-stdin <ECR_REGISTRY>/

docker push <ECR_REGISTRY>/<ECR_REPOSITORY>:$IMAGE_TAG
```

## 2. Install Dynamo Cloud

Build and Push Operator Image

```
cd deploy/cloud/operator

vim Earthfile # change ARG IMAGE_SUFFIX=<ECR_REPOSITORY>
earthly --push +docker --DOCKER_SERVER=$DOCKER_SERVER --IMAGE_TAG=$IMAGE_TAG
```

Create secrets

```
kubectl create namespace ${NAMESPACE}
kubectl create secret docker-registry docker-imagepullsecret \
--docker-server=${DOCKER_SERVER} \
--docker-username=${DOCKER_USERNAME} \
--docker-password=${DOCKER_PASSWORD} \
--namespace=${NAMESPACE}

export HF_TOKEN=<HF_TOKEN>
kubectl create secret generic hf-token-secret \
--from-literal=HF_TOKEN=${HF_TOKEN} \
-n ${NAMESPACE}
```

Install Dynamo Cloud

```
cd dynamo/cloud/helm

helm repo add bitnami https://charts.bitnami.com/bitnami
vim deploy.sh # Use the correct image name for dynamo-operator
./deploy.sh --crds
```

Your pods should be running like below

```
ubuntu@ip-192-168-83-157:~/dynamo/components/backends/vllm/deploy$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
dynamo-cloud dynamo-platform-dynamo-operator-controller-manager-86795c5f4j4k 2/2 Running 0 4h17m
dynamo-cloud dynamo-platform-etcd-0 1/1 Running 0 4h17m
dynamo-cloud dynamo-platform-nats-0 2/2 Running 0 4h17m
dynamo-cloud dynamo-platform-nats-box-5dbf45c748-bxqj7 1/1 Running 0 4h17m
```
50 changes: 50 additions & 0 deletions deployments/AWS_EKS_vLLM/3. Deploy_vLLM_Example.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Steps to deploy vLLM example

## 1. Deploy Dynamo Graph

```
cd dynamo/components/backends/vllm/deploy

vim agg_router.yaml # under metadata add namespace: dynamo-cloud and change image to your built base image
kubectl apply -f agg_router.yaml
```

Your pods should be running like below

```
ubuntu@ip-192-168-83-157:~/dynamo/components/backends/vllm/deploy$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
dynamo-cloud dynamo-platform-dynamo-operator-controller-manager-86795c5f4j4k 2/2 Running 0 4h17m
dynamo-cloud dynamo-platform-etcd-0 1/1 Running 0 4h17m
dynamo-cloud dynamo-platform-nats-0 2/2 Running 0 4h17m
dynamo-cloud dynamo-platform-nats-box-5dbf45c748-bxqj7 1/1 Running 0 4h17m
dynamo-cloud vllm-agg-router-frontend-79d599bb9c-fg97p 1/1 Running 0 4m9s
dynamo-cloud vllm-agg-router-vllmdecodeworker-787d575485-hrcjp 1/1 Running 0 4m9s
dynamo-cloud vllm-agg-router-vllmdecodeworker-787d575485-zkwdd 1/1 Running 0 4m9s
```

Test the Deployment

```
kubectl port-forward deployment/vllm-agg-router-frontend 8080:8000 -n dynamo-cloud

curl localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"messages": [
{
"role": "user",
"content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
}
],
"stream": false,
"max_tokens": 30
}'
```

You should output something similar to below

```
{"id":"chatcmpl-bbe52b36-90ed-4479-9872-89e1aa412aa7","choices":[{"index":0,"message":{"content":"<think>\nOkay, so the user wants me to develop a character background for an explorer named someone in Eldoria. The character is part of the","refusal":null,"tool_calls":null,"role":"assistant","function_call":null,"audio":null},"finish_reason":"stop","logprobs":null}],"created":1753417848,"model":"Qwen/Qwen3-0.6B","service_tier":null,"system_fingerprint":null,"object":"chat.completion","usage":{"prompt_tokens":196,"completion_tokens":29,"total_tokens":225,"prompt_tokens_details":null,"completion_tokens_details":null}}
```
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Default LLM example on AWS EKS
# Dynamo vLLM example on AWS EKS

This folder contains steps below to create an AWS EKS cluster with EFS to deploy default Dynamo LLM example tested on commit `d849f7eccabdd850e2c7cb5e6103d6f8b39b0a77`.
This folder contains steps below to create an AWS EKS cluster with EFS to deploy Dynamo vLLM example tested on commit `30942780de2eb6a2358b96caa9f6978c799aede6`.

1. [Create AWS EKS cluster and EFS](1.%20Create_EKS_EFS.md)
2. [Deploy Dynamo Cloud](2.%20Deploy_Dynamo_Cloud.md)
Expand Down