LLM Router & Dynamo Integration #16

arunraman · 2025-06-17T19:05:34Z

No description provided.

- Introduced `dynamo-llm-config.yaml` for full production deployment with multiple models. - Added `dynamo-single-llm-config.yaml` for minimal testing with a single model. - Created `llm-router-values-override.yaml` for Helm deployment customization. - Added routing configuration files: `router-config.yaml` for full deployment and `router-config-single.yaml` for single model verification. - Updated `README.md` with comprehensive deployment instructions and configuration options.

- Added license information to `dynamo-llm-config.yaml`, `dynamo-single-llm-config.yaml`, `llm-router-values-override.yaml`, `router-config-single.yaml`, and `router-config.yaml`. - Ensured compliance with Apache License 2.0 for all relevant configuration files.

- Added `deploy-dynamo-integration.sh` script for automated deployment of NVIDIA Dynamo Cloud Platform and LLM Router. - Introduced `dynamo-cloud-deployment.yaml` for configuring the Dynamo deployment. - Created `dynamo-llm-deployment.yaml` for deploying LLM inference graphs. - Added `router-config-dynamo.yaml` for routing configuration specific to Dynamo integration. - Removed outdated configuration files: `dynamo-single-llm-config.yaml` and `router-config-single.yaml`. - Updated `README.md` to reflect new deployment instructions and configuration details.

- Updated `llm-router-values-override.yaml` to enable ingress with NGINX configuration. - Added detailed ingress setup instructions to `README.md`, including host configuration and testing methods. - Provided examples for testing routing via ingress and port-forwarding.

…ntegration - Deleted `deploy-dynamo-integration.sh` and `dynamo-cloud-deployment.yaml` as they are no longer needed. - Updated `README.md` to reflect the removal of these files and adjusted deployment instructions accordingly.

Copilot

Pull Request Overview

This PR adds NVIDIA Dynamo Cloud Platform integration by introducing new router configurations, Helm value overrides, and a sample DynamoGraphDeployment.

Defines LLM routing policies for task and complexity routers pointing to Dynamo endpoints.
Provides Helm values to configure router controller/server, ingress, RBAC, and resource settings.
Includes an example DynamoGraphDeployment and corresponding Service manifest for LLM inference.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
customizations/LLM Router/router-config-dynamo.yaml	New YAML defining `task_router` and `complexity_router` policies for Dynamo endpoints.
customizations/LLM Router/llm-router-values-override.yaml	Helm values override configuring images, ingress, security, and RBAC for Dynamo integration.
customizations/LLM Router/dynamo-llm-deployment.yaml	Sample `DynamoGraphDeployment` and Service manifest for deploying LLMs on Dynamo.

Comments suppressed due to low confidence (3)

customizations/LLM Router/llm-router-values-override.yaml:25

[nitpick] Pin specific image tags instead of using latest to ensure reproducible and auditable deployments.

    tag: latest

customizations/LLM Router/llm-router-values-override.yaml:56

[nitpick] Pin specific image tags instead of using latest to ensure reproducible and auditable deployments.

    tag: latest

customizations/LLM Router/dynamo-llm-deployment.yaml:147

The selector value has a trailing space (Frontend ), which may prevent it from matching the Pod label; remove the extra space.

    dynamo-component: Frontend

Copilot · 2025-06-17T19:07:53Z

customizations/LLM Router/llm-router-values-override.yaml

+  className: "nginx"  # Use your cluster's ingress class
+  annotations:
+    nginx.ingress.kubernetes.io/rewrite-target: /
+    nginx.ingress.kubernetes.io/ssl-redirect: "false"


Disabling SSL redirect exposes the service over plain HTTP; consider enabling TLS and redirecting HTTPS to protect data in transit.

Suggested change

nginx.ingress.kubernetes.io/ssl-redirect: "false"

nginx.ingress.kubernetes.io/ssl-redirect: "true"

Copilot · 2025-06-17T19:07:54Z

customizations/LLM Router/router-config-dynamo.yaml

+    llms:
+      - name: Brainstorming
+        api_base: http://dynamo-llm-service.dynamo-cloud.svc.cluster.local:8080/v1
+        api_key: ""


Storing an empty api_key means there’s no authentication; use Kubernetes Secrets or environment variables to securely inject real credentials.

- Updated `README.md` to reflect the new directory name and provide an overview of NVIDIA Dynamo Customizations. - Added detailed sections on available customizations, including LLM Router integration and its benefits. - Modified `dynamo-llm-deployment.yaml` to clarify component reference instructions. - Enhanced `llm-router-values-override.yaml` with additional configuration options for API base and namespace. - Updated `router-config-dynamo.yaml` to utilize environment variables for API key management and service endpoints. - Expanded `README.md` in the LLM Router directory with comprehensive deployment instructions and configuration validation steps.

mohammedabdulwahhab

Left some comments about changes that need to be made given that Dynamo Deploy now has a vastly simplified UX. PIng me for any questions

mohammedabdulwahhab · 2025-07-09T19:41:01Z

customizations/LLM Router/README.md

+
+- **Kubernetes cluster** (1.24+) with kubectl configured
+- **Helm 3.x** for managing deployments
+- **Earthly** for building Dynamo components ([Install Guide](https://earthly.dev/get-earthly))


don't need earthly anymore with pre-published images

Again Quickstart mentions about earthly here

mohammedabdulwahhab · 2025-07-09T19:41:33Z

customizations/LLM Router/README.md

+
+You'll need to configure these environment variables before deployment:
+
+| Variable | Description | Example |


a lot of these env vars are not needed to deploy Dyn Cloud platform anymore

mohammedabdulwahhab · 2025-07-09T19:48:03Z

customizations/LLM Router/README.md

+docker compose -f deploy/metrics/docker-compose.yml up -d
+```
+
+### Step 2: Deploy Dynamo Cloud Platform (For Kubernetes)


lines 258-309 need refactoring based on the new simplified workflow captured in our new quickstart here

dynamo build and deploy commands are now gone and the new workflow should be a lot more concise.

The quickstart mentions to build images using earthly here

mohammedabdulwahhab · 2025-07-09T19:49:48Z

customizations/LLM Router/dynamo-llm-deployment.yaml

+# Based on: https://docs.nvidia.com/dynamo/latest/guides/dynamo_deploy/dynamo_operator.html
+
+---
+apiVersion: nvidia.com/v1alpha1


This CR needs to be updated. dynamoComponent and related fields are no longer supported as part of the new workflow. Each component needs to bring its own image. Please use the following CRs as reference.

The CR link seems to broken. Can you give me the correct link ?

- Added `disagg.yaml` for deploying the vLLM backend with disaggregated serving capabilities. - Removed the deprecated `dynamo-llm-deployment.yaml` file to streamline the configuration. - Updated `README.md` to include detailed instructions for the new disaggregated deployment setup. - Revised `router-config-dynamo.yaml` to reflect the models currently deployed and their configurations. - Enhanced documentation to clarify the integration of multiple models and their routing strategies.

- Revised sections for NVIDIA Dynamo and LLM Router to improve clarity and detail. - Updated descriptions to reflect disaggregated serving capabilities and multi-model deployment. - Enhanced task classification and complexity analysis descriptions for better understanding. - Improved overall documentation for optimal performance insights.

- Streamlined descriptions for NVIDIA Dynamo and LLM Router to enhance clarity. - Consolidated features into concise bullet points for better readability. - Updated task classification and complexity analysis sections for improved understanding. - Removed unnecessary table formatting to simplify the layout.

…Router documentation

… 2.0

… LLM Router configurations

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

- Introduced `agg.yaml` for deploying vLLM with a single GPU setup. - Updated `disagg.yaml` to use environment variables for model names and unified image references. - Enhanced `README.md` to include new environment variables and deployment instructions for both aggregated and disaggregated configurations.

- Removed frontend service definitions from `agg.yaml` and `disagg.yaml`, consolidating them into a new `frontend.yaml` for a shared API service. - Updated `llm-router-values-override.yaml` to adjust API base and repository configurations for improved deployment flexibility. - Enhanced `README.md` to reflect the new shared frontend architecture, detailing its benefits and deployment instructions. - Revised routing strategies and model configurations to streamline multi-model deployments and reduce resource overhead.

…n changes - Revised README.md to reflect updated worker templates and multi-model support, enhancing clarity on deployment configurations. - Changed model reference in router-config-dynamo.yaml from `meta-llama/Llama-3.1-70B-Instruct` to `mistralai/Mixtral-8x22B-Instruct-v0.1`, aligning with the new deployment strategy. - Improved health check configuration details in README.md to ensure proper service-level setup.

…M Router deployment - Revised environment variable section to include additional variables such as `DYNAMO_IMAGE`, `DYNAMO_API_BASE`, and `DYNAMO_API_KEY`, clarifying their usage in deployment. - Updated model recommendations to reflect current configurations and performance insights. - Improved instructions for setting up environment variables, ensuring clarity for users during deployment.

…loyment instructions - Added image pull secret configuration in llm-router-values-override.yaml to support private registry access. - Revised README.md to enhance clarity on environment variable usage and deployment steps, including updates to model deployment instructions and Kubernetes secret references. - Streamlined instructions for creating ConfigMaps and verifying deployments to ensure a smoother user experience.

arunraman added 5 commits June 15, 2025 20:28

arunraman requested a review from Copilot June 17, 2025 19:06

arunraman assigned nealvaidya Jun 17, 2025

Copilot AI reviewed Jun 17, 2025

View reviewed changes

arunraman changed the title ~~Arraman dev~~ LLM Router & Dynamo Integration Jun 17, 2025

mohammedabdulwahhab requested changes Jul 9, 2025

View reviewed changes

arunraman added 4 commits August 23, 2025 04:52

Update README.md to change badge logo for API Keys setup step in LLM …

190a334

…Router documentation

arunraman requested review from Copilot and mohammedabdulwahhab August 23, 2025 05:42

arunraman added 2 commits August 23, 2025 05:45

Add license headers to disagg.yaml for compliance with Apache License…

83c1f2d

… 2.0

Update README.md to include new routing strategy and model fields for…

27bb807

… LLM Router configurations

Copilot AI reviewed Aug 23, 2025

View reviewed changes

arunraman added 5 commits August 23, 2025 07:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

LLM Router & Dynamo Integration #16

LLM Router & Dynamo Integration #16

Uh oh!

arunraman commented Jun 17, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jun 17, 2025

Uh oh!

Copilot AI Jun 17, 2025

Uh oh!

mohammedabdulwahhab left a comment

Uh oh!

mohammedabdulwahhab Jul 9, 2025

Uh oh!

arunraman Aug 8, 2025

Uh oh!

mohammedabdulwahhab Jul 9, 2025

Uh oh!

mohammedabdulwahhab Jul 9, 2025

Uh oh!

arunraman Aug 8, 2025

Uh oh!

mohammedabdulwahhab Jul 9, 2025

Uh oh!

arunraman Aug 8, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

	nginx.ingress.kubernetes.io/ssl-redirect: "false"
	nginx.ingress.kubernetes.io/ssl-redirect: "true"


		You'll need to configure these environment variables before deployment:

		\| Variable \| Description \| Example \|

LLM Router & Dynamo Integration #16

Are you sure you want to change the base?

LLM Router & Dynamo Integration #16

Uh oh!

Conversation

arunraman commented Jun 17, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

mohammedabdulwahhab left a comment

Choose a reason for hiding this comment

Uh oh!

mohammedabdulwahhab Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

arunraman Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

mohammedabdulwahhab Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

mohammedabdulwahhab Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

arunraman Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

mohammedabdulwahhab Jul 9, 2025

Choose a reason for hiding this comment

Uh oh!

arunraman Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!