More flexibility for specifying GAIE plugins #294
Description
Component
Setup/Standup
Desired use case or feature
Users can specify which plugins to use for the endpoint picker by defining LLMDBENCH_VLLM_MODELSERVICE_GAIE_PRESETS. Currently, must be the name of a file in ${LLMDBENCH_MAIN_DIR}/setup/presets/gaie/. At standup, this file is used to define an inferenceExtension.pluginsCustomConfig (see https://github.com/llm-d/llm-d-benchmark/blob/main/setup/steps/08_deploy_gaie.py#L79-L80).
It is challenging to maintain the configurations in ${LLMDBENCH_MAIN_DIR}/setup/presets/gaie/. The upstream GAIE helm chart defines 2 options that can not be used directly since benchmark doesn't have definitions with these names. (default-plugins.yaml and plugins-v2.yaml -- see https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/v0.5.1/config/charts/inferencepool/templates/epp-config.yaml#L57).
Further, the upstream chart allows the user to add additional plugin definitions. To use them with benchmark, code changes are needed.
Proposed solution
Instead of relying on ${LLMDBENCH_MAIN_DIR}/setup/presets/gaie/ the benchmark tool should allow users to use:
- use the defaults in the upstream by reference
- add new (or override existing) plugins directly
- use those provided by benchmarking if desired -- though we should consider deprecating this approach
The current approach guarantees that the plugin will be defined. The proposed approach does not. If the user specifies a plugin that is not defined and provides no definition they will get a runtime error. With effort, this might also be mitigated.
Alternatives
No response
Additional context or screenshots
No response