Skip to content

refactor: Make base client concrete and usable #246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+1 −9
Open

refactor: Make base client concrete and usable #246

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion inference_perf/client/modelserver/openai_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ async def process_request(self, data: InferenceAPIData, stage_id: int, scheduled
)

def get_supported_apis(self) -> List[APIType]:
return []
return [APIType.Completion, APIType.Chat]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a type openai here? This could be the generic type that someone can run with if it is one of the model servers where we don't have specific metrics support?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update the documentation as well along with the change.


@abstractmethod
def get_prometheus_metric_metadata(self) -> PrometheusMetricMetadata:
Expand Down
2 changes: 0 additions & 2 deletions inference_perf/client/modelserver/sglang_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,6 @@ def __init__(
)
self.metric_filters = [f"model_name='{model_name}'", *additional_filters]

def get_supported_apis(self) -> List[APIType]:
return [APIType.Completion, APIType.Chat]

def get_prometheus_metric_metadata(self) -> PrometheusMetricMetadata:
return PrometheusMetricMetadata(
Expand Down
3 changes: 0 additions & 3 deletions inference_perf/client/modelserver/tgi_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,6 @@ def __init__(
)
self.metric_filters = additional_filters

def get_supported_apis(self) -> List[APIType]:
return [APIType.Completion, APIType.Chat]

def get_prometheus_metric_metadata(self) -> PrometheusMetricMetadata:
return PrometheusMetricMetadata(
avg_queue_length=ModelServerPrometheusMetric(
Expand Down
3 changes: 0 additions & 3 deletions inference_perf/client/modelserver/vllm_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,6 @@ def __init__(
)
self.metric_filters = [f"model_name='{model_name}'", *additional_filters]

def get_supported_apis(self) -> List[APIType]:
return [APIType.Completion, APIType.Chat]

def get_prometheus_metric_metadata(self) -> PrometheusMetricMetadata:
return PrometheusMetricMetadata(
avg_queue_length=ModelServerPrometheusMetric(
Expand Down
Loading