Initial drop of the activator code #387

lionelvillard · 2025-10-24T14:59:31Z

This PR introduces a new component, called the activator, scaling from zero and to zero Kubernetes object attached to InferencePool.

Overview

The code consists of:

charts/activator-filter: a helm chart for inserting the activator into the gateway HTTP filter chain (Istio only). Needs to be applied only once.
charts/activator: a helm chart configuring the activator for a given HTTPRoute and InferencePool
cmd/activator: the activator main
pkg/activator:
- a ext-proc server handling header requests and ensuring there is at least one replica in the InferencePool.
- a monitor scaling down to zero the object attached to the InferencePool after a configurable idleness period.

Control-knobs

Here are the control-knobs attached to InferencePool objects as annotations.

activator.llm-d.ai/target-apiversion: APIVersion of the target object to scale
activator.llm-d.ai/target-kind: Kind of the target object to scale
activator.llm-d.ai/target-name: Name of the target object to scale
activator.llm-d.ai/scale-from-zero-grace-period: time (in seconds) to wait for scale-from-zero decision to complete before aborting.
activator.llm-d.ai/scale-down-delay: the minimum of time (in seconds) before the system decides to scale down the last replica to zero

Here are the control-knobs specified on the command line:

--enable-scale-to-zero: Enable scaling down InferencePool to zero replicas after a period of idleness (Default: false)

Testing

e2e testing is done manually following these recipes.
- recipe 1: multiple models using url-rewriting method
- recipe 2: multiple models using BBR method
@dumb0002 recipes

Future Work

better documentation other than helm charts
load testing
more control-knobs

Co-authors

@dumb0002

@dumb0002 @nilig @elevran @kfswain @shmuelk

Makefile

charts/activator-filter/templates/rbac.yaml

charts/activator/README.md

pkg/activator/handlers/server.go

pkg/activator/server/controller_manager.go

pkg/activator/requestcontrol/activator.go

lionelvillard · 2025-11-03T16:46:29Z

@shmuelk @pierDipi All comments have been addressed.

I would like to rebase this PR after #416 is merged, as currently it's not possible to build the image when using make.

lionelvillard · 2025-11-03T18:16:12Z

rebase done @shmuelk

Dockerfile.activator

cmd/activator/main.go

cmd/activator/runner/health.go

cmd/activator/runner/runner.go

pkg/activator/server/runserver.go

pkg/activator/requestcontrol/deactivator.go

pkg/activator/datastore/datastore.go

nirrozenbaum · 2025-11-05T17:51:32Z

charts/activator-filter/templates/istio.yaml

+          filter:
+            name: "envoy.filters.network.http_connection_manager"
+    patch:
+      operation: INSERT_FIRST # TODO: insert before EPP


OOC: what will happen if this is deployed together with BBR - which ext proc will be invoked first?

it depends on the order of which filters are applied. It really does not matter which one run first, maybe more optimal to have BBR before the activator as the BBR can reject invalid requests.

exactly. you are touching exactly the point I was trying to highlight.
It might be worth adding this information so some section in the deployment instructions.
we should recommend in which order to deploy them (and I agree as you mentioned that BBR should run first).

I added a note in the readme.

charts/activator-filter/README.md

nirrozenbaum · 2025-11-05T17:57:07Z

charts/activator-filter/README.md

+
+## Install
+
+To install an activator-filter named `activator-filter`, you can run the following command:


whoever is going to use activator isn't necessarily familiar with what is filter. we can call it just activator.

there is already a chart named activator. What about activator-istio?

is there a use case for deploying one chart without the other?

both are needed. The only difference is one is applied once per namespace, the other one once per HTTPRoute.

charts/activator-filter/templates/rbac.yaml

charts/activator-filter/templates/istio.yaml

elevran · 2025-11-06T18:34:08Z

/approve
will let @shmuelk or @nirrozenbaum add the LGTM

elevran · 2025-11-06T20:27:29Z

/hold
@lionelvillard please sign commits - missing verified signature

lionelvillard · 2025-11-06T21:00:54Z

@elevran done.

elevran · 2025-11-07T07:44:19Z

/unhold

shmuelk · 2025-10-29T14:43:35Z

Dockerfile.activator

+ENV GOARCH=$TARGETARCH
+
+# Dependencies
+WORKDIR /src


Change this line to:
WORKDIR /workspace

shmuelk · 2025-10-29T14:43:40Z

Dockerfile.activator

+# Sources
+COPY cmd/activator ./cmd/activator
+COPY pkg/activator ./pkg/activator
+WORKDIR /src/cmd/activator


Remove this line

shmuelk · 2025-10-29T14:43:43Z

Dockerfile.activator

+COPY cmd/activator ./cmd/activator
+COPY pkg/activator ./pkg/activator
+WORKDIR /src/cmd/activator
+RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/version.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/version.BuildRef=${BUILD_REF}" -o /activator


Please make this Dockerfile more like the others in this repo:

Change: -o /activator to -o bin/activator cmd/activator/main.go

shmuelk · 2025-10-29T14:43:47Z

Dockerfile.activator

+FROM registry.access.redhat.com/ubi9/ubi-minimal:latest
+
+WORKDIR /
+COPY --from=builder /activator /activator


Change line to:

`COPY --from=builder /workspace/bin/activator /app/activator

shmuelk · 2025-10-29T14:43:54Z

Dockerfile.activator

+WORKDIR /
+COPY --from=builder /activator /activator
+
+ENTRYPOINT ["/activator"]


Change this line to:

ENTRYPOINT ["/app/activator"]

shmuelk · 2025-10-30T16:28:52Z

cmd/activator/runner/runner.go

+	// --- Setup Metrics Server ---
+	metricsServerOptions := metricsserver.Options{
+		BindAddress:    fmt.Sprintf(":%d", *metricsPort),
+		FilterProvider: filters.WithAuthenticationAndAuthorization,


See my above comment related to security on the README.md file

pkg/activator/server/controller_manager.go

Dockerfile.activator

Dockerfile.epp

pkg/activator/handlers/server.go

lionelvillard · 2025-11-10T15:08:38Z

@shmuelk I resolved your recent comments

Co-authored-by: Braulio Dumba <[email protected]> Signed-off-by: Lionel Villard <[email protected]>

Signed-off-by: Braulio Dumba <[email protected]>

Co-authored-by: Pierangelo Di Pilato <[email protected]> Signed-off-by: Lionel Villard <[email protected]>

Signed-off-by: Lionel Villard <[email protected]>

Co-authored-by: Shmuel Kallner <[email protected]> Signed-off-by: Lionel Villard <[email protected]>

Signed-off-by: Lionel Villard <[email protected]>

github-project-automation bot added this to llm-d-inference-scheduler Oct 24, 2025

lionelvillard force-pushed the activator branch 4 times, most recently from 138e9a4 to d929811 Compare October 24, 2025 15:47

shmuelk requested changes Oct 29, 2025

View reviewed changes

Makefile Show resolved Hide resolved

Makefile Show resolved Hide resolved

charts/activator-filter/templates/rbac.yaml Show resolved Hide resolved

charts/activator/README.md Outdated Show resolved Hide resolved

github-project-automation bot moved this to In review in llm-d-inference-scheduler Oct 29, 2025

pierDipi reviewed Nov 3, 2025

View reviewed changes

pkg/activator/handlers/server.go Show resolved Hide resolved

pkg/activator/server/controller_manager.go Outdated Show resolved Hide resolved

pkg/activator/requestcontrol/activator.go Outdated Show resolved Hide resolved

lionelvillard force-pushed the activator branch from c8e531c to 227547b Compare November 3, 2025 15:04

nilig assigned shmuelk Nov 3, 2025

lionelvillard force-pushed the activator branch from 1bf386c to 6bf3240 Compare November 3, 2025 16:37

lionelvillard force-pushed the activator branch from 6bf3240 to fd9c10d Compare November 3, 2025 18:15

lionelvillard force-pushed the activator branch from fd9c10d to 6108eae Compare November 3, 2025 19:15

elevran requested changes Nov 4, 2025

View reviewed changes

elevran requested a review from shmuelk November 4, 2025 15:43

elevran mentioned this pull request Nov 4, 2025

Activator clean up items #425

Open

lionelvillard force-pushed the activator branch 2 times, most recently from d57412b to ed0f30b Compare November 4, 2025 19:34

elevran self-requested a review November 5, 2025 14:30

nirrozenbaum reviewed Nov 5, 2025

View reviewed changes

charts/activator-filter/README.md Show resolved Hide resolved

nirrozenbaum reviewed Nov 5, 2025

View reviewed changes

charts/activator-filter/templates/rbac.yaml Show resolved Hide resolved

lionelvillard force-pushed the activator branch from 479ea3f to 125d7b1 Compare November 5, 2025 19:37

elevran reviewed Nov 6, 2025

View reviewed changes

charts/activator-filter/templates/rbac.yaml Show resolved Hide resolved

charts/activator-filter/templates/istio.yaml Show resolved Hide resolved

elevran self-requested a review November 6, 2025 07:13

lionelvillard force-pushed the activator branch from 20714ba to 773ceec Compare November 6, 2025 16:11

elevran requested a review from nirrozenbaum November 6, 2025 18:34

elevran previously approved these changes Nov 6, 2025

View reviewed changes

github-actions bot added the hold label Nov 6, 2025

lionelvillard force-pushed the activator branch 2 times, most recently from da1abca to 2540e93 Compare November 6, 2025 21:00

elevran removed the hold label Nov 7, 2025

dumb0002 mentioned this pull request Nov 7, 2025

Activator E2E Tests (Initial drop) #438

Open

shmuelk requested changes Nov 9, 2025

View reviewed changes

lionelvillard dismissed stale reviews from github-actions[bot] and elevran via 4900a46 November 10, 2025 14:40

lionelvillard force-pushed the activator branch 2 times, most recently from cbdad4b to 5fc9749 Compare November 10, 2025 15:03

lionelvillard and others added 13 commits November 13, 2025 08:57

initial drop of the activator code

641dd29

Co-authored-by: Braulio Dumba <[email protected]> Signed-off-by: Lionel Villard <[email protected]>

Adding for InferencePool v1

818c096

Signed-off-by: Braulio Dumba <[email protected]>

Adding parameter to enable/disable scale to zero

25cd974

Signed-off-by: Braulio Dumba <[email protected]>

Update pkg/activator/server/controller_manager.go

16c4bdd

Co-authored-by: Pierangelo Di Pilato <[email protected]> Signed-off-by: Lionel Villard <[email protected]>

address some review comments

0b278ad

Signed-off-by: Lionel Villard <[email protected]>

address review comments

9712865

Signed-off-by: Lionel Villard <[email protected]>

fix linter error

58735ff

Signed-off-by: Lionel Villard <[email protected]>

activator-filter should be applied before BBR

a03a272

Signed-off-by: Lionel Villard <[email protected]>

remove unused files and reorder some imports

f4b2b62

Signed-off-by: Lionel Villard <[email protected]>

Apply suggestion from @shmuelk

6c5fb88

Co-authored-by: Shmuel Kallner <[email protected]> Signed-off-by: Lionel Villard <[email protected]>

remove inferenceOjective from cache

9418a41

Signed-off-by: Lionel Villard <[email protected]>

revert to use go 1.24.1

54143fd

Signed-off-by: Lionel Villard <[email protected]>

use golang 1.24 as 1.24.1 does not exists

47bbf69

Signed-off-by: Lionel Villard <[email protected]>

lionelvillard force-pushed the activator branch from 5fc9749 to 47bbf69 Compare November 13, 2025 14:00


		## Install

		To install an activator-filter named `activator-filter`, you can run the following command:

Initial drop of the activator code #387

Are you sure you want to change the base?

Initial drop of the activator code #387

Uh oh!

Conversation

lionelvillard commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Control-knobs

Testing

Future Work

Co-authors

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lionelvillard commented Nov 3, 2025

Uh oh!

lionelvillard commented Nov 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elevran commented Nov 6, 2025

Uh oh!

elevran commented Nov 6, 2025

Uh oh!

lionelvillard commented Nov 6, 2025

Uh oh!

elevran commented Nov 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lionelvillard commented Nov 10, 2025

Uh oh!

lionelvillard commented Oct 24, 2025 •

edited

Loading