Support for vLLM sleep and wakeup endpoints in simulation mode

[aavarghese]

What would you like to be added:
Support for vLLM sleep and wakeup (https://docs.vllm.ai/en/latest/features/sleep_mode.html#usage) endpoints. Today is stable and enabled in DEV mode VLLM_SERVER_DEV_MODE=1

Why is this needed:
To run inference servers as part of our solution and put them to sleep when inactive and no incoming requests. Waking-up vllm is much quicker than starting new vllm instances (avoids cold start).

[mayabar]

In order to implement this feature we need to understand:

• whether kv-cache is copied to CPU too or discarded and which kv-events are sent
• which response should be returned for /completions call in sleep mode
• which metrics are returned while in the sleep mode

[irar2]

We are going to implement this feature as following:

• --enable-sleep-mode command line option, together with VLLM_SERVER_DEV_MODE=1 will enable the simulator to enter into sleep mode

• Endpoints:

  • POST /sleep?level=1/2 — Put the model to sleep (level=1 or 2), clean kv-cache (send AllBlocksCleared kv-event) and send true on /is_sleeping requests. All inference requests until wake_up with tags=kv-cache or without parameters, will not enter the kv-cache.
  • POST /wake_up — Wake up the model, optionally with ?tags=weights or ?tags=kv_cache, if only weights - the inference requests will not go to kv-cache
    GET /is_sleeping — Check if the model is sleeping.

No changes in metrics.

@aavarghese @rubambiza Sounds OK?