[VLM] Offline scenario, performance-only mode of the reference implementation

[wangshangsam]

This is the first PR towards the VLM reference implementation for the v6.0 round.
This PR currenlty supports the Offline scenario + performance-only mode. Server scenario and accuracy mode will be introduced through subsequent PRs.
The issue_query implemenation adopted the purely asyncio-based design from the DSR1 reference implementation, but the code here is simpler mostly because we only access the inference endpoint through OpenAI APIs.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[johncalesp]

Can we change this to float? Currently is timedelta. If I try to enter
--settings.min_duration 60 I get:

│ Invalid value for _pydantic_settings_min_duration: Input should be a valid timedelta, "day" identifier in duration not correctly formatted                                                                                                                                                                                                                                                 │```

[wangshangsam]

Hmmm this is not supposed to happen. The string format that this flag can take is defined in https://docs.pydantic.dev/2.0/usage/types/datetime/

timedelta fields will accept values of type:

• str; the following formats are accepted:
  [-][DD ][HH:MM]SS[.ffffff]
  [±]P[DD]DT[HH]H[MM]M[SS]S (ISO 8601 format for timedelta)

[johncalesp]

I see, so the input will be something like: --settings.min_duration 00:01:00

[wangshangsam]

00:01:00 can work but what I was trying to say is that it's supposed to be able to take in an single integer as well (in which case it would mean "seconds")

[johncalesp]

when, I entered 60 , it threw me the error

[johncalesp]

Here, when we use settings.use_token_latencies = True, the constrains become:

  /// Token latency parameters
  uint64_t server_ttft_latency = 100000000;
  uint64_t server_tpot_latency = 100000000;

If we use false the constrain is:

/// \brief The latency constraint for the Server scenario.
  uint64_t server_target_latency_ns = 100000000;

We may need to add more flags to let the user choose the constrains and values