Synchronous Tokenization

[vMaroon]

In the current design, tokenization is never done synchronously. This was driven by wanting to avoid duplicated tokenization on the active serving path, but recently this is seeming like less of a problem.

Handling this issue would mean investigating a change in which tokenization is done synchronously as part of the active path, but still cached. The handling should ideally remain by the workers pool.

Profiling should be ideally provided.

[sagiahrac]

Assign me

[sagiahrac]

I think tokenization could be simplified by not using the pool.
kvcache.GetPodScores can call the tokenizer directly, waiting only if the prefix cache misses.

Tokenizers are shared and cached across requests, but this is safe since the HuggingFace bindings use immutable references (see tokenizers lib.rs), so no internal state is modified. The only locking needed is in the prefix store, which can be handled with a sync map.

The scorer is already shared between requests in the scheduler without explicit locks; it assumes thread safety is handled by the prefix store, which remains preserved.

If we later want to scale tokenization, we can:

1. In-process parallelism: split long prompts into chunks for concurrent encoding.
2. Disaggregate: run tokenization as a sidecar/daemon with a TokenizerConnector.

This avoids the overhead of an external worker thread and simplifies debugging.

[vMaroon]

Disaggregated tokenization is a future-effort, but I think we'll want to support embedded tokenization anyway. Especially since we already have it. Therefore, if we drop the pool and run one instance of a tokenizer, as you said, requests served in parallel will share the same tokenizer - which can block throughput due to contention, regardless of if it is thread-safe.

The latter was initially intended with the pool, but not fully implemented.