Why Does Inference Speed Collapse After a Few Rounds of Conversation? Tried Cache-Reuse and Keep — Any Fixes?

[wang7x]

I’m running the llama.cpp service using the following command:

ramalama serve gpt-oss -c 64000 --ngl 99 --cache-reuse 256 --runtime-args="--keep -1 -ctk q8_0 -ctv q8_0 --jinja"

Model: gpt-oss pulled from hf://ggml-org/gpt-oss-20b-GGUF

I interact with the model via the web interface launched by the service (http://0.0.0.0:8080). As the conversation context grows over multiple rounds, I notice a significant drop in inference throughput — from ~80 tokens/sec initially to around 20–40 tokens/sec after a few exchanges.

I’ve experimented with various parameters including --cache-reuse, --keep, and --swa-checkpoint, but none of these adjustments seem to improve performance.

Is this slowdown expected behavior with long context windows? Or is there something I might be missing in terms of configuration or optimization?

The service logs: cache_reuse is not supported by this context, it will be disabled. Does this mean the model architecture or runtime configuration doesn’t support KV reuse in this setup?

Full log:

❯ ramalama serve gpt-oss -c 64000 --ngl 99 --cache-reuse 256 --runtime-args="--keep -1 -ctk q8_0 -ctv q8_0 --jinja"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4070 Ti SUPER, compute capability 8.9, VMM: yes
build: 1 (b0d5299) with cc (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) for x86_64-redhat-linux
system info: n_threads = 16, n_threads_batch = 16, total_threads = 32

system_info: n_threads = 16 (n_threads_batch = 16) / 32 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

main: binding port with default address family
main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 31
main: loading model
srv    load_model: loading model '/mnt/models/gpt-oss-20b-mxfp4.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4070 Ti SUPER) - 13265 MiB free
llama_model_loader: loaded meta data with 35 key-value pairs and 459 tensors from /mnt/models/gpt-oss-20b-mxfp4.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gpt-oss
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Gpt Oss 20b
llama_model_loader: - kv   3:                           general.basename str              = gpt-oss
llama_model_loader: - kv   4:                         general.size_label str              = 20B
llama_model_loader: - kv   5:                            general.license str              = apache-2.0
llama_model_loader: - kv   6:                               general.tags arr[str,2]       = ["vllm", "text-generation"]
llama_model_loader: - kv   7:                        gpt-oss.block_count u32              = 24
llama_model_loader: - kv   8:                     gpt-oss.context_length u32              = 131072
llama_model_loader: - kv   9:                   gpt-oss.embedding_length u32              = 2880
llama_model_loader: - kv  10:                gpt-oss.feed_forward_length u32              = 2880
llama_model_loader: - kv  11:               gpt-oss.attention.head_count u32              = 64
llama_model_loader: - kv  12:            gpt-oss.attention.head_count_kv u32              = 8
llama_model_loader: - kv  13:                     gpt-oss.rope.freq_base f32              = 150000.000000
llama_model_loader: - kv  14:   gpt-oss.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  15:                       gpt-oss.expert_count u32              = 32
llama_model_loader: - kv  16:                  gpt-oss.expert_used_count u32              = 4
llama_model_loader: - kv  17:               gpt-oss.attention.key_length u32              = 64
llama_model_loader: - kv  18:             gpt-oss.attention.value_length u32              = 64
llama_model_loader: - kv  19:           gpt-oss.attention.sliding_window u32              = 128
llama_model_loader: - kv  20:         gpt-oss.expert_feed_forward_length u32              = 2880
llama_model_loader: - kv  21:                  gpt-oss.rope.scaling.type str              = yarn
llama_model_loader: - kv  22:                gpt-oss.rope.scaling.factor f32              = 32.000000
llama_model_loader: - kv  23: gpt-oss.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  24:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  25:                         tokenizer.ggml.pre str              = gpt-4o
llama_model_loader: - kv  26:                      tokenizer.ggml.tokens arr[str,201088]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,201088]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  28:                      tokenizer.ggml.merges arr[str,446189]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  29:                tokenizer.ggml.bos_token_id u32              = 199998
llama_model_loader: - kv  30:                tokenizer.ggml.eos_token_id u32              = 200002
llama_model_loader: - kv  31:            tokenizer.ggml.padding_token_id u32              = 199999
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {#-\n  In addition to the normal input...
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - kv  34:                          general.file_type u32              = 38
llama_model_loader: - type  f32:  289 tensors
llama_model_loader: - type q8_0:   98 tensors
llama_model_loader: - type mxfp4:   72 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = MXFP4 MoE
print_info: file size   = 11.27 GiB (4.63 BPW) 
load: printing all EOG tokens:
load:   - 199999 ('<|endoftext|>')
load:   - 200002 ('<|return|>')
load:   - 200007 ('<|end|>')
load:   - 200012 ('<|call|>')
load: special_eog_ids contains both '<|return|>' and '<|call|>' tokens, removing '<|end|>' token from EOG list
load: special tokens cache size = 21
load: token to piece cache size = 1.3332 MB
print_info: arch             = gpt-oss
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 2880
print_info: n_layer          = 24
print_info: n_head           = 64
print_info: n_head_kv        = 8
print_info: n_rot            = 64
print_info: n_swa            = 128
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 2880
print_info: n_expert         = 32
print_info: n_expert_used    = 4
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = yarn
print_info: freq_base_train  = 150000.0
print_info: freq_scale_train = 0.03125
print_info: n_ctx_orig_yarn  = 4096
print_info: rope_finetuned   = unknown
print_info: model type       = 20B
print_info: model params     = 20.91 B
print_info: general.name     = Gpt Oss 20b
print_info: n_ff_exp         = 2880
print_info: vocab type       = BPE
print_info: n_vocab          = 201088
print_info: n_merges         = 446189
print_info: BOS token        = 199998 '<|startoftext|>'
print_info: EOS token        = 200002 '<|return|>'
print_info: EOT token        = 199999 '<|endoftext|>'
print_info: PAD token        = 199999 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 199999 '<|endoftext|>'
print_info: EOG token        = 200002 '<|return|>'
print_info: EOG token        = 200012 '<|call|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 24 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 25/25 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   586.82 MiB
load_tensors:        CUDA0 model buffer size = 10949.38 MiB
.................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 64000
llama_context: n_ctx_per_seq = 64000
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 150000.0
llama_context: freq_scale    = 0.03125
llama_context: n_ctx_per_seq (64000) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.77 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 64000 cells
llama_kv_cache:      CUDA0 KV buffer size =   796.88 MiB
llama_kv_cache: size =  796.88 MiB ( 64000 cells,  12 layers,  1/1 seqs), K (q8_0):  398.44 MiB, V (q8_0):  398.44 MiB
llama_kv_cache_iswa: creating     SWA KV cache, size = 768 cells
llama_kv_cache:      CUDA0 KV buffer size =     9.56 MiB
llama_kv_cache: size =    9.56 MiB (   768 cells,  12 layers,  1/1 seqs), K (q8_0):    4.78 MiB, V (q8_0):    4.78 MiB
llama_context:      CUDA0 compute buffer size =   398.38 MiB
llama_context:  CUDA_Host compute buffer size =   197.77 MiB
llama_context: graph nodes  = 1352
llama_context: graph splits = 50
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|return|> logit bias = -inf
common_init_from_params: added <|call|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 64000
srv    load_model: cache_reuse is not supported by this context, it will be disabled
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 64000
srv          init: Enable thinking? 0
main: model loaded
main: chat template, chat_template: {#-
  In addition to the normal inputs of `messages` and `tools`, this template also accepts the
  following kwargs:
  - "builtin_tools": A list, can contain "browser" and/or "python".
  - "model_identity": A string that optionally describes the model identity.
  - "reasoning_effort": A string that describes the reasoning effort, defaults to "medium".
 #}

--------------- DELETE LONG TEMPLATE --------------------------
...
------------------ TEMPLATE END -------------------------------
<|start|>assistant
{%- endif -%}, example_format: '<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-09-14

Reasoning: medium

# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|><|start|>developer<|message|># Instructions

You are a helpful assistant

<|end|><|start|>user<|message|>Hello<|end|><|start|>assistant<|channel|>final<|message|>Hi there<|end|><|start|>user<|message|>How are you?<|end|><|start|>assistant'
main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
srv  log_server_r: request: GET / 192.168.3.196 200
srv  log_server_r: request: GET /props 192.168.3.196 200
srv  params_from_: Chat format: GPT-OSS
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 64000, n_keep = -1, n_prompt_tokens = 74
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 74, n_tokens = 74, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 74, n_tokens = 74
slot update_slots: id  0 | task 0 | SWA checkpoint create, pos_min = 0, pos_max = 73, size = 0.923 MiB, total = 1/3 (0.923 MiB)
slot      release: id  0 | task 0 | stop processing: n_past = 2861, truncated = 0
slot print_timing: id  0 | task 0 | 
prompt eval time =     148.40 ms /    74 tokens (    2.01 ms per token,   498.65 tokens per second)
       eval time =   33755.92 ms /  2788 tokens (   12.11 ms per token,    82.59 tokens per second)
      total time =   33904.32 ms /  2862 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 192.168.3.196 200
srv  params_from_: Chat format: GPT-OSS
slot launch_slot_: id  0 | task 2789 | processing task
slot update_slots: id  0 | task 2789 | new prompt, n_ctx_slot = 64000, n_keep = -1, n_prompt_tokens = 2093
slot update_slots: id  0 | task 2789 | n_past = 75, cache_tokens.size() = 2861, seq_id = 0, pos_min = 2093, n_swa = 128
slot update_slots: id  0 | task 2789 | SWA checkpoint restore, pos_min = 0, pos_max = 73, size = 0.923 MiB
slot update_slots: id  0 | task 2789 | kv cache rm [73, end)
slot update_slots: id  0 | task 2789 | prompt processing progress, n_past = 2093, n_tokens = 2020, progress = 0.965122
slot update_slots: id  0 | task 2789 | prompt done, n_past = 2093, n_tokens = 2020
slot update_slots: id  0 | task 2789 | SWA checkpoint create, pos_min = 1325, pos_max = 2092, size = 9.572 MiB, total = 2/3 (10.494 MiB)
slot      release: id  0 | task 2789 | stop processing: n_past = 4195, truncated = 0
slot print_timing: id  0 | task 2789 | 
prompt eval time =    3409.01 ms /  2020 tokens (    1.69 ms per token,   592.55 tokens per second)
       eval time =   36031.66 ms /  2103 tokens (   17.13 ms per token,    58.37 tokens per second)
      total time =   39440.67 ms /  4123 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 192.168.3.196 200
srv  params_from_: Chat format: GPT-OSS
slot launch_slot_: id  0 | task 4893 | processing task
slot update_slots: id  0 | task 4893 | new prompt, n_ctx_slot = 64000, n_keep = -1, n_prompt_tokens = 4107
slot update_slots: id  0 | task 4893 | n_past = 2094, cache_tokens.size() = 4195, seq_id = 0, pos_min = 3427, n_swa = 128
slot update_slots: id  0 | task 4893 | SWA checkpoint restore, pos_min = 1325, pos_max = 2092, size = 9.572 MiB
slot update_slots: id  0 | task 4893 | kv cache rm [2092, end)
slot update_slots: id  0 | task 4893 | prompt processing progress, n_past = 4107, n_tokens = 2015, progress = 0.490626
slot update_slots: id  0 | task 4893 | prompt done, n_past = 4107, n_tokens = 2015
slot update_slots: id  0 | task 4893 | SWA checkpoint create, pos_min = 3339, pos_max = 4106, size = 9.572 MiB, total = 3/3 (20.066 MiB)
slot      release: id  0 | task 4893 | stop processing: n_past = 6060, truncated = 0
slot print_timing: id  0 | task 4893 | 
prompt eval time =    8622.14 ms /  2015 tokens (    4.28 ms per token,   233.70 tokens per second)
       eval time =   44836.68 ms /  1954 tokens (   22.95 ms per token,    43.58 tokens per second)
      total time =   53458.83 ms /  3969 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 192.168.3.196 200
srv  params_from_: Chat format: GPT-OSS
slot launch_slot_: id  0 | task 6848 | processing task
slot update_slots: id  0 | task 6848 | new prompt, n_ctx_slot = 64000, n_keep = -1, n_prompt_tokens = 5962
slot update_slots: id  0 | task 6848 | n_past = 4108, cache_tokens.size() = 6060, seq_id = 0, pos_min = 5292, n_swa = 128
slot update_slots: id  0 | task 6848 | SWA checkpoint restore, pos_min = 3339, pos_max = 4106, size = 9.572 MiB
slot update_slots: id  0 | task 6848 | kv cache rm [4106, end)
slot update_slots: id  0 | task 6848 | prompt processing progress, n_past = 5962, n_tokens = 1856, progress = 0.311305
slot update_slots: id  0 | task 6848 | prompt done, n_past = 5962, n_tokens = 1856
slot update_slots: id  0 | task 6848 | SWA checkpoint erase, pos_min = 3339, pos_max = 4106, size = 9.572 MiB
slot update_slots: id  0 | task 6848 | SWA checkpoint create, pos_min = 5194, pos_max = 5961, size = 9.572 MiB, total = 3/3 (28.715 MiB)
slot      release: id  0 | task 6848 | stop processing: n_past = 6320, truncated = 0
slot print_timing: id  0 | task 6848 | 
prompt eval time =   13466.61 ms /  1856 tokens (    7.26 ms per token,   137.82 tokens per second)
       eval time =    9276.91 ms /   359 tokens (   25.84 ms per token,    38.70 tokens per second)
      total time =   22743.53 ms /  2215 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 192.168.3.196 200
^Csrv    operator(): operator(): cleaning up before exit...

[wang7x]

The issue stemmed from using quantized KV cache (--ctk and --ctv set to q8_0). After removing these parameters, inference worked correctly. I verified in the Ollama source code that GPT-OSS explicitly disallows quantized cache types due to its use of attention sinks. The code includes a check that disables quantized KV cache when the model architecture is gpt-oss.