KV Cache and Speculative Decoding: The Memory Budget

A draft head can already be inside a checkpoint and still need extra serving memory. The useful question is how the complete deployment changes: allocated KV capacity, request shape, first-token waiting, throughput and correctness all matter.

This note reads a corrected same-model campaign in WeZZard’s two-Spark report. The comparison uses Qwen3.8 Flash-Next with speculation disabled or NEXTN enabled. It is source-reported evidence, not a Run AI Here model test.

Figure: drawn directly from two reported KV pools and twelve original rate observations. Rate labels are derived medians; dots show the three source cells, and whiskers show their observed minimum and maximum. These ranges are not standard deviations or confidence intervals. The pool panel uses tokens, not bytes of RAM.

Three token numbers answer different questions

The selected launch profiles retain a 120,000-token context cap and --max-total-tokens 600000. The reported allocation is 600,000 KV tokens without speculation and 161,728 with NEXTN. A configured ceiling does not guarantee the engine allocated that many tokens.

Treat context cap, allocated pool and active request demand as separate observations. Do not divide the pool by user count and advertise the result as verified context capacity. Engine accounting and scheduling need their own evidence, especially when a model also maintains non-attention state.

The exact comparison boundary

The corrected campaign is dated 5 September 2026. It uses two GB10 Sparks, 121 GiB per node, TP2 and reported NCCL over RoCE. The artifact is RadixArk/Qwen3.8-Flash-Next-NVFP4, revision 7b719225242aacd3dbd3f9407468c2ee9a9d2594. Both profiles name sglang-qwen38fn:sm121-ours, bfloat16 KV and static memory fraction 0.80. Prefix caching is off; requests generate 512 tokens with ignore_eos. Campaign controls.

The recorded launch commands share their base options. The enabled arm adds NEXTN, one speculative step, top-k 1 and two draft tokens. These are source-declared flags. Exact image digests and immutable executed-runtime bindings remain unknown; a document revision does not fill those fields.

Read the raw cells before the summary

Original rows: short, long agentic. Their prompt column reports 74 and 30,185 tokens respectively. The long cell’s combined output is 3,072 tokens across six users, rather than 3,072 per user. Served rate includes first-token waiting; it is not per-user decode speed.

The selected medians are higher with NEXTN, while the pool is smaller. The observed long-workload ranges overlap. Three observations do not establish statistical significance or identify whether scheduling, memory pressure or another mechanism caused a particular variation. Neither workload establishes a general speedup for other prompts or deployments.

Why more draft work can leave less cache

The author’s speculation investigation explains that draft KV and speculative state draw from the serving budget before the main pool is sized. Its earlier tuning campaign is dated 4 September. Keep those earlier pool values separate from the corrected cohort above.

This suggests a practical inspection order: read the allocation after changing speculation, then test the request mix that actually matters. A headline improvement on a short request cannot settle a choice for multiple long requests. A useful run record includes both the successful cells and any failures or queuing observed under the target load.

A smaller KV dtype still needs a working kernel path

In the historical FP8-KV investigation, short-input success did not establish long-input support: a single longer prompt could enter the failing chunked-prefill path. Keep that incident attached to its tested image and settings. It does not prove that every current engine has the same limitation, or that a dtype name alone guarantees support.

Do not convert a token-pool difference into “freed GiB” without the model’s actual cache layout and engine accounting. Likewise, lowering a static budget is not automatically a way to create room for the cache charged against that budget.

A run record that supports the next decision

Retain weight revision, image digest, executed engine, allocated KV dtype and pool, speculation options, context cap, cache policy, input/output lengths, actual concurrency, timing definition, warmup, repetitions and failures. Keep correctness checks alongside performance, and state which fields are missing.

The two-Spark configuration note supplies the wider deployment map. For AI shows how to query the raw observations and computed summaries while keeping their sources and conditions. Historical community evidence can guide a test plan; it cannot certify your current workload.