# KV Cache and Speculative Decoding: The Memory Budget A draft head can already be inside a checkpoint and still need extra serving memory. The useful question is how the complete deployment changes: allocated KV capacity, request shape, first-token waiting, throughput and correctness all matter. This note reads a corrected same-model campaign in [WeZZard’s two-Spark report](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md). The comparison uses Qwen3.8 Flash-Next with speculation disabled or NEXTN enabled. It is source-reported evidence, not a Run AI Here model test. ![Source-reported Qwen KV pools and served-rate medians, with three raw campaign values and observed ranges.](/assets/kv-cache-speculation-source-chart.webp) Figure: drawn directly from two reported KV pools and twelve original rate observations. Rate labels are derived medians; dots show the three source cells, and whiskers show their observed minimum and maximum. These ranges are not standard deviations or confidence intervals. The pool panel uses tokens, not bytes of RAM. ## Three token numbers answer different questions The [selected launch profiles](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L562) retain a 120,000-token context cap and `--max-total-tokens 600000`. The reported allocation is 600,000 KV tokens without speculation and 161,728 with NEXTN. A configured ceiling does not guarantee the engine allocated that many tokens. Treat context cap, allocated pool and active request demand as separate observations. Do not divide the pool by user count and advertise the result as verified context capacity. Engine accounting and scheduling need their own evidence, especially when a model also maintains non-attention state. ## The exact comparison boundary The corrected campaign is dated 5 September 2026. It uses two GB10 Sparks, 121 GiB per node, TP2 and reported NCCL over RoCE. The artifact is `RadixArk/Qwen3.8-Flash-Next-NVFP4`, revision `7b719225242aacd3dbd3f9407468c2ee9a9d2594`. Both profiles name `sglang-qwen38fn:sm121-ours`, bfloat16 KV and static memory fraction 0.80. Prefix caching is off; requests generate 512 tokens with `ignore_eos`. [Campaign controls](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L25). The recorded launch commands share their base options. The enabled arm adds NEXTN, one speculative step, top-k 1 and two draft tokens. These are source-declared flags. Exact image digests and immutable executed-runtime bindings remain unknown; a document revision does not fill those fields. ## Read the raw cells before the summary | Workload / users | Arm | Three source rates, tokens/s | Derived median | |---|---|---|---:| | Short / 1 | Off | 26.0, 19.9, 22.6 | 22.6 | | Short / 1 | NEXTN | 42.6, 43.0, 42.2 | 42.6 | | Long agentic / 6 | Off | 22.5, 27.6, 24.7 | 24.7 | | Long agentic / 6 | NEXTN | 27.4, 28.6, 30.1 | 28.6 | Original rows: [short](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L472), [long agentic](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L345). Their prompt column reports 74 and 30,185 tokens respectively. The long cell’s combined output is 3,072 tokens across six users, rather than 3,072 per user. Served rate includes first-token waiting; it is not per-user decode speed. The selected medians are higher with NEXTN, while the pool is smaller. The observed long-workload ranges overlap. Three observations do not establish statistical significance or identify whether scheduling, memory pressure or another mechanism caused a particular variation. Neither workload establishes a general speedup for other prompts or deployments. ## Why more draft work can leave less cache The author’s [speculation investigation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L44) explains that draft KV and speculative state draw from the serving budget before the main pool is sized. Its earlier tuning campaign is dated 4 September. Keep those earlier pool values separate from the corrected cohort above. This suggests a practical inspection order: read the allocation after changing speculation, then test the request mix that actually matters. A headline improvement on a short request cannot settle a choice for multiple long requests. A useful run record includes both the successful cells and any failures or queuing observed under the target load. ## A smaller KV dtype still needs a working kernel path In the historical [FP8-KV investigation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L100), short-input success did not establish long-input support: a single longer prompt could enter the failing chunked-prefill path. Keep that incident attached to its tested image and settings. It does not prove that every current engine has the same limitation, or that a dtype name alone guarantees support. Do not convert a token-pool difference into “freed GiB” without the model’s actual cache layout and engine accounting. Likewise, lowering a static budget is not automatically a way to create room for the cache charged against that budget. ## A run record that supports the next decision Retain weight revision, image digest, executed engine, allocated KV dtype and pool, speculation options, context cap, cache policy, input/output lengths, actual concurrency, timing definition, warmup, repetitions and failures. Keep correctness checks alongside performance, and state which fields are missing. The [two-Spark configuration note](/guides/two-spark-deployment-conditions/) supplies the wider deployment map. [For AI](/for-ai/) shows how to query the raw observations and computed summaries while keeping their sources and conditions. Historical community evidence can guide a test plan; it cannot certify your current workload. ## Original sources and conditions - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469 - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/engines.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 421632af313b5cfe7fca8b65ddabe0afb7b13e34229aca03e2af00c6a6f5452e Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "id": "qwen-nospec-v3-short-c1-s1", "label": "qwen-nospec-v3-s1 short C1", "metric": "served_rate", "value": 26.0, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s1", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L475", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 475 }, { "id": "qwen-nospec-v3-short-c1-s2", "label": "qwen-nospec-v3-s2 short C1", "metric": "served_rate", "value": 19.9, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s2", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L476", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 476 }, { "id": "qwen-nospec-v3-short-c1-s3", "label": "qwen-nospec-v3-s3 short C1", "metric": "served_rate", "value": 22.6, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s3", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L477", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 477 }, { "id": "qwen-v3-short-c1-s1", "label": "qwen-v3-s1 short C1", "metric": "served_rate", "value": 42.6, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s1", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L472", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 472 }, { "id": "qwen-v3-short-c1-s2", "label": "qwen-v3-s2 short C1", "metric": "served_rate", "value": 43.0, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s2", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L473", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 473 }, { "id": "qwen-v3-short-c1-s3", "label": "qwen-v3-s3 short C1", "metric": "served_rate", "value": 42.2, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s3", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L474", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 474 }, { "id": "qwen-nospec-v3-agentic-33k-c6-s1", "label": "qwen-nospec-v3-s1 agentic-33k C6", "metric": "served_rate", "value": 22.5, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s1", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L348", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 348 }, { "id": "qwen-nospec-v3-agentic-33k-c6-s2", "label": "qwen-nospec-v3-s2 agentic-33k C6", "metric": "served_rate", "value": 27.6, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s2", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L349", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 349 }, { "id": "qwen-nospec-v3-agentic-33k-c6-s3", "label": "qwen-nospec-v3-s3 agentic-33k C6", "metric": "served_rate", "value": 24.7, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s3", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L350", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 350 }, { "id": "qwen-v3-agentic-33k-c6-s1", "label": "qwen-v3-s1 agentic-33k C6", "metric": "served_rate", "value": 27.4, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s1", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L345", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 345 }, { "id": "qwen-v3-agentic-33k-c6-s2", "label": "qwen-v3-s2 agentic-33k C6", "metric": "served_rate", "value": 28.6, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s2", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L346", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 346 }, { "id": "qwen-v3-agentic-33k-c6-s3", "label": "qwen-v3-s3 agentic-33k C6", "metric": "served_rate", "value": 30.1, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s3", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L347", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 347 }, { "id": "qwen-nospec-v3-allocated-kv-pool", "label": "qwen-nospec-v3 allocated KV pool", "metric": "allocated_kv_pool", "value": 600000, "unit": "tokens", "statistic": "engine-allocated pool reported in the selected source cell; not maximum per-request context", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s1", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L475", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 475 }, { "id": "qwen-v3-allocated-kv-pool", "label": "qwen-v3 allocated KV pool", "metric": "allocated_kv_pool", "value": 161728, "unit": "tokens", "statistic": "engine-allocated pool reported in the selected source cell; not maximum per-request context", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s1", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L472", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 472 } ], "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null }, "comparison_groups": [ { "id": "qwen-nospec-v3-short-c1", "workload": "short", "concurrency": 1, "arm": "qwen-nospec-v3", "speculative_decoding": "off", "kv_pool_tokens": 600000, "reported_prompt_tokens": 74, "output_tokens_total": 512, "output_tokens_per_request": 512, "samples": [ { "point_id": "qwen-nospec-v3-short-c1-s1", "value": 26.0, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L475" }, { "point_id": "qwen-nospec-v3-short-c1-s2", "value": 19.9, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L476" }, { "point_id": "qwen-nospec-v3-short-c1-s3", "value": 22.6, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L477" } ], "summary": { "median": 22.6, "minimum": 19.9, "maximum": 26.0, "unit": "tokens/s", "derivation": "computed from the three source cells", "interval_kind": "observed minimum and maximum, not SD or confidence interval" } }, { "id": "qwen-v3-short-c1", "workload": "short", "concurrency": 1, "arm": "qwen-v3", "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "kv_pool_tokens": 161728, "reported_prompt_tokens": 74, "output_tokens_total": 512, "output_tokens_per_request": 512, "samples": [ { "point_id": "qwen-v3-short-c1-s1", "value": 42.6, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L472" }, { "point_id": "qwen-v3-short-c1-s2", "value": 43.0, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L473" }, { "point_id": "qwen-v3-short-c1-s3", "value": 42.2, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L474" } ], "summary": { "median": 42.6, "minimum": 42.2, "maximum": 43.0, "unit": "tokens/s", "derivation": "computed from the three source cells", "interval_kind": "observed minimum and maximum, not SD or confidence interval" } }, { "id": "qwen-nospec-v3-agentic-33k-c6", "workload": "agentic-33k", "concurrency": 6, "arm": "qwen-nospec-v3", "speculative_decoding": "off", "kv_pool_tokens": 600000, "reported_prompt_tokens": 30185, "output_tokens_total": 3072, "output_tokens_per_request": 512, "samples": [ { "point_id": "qwen-nospec-v3-agentic-33k-c6-s1", "value": 22.5, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L348" }, { "point_id": "qwen-nospec-v3-agentic-33k-c6-s2", "value": 27.6, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L349" }, { "point_id": "qwen-nospec-v3-agentic-33k-c6-s3", "value": 24.7, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L350" } ], "summary": { "median": 24.7, "minimum": 22.5, "maximum": 27.6, "unit": "tokens/s", "derivation": "computed from the three source cells", "interval_kind": "observed minimum and maximum, not SD or confidence interval" } }, { "id": "qwen-v3-agentic-33k-c6", "workload": "agentic-33k", "concurrency": 6, "arm": "qwen-v3", "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "kv_pool_tokens": 161728, "reported_prompt_tokens": 30185, "output_tokens_total": 3072, "output_tokens_per_request": 512, "samples": [ { "point_id": "qwen-v3-agentic-33k-c6-s1", "value": 27.4, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L345" }, { "point_id": "qwen-v3-agentic-33k-c6-s2", "value": 28.6, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L346" }, { "point_id": "qwen-v3-agentic-33k-c6-s3", "value": 30.1, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L347" } ], "summary": { "median": 28.6, "minimum": 27.4, "maximum": 30.1, "unit": "tokens/s", "derivation": "computed from the three source cells", "interval_kind": "observed minimum and maximum, not SD or confidence interval" } } ], "launch_comparison": { "off_config": "qwen-nospec-v3-s1", "on_config": "qwen-v3-s1", "source_command_prefix_identical": true, "added_options": [ "--speculative-algorithm", "NEXTN", "--speculative-num-steps", "1", "--speculative-eagle-topk", "1", "--speculative-num-draft-tokens", "2" ], "scope": "source-declared flags; not independently observed running processes" } } ```