# Run AI Here — evidence explained # LLM Speed: TTFT, Decode and Served Rate TTFT tells you how long you wait for the first token. Decode rate describes output generation after that point. Served rate divides output tokens by the request's total elapsed time. Read all three with the workload and timing definition beside them: a fast stream can still follow a long wait. The examples below come from separate community experiments. Each explains a metric in its own configuration. RunAIHere has not independently reproduced these measurements. ![Conceptual timeline showing the three intervals used for TTFT, decode and served rate.](/assets/llm-speed-metrics-spark.webp) *Conceptual illustration generated as SVG by the local Spark text model and reviewed by RunAIHere. Not to scale; no benchmark values.* ## Read the clock before the headline Start by identifying when the author starts and stops the timer. Does the reported interval include queueing, prompt processing or a cold start? Was the server already warm? Does an output-rate calculation include the first-token wait? Those choices determine what a reader can infer from a tokens-per-second figure. For an interactive session, both the initial wait and subsequent delivery affect the experience. A batch-serving report also needs its concurrency and aggregation rules. Keep an aggregate throughput figure separate from a per-request or per-stream rate. ## TTFT: waiting before the first token One dated report used an Apple M4 Pro with 64 GB unified memory, MLX 0.31.0 and mlx-lm 0.30.7. For Qwen3.5-4B int4, the 4,096-token-input test reported mean TTFT of **6.14 seconds**, with standard deviation **0.25 seconds**, across 20 measurements after one warmup iteration. This describes that input shape and setup. The report's generation-rate test used a different input length, so keep those profiles distinct. [Original measurement row](https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L40) ## Decode: the generation phase A four-DGX-Spark report using TP4 and DSpark k5 reported **43.12 tokens/s** as mean per-stream decode at concurrency one, averaged across eight categories with counting excluded. Input and output shapes varied by category; the source reports one batch per cell. It concerns a DeepSeek V4.1 Flash launch checkpoint. The record does not establish performance for DeepSeek-V4.1-Flash-1017. [Original report and row](https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L130) ## Served rate: the whole request window A different operator defines served rate as output tokens divided by whole-request wall-clock time, including first-token waiting. Its two-Spark DeepSeek-V4-Flash-0731 agentic test at approximately 33k context reported **22.5 tokens/s**, the median of three campaigns. That cell used concurrency one, 512 output tokens with ignore-eos enabled, prefix caching off and full-shape warmup. The reported spread ratio was **1.04**; this is a descriptive spread across those campaigns, not a confidence interval. [Original method and result](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L70) The two DeepSeek examples use different checkpoints and hardware counts. Their values do not provide a controlled comparison between the machines or models. ## A checklist before choosing a deployment - Match the exact checkpoint, quantization, runtime and hardware topology. - Record input length, output length, concurrency and stopping rules. - Keep cache, warmup and speculative decoding settings with each result. - Retain the statistic and repetitions; distinguish standard deviation, observed spread and confidence intervals. - Check missing conditions in the original report before treating a nearby configuration as equivalent. The accompanying timeline is conceptual and has no measured scale. Its purpose is to show which interval each metric describes. The linked original reports remain the authority for their measurements. ## Original sources and conditions - weklund: https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md - Document revision: 5ae9b491f7a0539d5abef949eded6bd57785a981 - SHA-256: ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3 - tonyd2wild: https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md - Document revision: d45538f67366c62da668c35ec1afd4fcd0e4c8b4 - SHA-256: 8c72dbb42a49a0f9b5b5f8a70055a467823993012bdb47cad748b5b0e04e36a7 - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "Qwen3.5-4B int4 first token", "metric": "ttft", "value": 6.14, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.25 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-4B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L40", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306" }, { "label": "Eight-category mean, C1", "metric": "per_stream_decode", "value": 43.12, "unit": "tokens/s", "statistic": "mean across eight categories; counting excluded", "uncertainty": null, "conditions": { "system": "4 x NVIDIA DGX Spark", "node_count": 4, "parallelism": "TP4", "kv_dtype": null, "context_limit_tokens": 300000, "experiment_date": null, "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "speculative_decoding": "DSpark k5", "temperature": 0, "thinking": false, "warmup": "performed", "repetitions": "one batch per cell", "input_tokens": "about 30–120; summary prompt longer", "output_token_budget": "150–256; natural stopping", "weight_precision": "MXFP4 experts / MXFP8 dense", "concurrency": 1, "denominator": "after first token" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L130", "source_document_sha256": "8c72dbb42a49a0f9b5b5f8a70055a467823993012bdb47cad748b5b0e04e36a7", "source_record_id": "deepseek-v41-four-spark-boot10" }, { "label": "DeepSeek V4 Flash 0731, long agentic", "metric": "served_rate", "value": 22.5, "unit": "tokens/s", "statistic": "median of three campaigns", "uncertainty": { "spread_ratio": 1.04 }, "conditions": { "system": "2 x DGX Spark; 121 GiB unified memory each; 200 GbE", "node_count": 2, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-05", "output_tokens": 512, "ignore_eos": true, "prefix_cache": false, "warmup": "all workload/user shapes, full generation", "repetitions": 3, "concurrency": 1, "measurement_denominator": "whole request wall clock including TTFT", "workload": "agentic 33k context", "model": "DeepSeek-V4-Flash-0731", "kv": "fp8_ds_mla", "kv_pool_tokens": 517484, "draft": "DSpark k5 probabilistic" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L70", "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469", "source_record_id": "spark-pair-moe-served-rate" } ], "conditions": { "cross_source_ranking_allowed": false, "one_shared_experiment": false, "independent_site_runs": 0 } } ``` --- # DeepSeek V4.1 Flash 1017: Deployment Evidence The pinned model card and vLLM recipe identify the 1017 artifact and provide configuration starting points. They do not establish its speed on your machine. A useful deployment decision needs the exact model identity, a fully resolved recipe and evidence for the intended hardware and workload. ![Conceptual model card and recipe linked to an empty run-evidence record.](/assets/deepseek-1017-evidence-cover.webp) *AI-generated conceptual illustration. Configuration documents and independent measurements are separate kinds of evidence.* ## Start with the exact checkpoint The selected recipe defaults to `deepseek-ai/DeepSeek-V4.1-Flash-1017`. Keep that identity when reading performance reports. A document about an earlier V4.1 checkpoint remains evidence about its original experiment, even if a newer recipe now selects 1017. [Pinned default artifact](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114) Preserve three separate references: the document version you read, the model artifact you intend to load and the runtime version actually executed. Recording a recipe commit alone does not establish the latter two. ## What the publisher recommends The official card recommends `temperature 1.0` and `top_p 1.0`. It declares a `1M` context and recommends a `max_tokens` budget of at least `256K`. These are publisher settings and limits; they do not demonstrate an allocation on a particular workstation or require every answer to reach that output length. [Sampling](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L81), [context and output budget](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L84) The card supplies a Python encoding reference rather than a Jinja-format chat template. Check how the chosen serving integration handles the model's protocol; that fact alone does not mean every application needs a new custom implementation. [Prompt interface](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L69) ## Read the recipe's declarations precisely | Selected declaration | Meaning for planning | |---|---| | Model-wide minimum vLLM: `0.31.0` | A documented baseline; check additional variant and hardware requirements | | Default precision label: `fp8` | A recipe label, not a tensor-by-tensor precision audit | | Default planning minimum: `614 GB` | A declared planning value for this variant, not measured peak allocation or guaranteed capacity | | Shared base arguments: `--tokenizer-mode deepseek_v41` | One layer of the configuration, not a complete launch command | | Default selected features: `spec_decoding`, `text_only` | Configuration selections, not measured acceleration or a full model-capability statement | Source: [minimum runtime](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L42), [default variant and selections](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114), [planning minimum](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L125). ## Turn the sources into a deployment decision Choose an artifact variant and hardware profile before assembling settings. Read the shared, variant, hardware, strategy and optional-feature layers together. A copied fragment can lose the conditions that made the original configuration meaningful. Use the documents to prepare a reproducible configuration record. A planning value can help reject an unsuitable proposal, but it cannot replace a traceable allocation and successful request on the intended system. Keep advertised context, configured context, tested input length and available serving capacity as separate fields. ## What a useful run report still needs For evidence that someone actually ran the intended configuration, look for: - Exact artifact, conversion and runtime identities. - Device count, memory, topology and complete resolved launch settings. - Input and output lengths, concurrency and stopping rules. - Cache, warmup, sampling and speculative-decoding settings. - Timing definitions, repetitions, failures and uncertainty. Missing details stay unknown. The indexed sources used for this article do not establish a RunAIHere speed measurement for 1017, and this page does not supply a hardware performance ranking. RunAIHere has not independently executed or benchmarked this checkpoint. ## Original sources and conditions - deepseek-ai: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md - Document revision: 1bae0e1088993b9b82de08f59de5be5a34e42170 - SHA-256: b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a - vllm-project: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml - Document revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - SHA-256: 8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [ { "id": "deepseek-v41-flash-1017-official-fact-0", "label": "Recommended sampling", "value": "temperature 1.0; top_p 1.0", "classification": "publisher_declaration", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L81", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" }, { "id": "deepseek-v41-flash-1017-official-fact-1", "label": "Published context / recommended output budget", "value": "1M context; max_tokens at least 256K", "classification": "publisher_declaration", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L84", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" }, { "id": "deepseek-v41-flash-1017-official-fact-2", "label": "Prompt interface", "value": "Python encoder; no Jinja-format chat template", "classification": "publisher_declaration", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L69", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" }, { "id": "deepseek-v41-flash-vllm-fact-4", "label": "Model-wide minimum vLLM", "value": "0.31.0", "classification": "recipe_declaration", "url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L42", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "id": "deepseek-v41-flash-vllm-fact-6", "label": "Default artifact", "value": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "classification": "recipe_declaration", "url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "id": "deepseek-v41-flash-vllm-fact-7", "label": "Default precision label", "value": "fp8", "classification": "recipe_declaration", "url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "id": "deepseek-v41-flash-vllm-fact-8", "label": "Default planning minimum (GB; recipe declaration)", "value": 614, "classification": "recipe_declaration", "url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L125", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "id": "deepseek-v41-flash-vllm-fact-10", "label": "Shared base arguments (incomplete launch)", "value": [ "--tokenizer-mode", "deepseek_v41" ], "classification": "recipe_declaration", "url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "id": "deepseek-v41-flash-vllm-fact-13", "label": "Features selected by default", "value": [ "spec_decoding", "text_only" ], "classification": "recipe_declaration", "url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" } ], "reported_results": [], "conditions": { "model_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "weight_revision": null, "runtime_execution_revision": null, "hardware": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "independently_measured": false } } ``` --- # Qwen3.8 Flash-Next: Resolve the Deployment Recipe Choose an exact artifact, a hardware profile and a complete set of settings before treating a recipe as a deployment plan. A model-family name, a memory label and a version floor answer only part of that question. This note explains the pinned documents; RunAIHere has not run this model as a benchmark. ![Conceptual layers for artifact, hardware and settings leading to an empty verification checklist.](/assets/qwen38-recipe-layers.webp) *AI-generated conceptual illustration. A resolved configuration still needs evidence from an actual run.* ## Identify the artifact before budgeting memory The recipe defaults to the Inferact NVFP4 conversion. Its three declared variant minimums are: | Variant | Artifact | Declared planning minimum | |---|---|---| | NVFP4 default | `Inferact/Qwen3.8-Flash-Next-NVFP4` | 130 GB | | FP8 | `Qwen/Qwen3.8-Flash-Next-FP8` | 250 GB | | BF16 | `Qwen/Qwen3.8-Flash-Next` | 423 GB | These are recipe declarations, not measured peaks or guarantees for a selected prompt length. [Pinned variants](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89) For a machine-specific budget, keep the conversion revision, actual files, runtime allocation and workload together. A smaller planning number cannot establish an equivalent quality outcome. Do not silently replace a conversion with the publisher's original repository when writing a result record. ## Preserve the runtime requirement's qualification The recipe gives a minimum vLLM version of `0.30.0`, marks `nightly_required: true` and selects nightly images. Its pip note also says stable vLLM ≥0.30.0 runs the model. Keep both statements; a version floor alone does not identify the image the maintainer verified. [Engine and installation notes](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L25) Save the executed version or container digest alongside the documentation revision. If an installation differs from the documented path, make that difference visible before interpreting a success or failure. ## Resolve settings as a set Shared settings enable prefix caching. The default strategy is `single_node_tep`; selected features include `text_only` and `long_context`. MTP speculation is an optional configuration, with a hardware-specific selection as well. [Shared settings](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L40), [features](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L63), [strategy](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L158) Read variant, hardware, strategy and feature overrides together. A shared argument fragment is insufficient as a launch command. In a later performance report, record the resolved settings rather than just linking the recipe. A feature's name does not prove acceleration in that workload. ## Keep context and component counts precise The publisher declares 262,144 native context tokens, extensible to 1,000,000. Its component description lists a 125B language model with 6B active parameters, plus 51B n-gram embeddings and 4B MTP. The recipe's 176B summary uses a different boundary. [Publisher breakdown](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L46), [native and extended context](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L71), [recipe summary](https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L37) Retain those labels instead of manufacturing a new total. For capacity decisions, distinguish advertised context, configured context, tested input length and space left for output. None supplies a concurrent-user capacity measurement. ## What a useful result must preserve - Exact publisher or conversion artifact, weight revision and runtime identity. - Device count, memory, topology and complete launch configuration. - Input/output lengths, modality, concurrency and stopping rules. - Prefix-cache state, KV settings, warmup and speculation mode. - Timing definition, repetitions, failures and uncertainty. Unknown fields remain unknown in this article's export. Use the source documents to prepare a configuration, then seek an original report for that exact setup. This page supplies no tokens-per-second claim or cross-hardware ranking. ## Original sources and conditions - Qwen: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md - Document revision: de4b8e4d43b917e7706784d8bb445c9af86a3540 - SHA-256: 35ca37ccc366f1ba478dab33841a2c0c18ce53fd62f291ca05341f7728b225b2 - vllm-project: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml - Document revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - SHA-256: 609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [ { "name": "LM component / active / n-gram / MTP", "value": "125B / 6B / 51B / 4B", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L46", "source_document_sha256": "35ca37ccc366f1ba478dab33841a2c0c18ce53fd62f291ca05341f7728b225b2" }, { "name": "Native / extended context", "value": "262144 / up to 1000000 tokens", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L71", "source_document_sha256": "35ca37ccc366f1ba478dab33841a2c0c18ce53fd62f291ca05341f7728b225b2" }, { "name": "Total parameters as counted by recipe", "value": "176B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L37", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Model-wide minimum vLLM", "value": "0.30.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L25", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Default artifact", "value": "Inferact/Qwen3.8-Flash-Next-NVFP4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 130, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L93", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--max-num-seqs", "256", "--gpu-memory-utilization", "0.90", "--enable-prefix-caching" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Features selected by default", "value": [ "text_only", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Runtime installation qualification", "value": { "nightly_required_flag": true, "minimum_vllm_version": "0.30.0", "pip_note": "Nightly wheels recommended and verified; stable vLLM >=0.30.0 also runs the model." }, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L25", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Default strategy", "value": "single_node_tep", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L158", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "default declared planning minimum", "value": 130, "unit": "GB", "classification": "recipe_declaration_not_measured_peak", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L93", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "fp8 declared planning minimum", "value": 250, "unit": "GB", "classification": "recipe_declaration_not_measured_peak", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L125", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "bf16 declared planning minimum", "value": 423, "unit": "GB", "classification": "recipe_declaration_not_measured_peak", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L155", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" } ], "reported_results": [], "conditions": { "weight_revision": null, "runtime_execution_revision": null, "hardware": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "prefix_cache_measured_state": null, "independently_measured": false } } ``` --- # Mac Weight Precision: Speed, First Token and Memory Weight precision is a useful deployment choice only when the model and workload stay attached to the result. A smaller memory footprint does not tell you how accurate the answers remain. A fast generation rate does not tell you how long the first token takes. This note selects the 4B and 9B rows from [weklund’s pinned Qwen3.5 MLX report](https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md). They provide a useful same-machine example. They are historical Qwen3.5 measurements, not measurements of the latest Qwen checkpoint or every Mac. ![Source-reported Qwen3.5 4B and 9B comparison with separate generation, TTFT and peak-memory panels.](/assets/mac-precision-memory-source-chart.webp) Figure: deterministically drawn from the 18 selected source values. Rate and TTFT bars show the reported standard deviation; memory bars have no error bars because per-cell dispersion was not supplied. The workloads differ between panels. Full values and conditions are also available in text and JSON. ## The conditions that make this comparison meaningful The report identifies an M4 Pro with 10P+4E CPU cores, 20 GPU cores and 64 GB unified memory; MLX 0.31.0 and mlx-lm 0.30.7; experiment date 6 March 2026; one warmup followed by 20 measurements. Its rate table uses 256 input and 100 generated tokens. The TTFT and memory tables use 4,096 input tokens. These are [the report’s selected workloads](https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L19), rather than one common request profile. For the two long-input tables, the selected sections do not specify output length. That field stays unknown. The general methodology’s maximum generation length does not establish the exact length used for every section. ## Same model, three weight formats | Model / weight format | Generation rate, tokens/s | TTFT, seconds | Reported peak, GiB | |---|---:|---:|---:| | 4B int4 | 68.0 ± 0.4 | 6.14 ± 0.25 | 6.4 | | 4B int8 | 42.7 ± 0.3 | 6.14 ± 0.25 | 8.5 | | 4B bf16 | 27.7 ± 0.2 | 4.67 ± 0.03 | 10.2 | | 9B int4 | 38.5 ± 0.4 | 10.88 ± 0.61 | 9.0 | | 9B int8 | 24.1 ± 0.2 | 10.63 ± 0.40 | 13.4 | | 9B bf16 | 15.0 ± 0.4 | 8.20 ± 0.06 | 19.7 | Attribution: weklund; [generation rows](https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L27), [TTFT rows](https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L40), [memory rows](https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L53). ± means standard deviation, not a confidence interval. The memory column is a reported peak, not a model-file size or total-system free-memory reading. ## Read the tradeoff against your task In these selected rows, int4 has the higher generation rate and lower reported memory, while bf16 has the shorter first-token wait. That pattern explains why “the fastest precision” needs a stated objective. For a long output, investigate sustained delivery and total completion time. For a short answer after a long prompt, investigate first-token wait. For a Mac used alongside other applications, record actual system pressure and swap during a representative request. The table can narrow a test plan; it cannot replace those observations. Avoid subtracting the GiB peak from the machine’s GB specification to advertise “free RAM.” The units and measurement boundaries differ. Likewise, do not combine the short-input rate with the long-input TTFT to calculate an end-to-end response time: they describe different workloads. ## What is still unknown The selected report does not pin exact converted-weight hashes, conversion execution revisions, KV dtype, concurrency, sampling or cache state. Those fields remain null in the export. The reported cooldown policy is useful context, but it is not a thermal trace. There is no matched task-quality or perplexity evaluation here. The original document also includes broader recommendations; this note does not turn those recommendations into measured quantization accuracy. Before choosing a smaller format for extraction, coding or reasoning, compare its outputs against the same task set and acceptance criteria. ## How to use the evidence without overextending it Keep one candidate model fixed, record its conversion and runtime, and choose representative input and output lengths. Check correctness separately from responsiveness and memory. Record warmup, cache, concurrency and repeated-run variation. Only then decide which format serves the task well enough. Run AI Here has not independently benchmarked these models. The adjacent Markdown and JSON retain the source revision, row anchors, uncertainty and missing fields so an AI reader can preserve the same limits. ## Original sources and conditions - weklund: https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md - Document revision: 5ae9b491f7a0539d5abef949eded6bd57785a981 - SHA-256: ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "Qwen3.5-4B int4 generation_rate", "metric": "generation_rate", "value": 68.0, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.4 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "int4", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L27", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 27 }, { "label": "Qwen3.5-4B int8 generation_rate", "metric": "generation_rate", "value": 42.7, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.3 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "int8", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L27", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 27 }, { "label": "Qwen3.5-4B bf16 generation_rate", "metric": "generation_rate", "value": 27.7, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.2 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "bf16", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L27", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 27 }, { "label": "Qwen3.5-9B int4 generation_rate", "metric": "generation_rate", "value": 38.5, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.4 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "int4", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L28", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 28 }, { "label": "Qwen3.5-9B int8 generation_rate", "metric": "generation_rate", "value": 24.1, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.2 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "int8", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L28", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 28 }, { "label": "Qwen3.5-9B bf16 generation_rate", "metric": "generation_rate", "value": 15.0, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.4 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "bf16", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L28", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 28 }, { "label": "Qwen3.5-4B int4 ttft", "metric": "ttft", "value": 6.14, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.25 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L40", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 40 }, { "label": "Qwen3.5-4B int8 ttft", "metric": "ttft", "value": 6.14, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.25 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "int8", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L40", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 40 }, { "label": "Qwen3.5-4B bf16 ttft", "metric": "ttft", "value": 4.67, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.03 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "bf16", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L40", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 40 }, { "label": "Qwen3.5-9B int4 ttft", "metric": "ttft", "value": 10.88, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.61 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L41", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 41 }, { "label": "Qwen3.5-9B int8 ttft", "metric": "ttft", "value": 10.63, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.4 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "int8", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L41", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 41 }, { "label": "Qwen3.5-9B bf16 ttft", "metric": "ttft", "value": 8.2, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.06 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "bf16", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L41", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 41 }, { "label": "Qwen3.5-4B int4 peak_memory", "metric": "peak_memory", "value": 6.4, "unit": "GiB", "statistic": "reported peak; no per-cell dispersion supplied", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L53", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 53 }, { "label": "Qwen3.5-4B int8 peak_memory", "metric": "peak_memory", "value": 8.5, "unit": "GiB", "statistic": "reported peak; no per-cell dispersion supplied", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "int8", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L53", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 53 }, { "label": "Qwen3.5-4B bf16 peak_memory", "metric": "peak_memory", "value": 10.2, "unit": "GiB", "statistic": "reported peak; no per-cell dispersion supplied", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-4B", "weight_precision": "bf16", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L53", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 53 }, { "label": "Qwen3.5-9B int4 peak_memory", "metric": "peak_memory", "value": 9.0, "unit": "GiB", "statistic": "reported peak; no per-cell dispersion supplied", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L54", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 54 }, { "label": "Qwen3.5-9B int8 peak_memory", "metric": "peak_memory", "value": 13.4, "unit": "GiB", "statistic": "reported peak; no per-cell dispersion supplied", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "int8", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L54", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 54 }, { "label": "Qwen3.5-9B bf16 peak_memory", "metric": "peak_memory", "value": 19.7, "unit": "GiB", "statistic": "reported peak; no per-cell dispersion supplied", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "model": "Qwen3.5-9B", "weight_precision": "bf16", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md#L54", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3", "source_record_id": "qwen35-m4pro-20260306", "source_line": 54 } ], "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "experiment_date": "2026-03-06", "repetitions": 20, "warmup_iterations": 1, "weight_revision": null, "conversion_revision": null, "kv_dtype": null, "concurrency": null, "cache_state": null, "sampling": null, "runtime_execution_revision": null, "quality_evaluation": null, "comparison_scope": "same machine, same model, same metric workload only", "hardware_memory_unit": "GB as reported; not GiB", "generation_workload": { "input_tokens": 256, "output_tokens": 100 }, "ttft_and_memory_workload": { "input_tokens": 4096, "output_tokens": null }, "catalog_point_count_unchanged": 49, "selection_status": "additional selection from an existing primary document, not new experiments" } } ``` --- # Two DGX Sparks: Read the Deployment Conditions A two-node model deployment is a combination of weights, runtime, attention cache, speculation and transport. A model name and a cable speed leave too much unspecified to reproduce it. This note reads [WeZZard’s pinned two-Spark campaign](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md), remeasured on 5 September 2026, alongside its detailed configuration ledger. It explains what to record before trying a similar deployment. These are community observations; Run AI Here has not run the models. ![Conceptual two-node deployment with separate local-memory spaces, a network link and configuration symbols.](/assets/dual-spark-conditions-concept.webp) AI-generated conceptual illustration. It is not a measured topology, a combined-memory claim or a photograph of the author’s cluster. ## Start with the campaign boundary The source reports two GB10 DGX Sparks, 121 GiB per node and a back-to-back 200 GbE link. Its corrected campaign has prefix caching off, 512 output tokens per request with `--ignore-eos`, full-length warmup for each workload and user shape, and median summaries from three campaigns. Served rate includes first-token waiting time in the request wall clock. [Original conditions](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L15). Each node retains its own memory and runtime state. Do not translate the two memory specifications into a promise of one freely allocatable memory pool. A launch-time context cap, an allocated KV pool and measured simultaneous users describe different constraints. ## Preserve the actual artifact and launch profile The artifact names and checkpoint revisions below come from the [detailed REPORT ledger](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md). Short revision prefixes are displayed here; the full revisions and separate source-document hashes are retained in JSON. | Campaign profile | Artifact / revision prefix | Weight format | Runtime image tag / parallelism | |---|---|---|---| | `ds-v3` | `deepseek-ai/DeepSeek-V4-Flash-0731` / `7872f01b` | MXFP4 experts | `vllm-dsv4:ray` / TP2, Ray | | `qwen-v3` | `RadixArk/Qwen3.8-Flash-Next-NVFP4` / `7b719225` | NVFP4 | `sglang-qwen38fn:sm121-ours` / TP2 | | `glm-v3` | `LibertAIDAI/GLM-5.3-Flash-NVFP4` / `aa28e1f5` | NVFP4 experts | `sglang-glm53:gb10-tilelang` / TP2, EP2 | The local image tags identify the reported environments, but a tag is not an immutable image digest. They are not a general installation command or an endorsement of an unverified image with the same name. ## Keep context, KV pool and concurrency separate | Profile | Allocated KV dtype / pool tokens | Speculation | Launch context cap | Measured users / launch maximum | |---|---|---|---:|---| | `ds-v3` | `fp8_ds_mla` / 517,484 | DSpark k=5, probabilistic | 262,144 | 1 and 6 / 12 | | `qwen-v3` | `bfloat16` / 161,728 | NEXTN, 2 draft tokens | 120,000 | 1 and 6 / unspecified | | `glm-v3` | `bfloat16` / 162,688 | DFlash2, 8 draft tokens | 131,072 | 1 and 8 / 8 | Sources: [corrected pools](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L90) and launch profiles [DeepSeek](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L928), [Qwen](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L562), [GLM](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L270). GLM’s static memory fraction is 0.90. The configured maximum is not a successful measurement at that load. Nor does a pool value establish a usable per-request context by division: engine accounting, shared state and scheduling need inspection. Preserve the input-token count for each measured cell; a workload label is insufficient. ## Verify the transport inside the container The source’s [interconnect investigation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md) found that host networking worked while container verbs devices were absent and NCCL used sockets. A visible HCA listing alone did not prove active RDMA. For a comparable setup, inspect the devices exposed to the running containers and the transport actually used during a request. Retain observations from both nodes. A successful host bandwidth test and a server-ready message answer different questions from a completed distributed inference request. ## Separate historical tuning incidents from the final campaign The earlier [4 September speculation investigation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L44) recorded a Qwen pool change from 600,000 to 145,024 tokens after enabling MTP. That earlier value is not the corrected campaign’s 161,728-token pool. It illustrates why the allocated pool must be re-read after a configuration change, rather than supplying a universal MTP memory formula. That investigation also corrected an FP8-KV failure explanation: a single long prompt could trigger the chunked-prefill path. “Short prompt succeeded” therefore did not establish that the tested image supported long-context FP8 KV. The successful profile in the table uses bfloat16 KV. Keep failure records attached to their own build and settings. ## What to capture before calling a reproduction successful Record the exact artifact revision, conversion, image digest and executed runtime version. Read the allocated KV dtype and pool from startup evidence. Preserve speculation settings, context cap, prefix-cache state, actual input/output lengths, measured users and request failures. Warm the shapes you will measure on both nodes and retain clock and power observations under load. The [engine inventory](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/engines.md#L14) separately records a Qwen-image version observation on 4 September. It does not supply an immutable image digest binding all three corrected campaign launches, so those execution fields remain unknown here. GLM’s exact draft-model revision also remains unknown in this extraction. These profiles do not establish a cross-model ranking, current best settings or DeepSeek 1017 compatibility. Use the separate [1017 evidence note](/guides/deepseek-v41-flash-1017/) for checkpoint identity and the [Spark preflight checklist](/hardware/dgx-spark/deployment-checklist/) to prepare a run record. ## Original sources and conditions - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469 - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 9e7904db6d48fc44691c9f170c03f402c77480dcbf54f66bc107ef57ef0ea94f - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/engines.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 421632af313b5cfe7fca8b65ddabe0afb7b13e34229aca03e2af00c6a6f5452e Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [ { "label": "Physical link versus actual transport", "value": "200 GbE wire did not establish container RDMA; verbs devices and active NCCL transport had to be checked", "classification": "source_reported_configuration_incident", "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md#L9", "source_document_sha256": "9e7904db6d48fc44691c9f170c03f402c77480dcbf54f66bc107ef57ef0ea94f", "source_document_id": "spark-docs-interconnect", "source_line": 9 }, { "label": "Earlier speculation campaign pool", "value": { "date": "2026-09-04", "no_speculation_tokens": 600000, "with_speculation_tokens": 145024, "not_the_September_5_pool": true }, "classification": "source_reported_configuration_not_general_capacity", "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L44", "source_document_sha256": "46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f", "source_document_id": "spark-docs-speculation", "source_line": 44 }, { "label": "Qwen image version observation", "value": { "observed_on": "2026-09-04", "image_tag": "sglang-qwen38fn:sm121-ours", "reported_engine_version": "0.0.0.dev1+gd91c3682b", "campaign_immutable_digest": null }, "classification": "source_reported_image_observation_not_immutable_campaign_version", "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/engines.md#L14", "source_document_sha256": "421632af313b5cfe7fca8b65ddabe0afb7b13e34229aca03e2af00c6a6f5452e", "source_document_id": "spark-docs-engines", "source_line": 14 }, { "label": "Qwen long-prompt FP8 KV failure", "value": "A single 30,301-token prompt also failed; the source identifies chunked prefill with a nonzero prefix, not concurrency itself, as the trigger in that image", "classification": "source_reported_failure_bound_to_build", "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L100", "source_document_sha256": "46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f", "source_document_id": "spark-docs-speculation", "source_line": 100 } ], "reported_results": [], "conditions": { "hardware": "2 x DGX Spark GB10 sm_121", "node_count": 2, "memory_per_node_gib": 121, "physical_interconnect_gbps": 200, "campaign_date": "2026-09-05", "prefix_cache": false, "output_tokens_per_request": 512, "ignore_eos": true, "warmup": "every workload/concurrency shape, full generation; each node has its own JIT cache", "repetitions": 3, "summary_statistic": "median of three campaigns", "served_rate_denominator": "all output tokens / whole request wall clock, including TTFT", "runtime_execution_immutable_digest": null, "shared_memory_pool_claim": false, "cross_model_ranking_allowed": false, "current_1017_reproduction_claim": false }, "deployment_profiles": [ { "id": "ds-v3", "model_artifact": "deepseek-ai/DeepSeek-V4-Flash-0731", "checkpoint_revision": "7872f01b1d1fe23eabc4c98b48bffcef5a386062", "weight_precision": "MXFP4 experts (block 32; source weight audit)", "engine": "vLLM", "image_tag": "vllm-dsv4:ray", "executed_engine_version": null, "image_digest": null, "parallelism": "TP2; Ray across two nodes", "context_limit_tokens": 262144, "configured_max_requests": 12, "measured_concurrency": [ 1, 6 ], "kv_dtype": "fp8_ds_mla", "kv_pool_tokens": 517484, "speculation": "DSpark k=5, probabilistic", "prefix_cache": false, "configuration_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L928", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_line": 928 }, "artifact_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L591", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_line": 591 }, "pool_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L97", "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469", "source_document_id": "spark-pair-moe-served-rate", "source_line": 97 } }, { "id": "qwen-v3", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "checkpoint_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "executed_engine_version": null, "image_digest": null, "parallelism": "TP2; two nodes", "context_limit_tokens": 120000, "configured_max_requests": null, "measured_concurrency": [ 1, 6 ], "kv_dtype": "bfloat16", "kv_pool_tokens": 161728, "speculation": "NEXTN; one step, top-k 1, draft tokens 2", "prefix_cache": false, "configuration_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L562", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_line": 562 }, "artifact_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L303", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_line": 303 }, "pool_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L103", "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469", "source_document_id": "spark-pair-moe-served-rate", "source_line": 103 } }, { "id": "glm-v3", "model_artifact": "LibertAIDAI/GLM-5.3-Flash-NVFP4", "checkpoint_revision": "aa28e1f54130286c95fee10d0705c74ce8743734", "weight_precision": "NVFP4 experts (block 16; source weight audit)", "engine": "SGLang", "image_tag": "sglang-glm53:gb10-tilelang", "executed_engine_version": null, "image_digest": null, "parallelism": "TP2 / EP2; two nodes", "context_limit_tokens": 131072, "configured_max_requests": 8, "measured_concurrency": [ 1, 8 ], "kv_dtype": "bfloat16", "kv_pool_tokens": 162688, "speculation": "DFlash2; draft tokens 8", "prefix_cache": false, "mem_fraction_static": 0.9, "draft_artifact_revision": null, "configuration_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L270", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_line": 270 }, "artifact_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L18", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_line": 18 }, "pool_evidence": { "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L106", "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469", "source_document_id": "spark-pair-moe-served-rate", "source_line": 106 } } ] } ``` --- # KV Cache and Speculative Decoding: The Memory Budget A draft head can already be inside a checkpoint and still need extra serving memory. The useful question is how the complete deployment changes: allocated KV capacity, request shape, first-token waiting, throughput and correctness all matter. This note reads a corrected same-model campaign in [WeZZard’s two-Spark report](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md). The comparison uses Qwen3.8 Flash-Next with speculation disabled or NEXTN enabled. It is source-reported evidence, not a Run AI Here model test. ![Source-reported Qwen KV pools and served-rate medians, with three raw campaign values and observed ranges.](/assets/kv-cache-speculation-source-chart.webp) Figure: drawn directly from two reported KV pools and twelve original rate observations. Rate labels are derived medians; dots show the three source cells, and whiskers show their observed minimum and maximum. These ranges are not standard deviations or confidence intervals. The pool panel uses tokens, not bytes of RAM. ## Three token numbers answer different questions The [selected launch profiles](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L562) retain a 120,000-token context cap and `--max-total-tokens 600000`. The reported allocation is 600,000 KV tokens without speculation and 161,728 with NEXTN. A configured ceiling does not guarantee the engine allocated that many tokens. Treat context cap, allocated pool and active request demand as separate observations. Do not divide the pool by user count and advertise the result as verified context capacity. Engine accounting and scheduling need their own evidence, especially when a model also maintains non-attention state. ## The exact comparison boundary The corrected campaign is dated 5 September 2026. It uses two GB10 Sparks, 121 GiB per node, TP2 and reported NCCL over RoCE. The artifact is `RadixArk/Qwen3.8-Flash-Next-NVFP4`, revision `7b719225242aacd3dbd3f9407468c2ee9a9d2594`. Both profiles name `sglang-qwen38fn:sm121-ours`, bfloat16 KV and static memory fraction 0.80. Prefix caching is off; requests generate 512 tokens with `ignore_eos`. [Campaign controls](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L25). The recorded launch commands share their base options. The enabled arm adds NEXTN, one speculative step, top-k 1 and two draft tokens. These are source-declared flags. Exact image digests and immutable executed-runtime bindings remain unknown; a document revision does not fill those fields. ## Read the raw cells before the summary | Workload / users | Arm | Three source rates, tokens/s | Derived median | |---|---|---|---:| | Short / 1 | Off | 26.0, 19.9, 22.6 | 22.6 | | Short / 1 | NEXTN | 42.6, 43.0, 42.2 | 42.6 | | Long agentic / 6 | Off | 22.5, 27.6, 24.7 | 24.7 | | Long agentic / 6 | NEXTN | 27.4, 28.6, 30.1 | 28.6 | Original rows: [short](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L472), [long agentic](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L345). Their prompt column reports 74 and 30,185 tokens respectively. The long cell’s combined output is 3,072 tokens across six users, rather than 3,072 per user. Served rate includes first-token waiting; it is not per-user decode speed. The selected medians are higher with NEXTN, while the pool is smaller. The observed long-workload ranges overlap. Three observations do not establish statistical significance or identify whether scheduling, memory pressure or another mechanism caused a particular variation. Neither workload establishes a general speedup for other prompts or deployments. ## Why more draft work can leave less cache The author’s [speculation investigation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L44) explains that draft KV and speculative state draw from the serving budget before the main pool is sized. Its earlier tuning campaign is dated 4 September. Keep those earlier pool values separate from the corrected cohort above. This suggests a practical inspection order: read the allocation after changing speculation, then test the request mix that actually matters. A headline improvement on a short request cannot settle a choice for multiple long requests. A useful run record includes both the successful cells and any failures or queuing observed under the target load. ## A smaller KV dtype still needs a working kernel path In the historical [FP8-KV investigation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L100), short-input success did not establish long-input support: a single longer prompt could enter the failing chunked-prefill path. Keep that incident attached to its tested image and settings. It does not prove that every current engine has the same limitation, or that a dtype name alone guarantees support. Do not convert a token-pool difference into “freed GiB” without the model’s actual cache layout and engine accounting. Likewise, lowering a static budget is not automatically a way to create room for the cache charged against that budget. ## A run record that supports the next decision Retain weight revision, image digest, executed engine, allocated KV dtype and pool, speculation options, context cap, cache policy, input/output lengths, actual concurrency, timing definition, warmup, repetitions and failures. Keep correctness checks alongside performance, and state which fields are missing. The [two-Spark configuration note](/guides/two-spark-deployment-conditions/) supplies the wider deployment map. [For AI](/for-ai/) shows how to query the raw observations and computed summaries while keeping their sources and conditions. Historical community evidence can guide a test plan; it cannot certify your current workload. ## Original sources and conditions - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469 - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/engines.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 421632af313b5cfe7fca8b65ddabe0afb7b13e34229aca03e2af00c6a6f5452e Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "id": "qwen-nospec-v3-short-c1-s1", "label": "qwen-nospec-v3-s1 short C1", "metric": "served_rate", "value": 26.0, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s1", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L475", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 475 }, { "id": "qwen-nospec-v3-short-c1-s2", "label": "qwen-nospec-v3-s2 short C1", "metric": "served_rate", "value": 19.9, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s2", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L476", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 476 }, { "id": "qwen-nospec-v3-short-c1-s3", "label": "qwen-nospec-v3-s3 short C1", "metric": "served_rate", "value": 22.6, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s3", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L477", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 477 }, { "id": "qwen-v3-short-c1-s1", "label": "qwen-v3-s1 short C1", "metric": "served_rate", "value": 42.6, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s1", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L472", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 472 }, { "id": "qwen-v3-short-c1-s2", "label": "qwen-v3-s2 short C1", "metric": "served_rate", "value": 43.0, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s2", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L473", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 473 }, { "id": "qwen-v3-short-c1-s3", "label": "qwen-v3-s3 short C1", "metric": "served_rate", "value": 42.2, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s3", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L474", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 474 }, { "id": "qwen-nospec-v3-agentic-33k-c6-s1", "label": "qwen-nospec-v3-s1 agentic-33k C6", "metric": "served_rate", "value": 22.5, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s1", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L348", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 348 }, { "id": "qwen-nospec-v3-agentic-33k-c6-s2", "label": "qwen-nospec-v3-s2 agentic-33k C6", "metric": "served_rate", "value": 27.6, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s2", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L349", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 349 }, { "id": "qwen-nospec-v3-agentic-33k-c6-s3", "label": "qwen-nospec-v3-s3 agentic-33k C6", "metric": "served_rate", "value": 24.7, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s3", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L350", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 350 }, { "id": "qwen-v3-agentic-33k-c6-s1", "label": "qwen-v3-s1 agentic-33k C6", "metric": "served_rate", "value": 27.4, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s1", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L345", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 345 }, { "id": "qwen-v3-agentic-33k-c6-s2", "label": "qwen-v3-s2 agentic-33k C6", "metric": "served_rate", "value": 28.6, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s2", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L346", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 346 }, { "id": "qwen-v3-agentic-33k-c6-s3", "label": "qwen-v3-s3 agentic-33k C6", "metric": "served_rate", "value": 30.1, "unit": "tokens/s", "statistic": "one campaign cell; median is derived separately from three cells", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s3", "campaign_repetitions": 3, "workload": "agentic-33k", "concurrency": 6, "reported_prompt_tokens": 30185, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 3072, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L347", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 347 }, { "id": "qwen-nospec-v3-allocated-kv-pool", "label": "qwen-nospec-v3 allocated KV pool", "metric": "allocated_kv_pool", "value": 600000, "unit": "tokens", "statistic": "engine-allocated pool reported in the selected source cell; not maximum per-request context", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-nospec-v3-s1", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 600000, "speculative_decoding": "off", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L475", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 475 }, { "id": "qwen-v3-allocated-kv-pool", "label": "qwen-v3 allocated KV pool", "metric": "allocated_kv_pool", "value": 161728, "unit": "tokens", "statistic": "engine-allocated pool reported in the selected source cell; not maximum per-request context", "uncertainty": null, "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null, "configuration": "qwen-v3-s1", "campaign_repetitions": 3, "workload": "short", "concurrency": 1, "reported_prompt_tokens": 74, "prompt_token_count_scope": "REPORT prompt column; preserved as reported", "output_tokens_total": 512, "kv_pool_tokens": 161728, "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "failed_requests": 0 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L472", "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e", "source_document_id": "spark-REPORT", "source_record_id": "spark-pair-moe-served-rate", "source_line": 472 } ], "conditions": { "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "system": "2 x DGX Spark GB10 sm_121; 121 GiB unified memory each", "node_count": 2, "parallelism": "TP2; 2 nodes", "physical_interconnect_gbps": 200, "reported_transport": "NCCL over RoCE", "experiment_date": "2026-09-05", "engine": "SGLang", "image_tag": "sglang-qwen38fn:sm121-ours", "image_digest": null, "runtime_execution_revision": null, "executed_engine_version": null, "kv_dtype": "bfloat16", "prefix_cache": false, "ignore_eos": true, "output_tokens_per_request": 512, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "mem_fraction_static": 0.8, "warmup": "corrected campaign: every workload/user shape, full generation, per-node JIT cache", "clock_state": "author reports corrected underclock before this campaign; no independent traces here", "measurement_denominator": "all request output tokens / entire cell wall clock including TTFT", "temperature": null, "top_p": null }, "comparison_groups": [ { "id": "qwen-nospec-v3-short-c1", "workload": "short", "concurrency": 1, "arm": "qwen-nospec-v3", "speculative_decoding": "off", "kv_pool_tokens": 600000, "reported_prompt_tokens": 74, "output_tokens_total": 512, "output_tokens_per_request": 512, "samples": [ { "point_id": "qwen-nospec-v3-short-c1-s1", "value": 26.0, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L475" }, { "point_id": "qwen-nospec-v3-short-c1-s2", "value": 19.9, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L476" }, { "point_id": "qwen-nospec-v3-short-c1-s3", "value": 22.6, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L477" } ], "summary": { "median": 22.6, "minimum": 19.9, "maximum": 26.0, "unit": "tokens/s", "derivation": "computed from the three source cells", "interval_kind": "observed minimum and maximum, not SD or confidence interval" } }, { "id": "qwen-v3-short-c1", "workload": "short", "concurrency": 1, "arm": "qwen-v3", "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "kv_pool_tokens": 161728, "reported_prompt_tokens": 74, "output_tokens_total": 512, "output_tokens_per_request": 512, "samples": [ { "point_id": "qwen-v3-short-c1-s1", "value": 42.6, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L472" }, { "point_id": "qwen-v3-short-c1-s2", "value": 43.0, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L473" }, { "point_id": "qwen-v3-short-c1-s3", "value": 42.2, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L474" } ], "summary": { "median": 42.6, "minimum": 42.2, "maximum": 43.0, "unit": "tokens/s", "derivation": "computed from the three source cells", "interval_kind": "observed minimum and maximum, not SD or confidence interval" } }, { "id": "qwen-nospec-v3-agentic-33k-c6", "workload": "agentic-33k", "concurrency": 6, "arm": "qwen-nospec-v3", "speculative_decoding": "off", "kv_pool_tokens": 600000, "reported_prompt_tokens": 30185, "output_tokens_total": 3072, "output_tokens_per_request": 512, "samples": [ { "point_id": "qwen-nospec-v3-agentic-33k-c6-s1", "value": 22.5, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L348" }, { "point_id": "qwen-nospec-v3-agentic-33k-c6-s2", "value": 27.6, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L349" }, { "point_id": "qwen-nospec-v3-agentic-33k-c6-s3", "value": 24.7, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L350" } ], "summary": { "median": 24.7, "minimum": 22.5, "maximum": 27.6, "unit": "tokens/s", "derivation": "computed from the three source cells", "interval_kind": "observed minimum and maximum, not SD or confidence interval" } }, { "id": "qwen-v3-agentic-33k-c6", "workload": "agentic-33k", "concurrency": 6, "arm": "qwen-v3", "speculative_decoding": "NEXTN; one step; top-k 1; draft tokens 2", "kv_pool_tokens": 161728, "reported_prompt_tokens": 30185, "output_tokens_total": 3072, "output_tokens_per_request": 512, "samples": [ { "point_id": "qwen-v3-agentic-33k-c6-s1", "value": 27.4, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L345" }, { "point_id": "qwen-v3-agentic-33k-c6-s2", "value": 28.6, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L346" }, { "point_id": "qwen-v3-agentic-33k-c6-s3", "value": 30.1, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L347" } ], "summary": { "median": 28.6, "minimum": 27.4, "maximum": 30.1, "unit": "tokens/s", "derivation": "computed from the three source cells", "interval_kind": "observed minimum and maximum, not SD or confidence interval" } } ], "launch_comparison": { "off_config": "qwen-nospec-v3-s1", "on_config": "qwen-v3-s1", "source_command_prefix_identical": true, "added_options": [ "--speculative-algorithm", "NEXTN", "--speculative-num-steps", "1", "--speculative-eagle-topk", "1", "--speculative-num-draft-tokens", "2" ], "scope": "source-declared flags; not independently observed running processes" } } ``` --- # Ollama, llama.cpp, vLLM or SGLang: Choose a Serving Path Start with the exact model artifact, the hardware backend that can execute it, and the workload you need to serve. Then shortlist a runtime whose documented interfaces and maintenance requirements fit. A tool name alone does not tell you whether a particular checkpoint, quantization and attention kernel will work together. This is a documentation-informed selection guide. The four maintainer READMEs were checked on 18 October 2026 and pinned below. Run AI Here did not install or benchmark these revisions. The recommendations are editorial starting points; the feature statements are attributed to the maintainers. ![Conceptual selection sequence from artifact to backend to workload, with four runtime options.](/assets/a07-batch3-local-spark.webp) Figure: a conceptual selection sequence, generated as SVG by the local Spark model and reviewed for publication. The four runtime names are options, with no speed scale or ranking. ## A practical shortlist | Your first requirement | Start the evaluation with | What still needs proof | |---|---|---| | Packaged model workflow and application integration | Ollama | The selected model runs where you intend, and the application uses the right endpoint and prompt format | | A GGUF artifact and direct control of a CLI or server | llama.cpp | That architecture and quantization work with your chosen build and backend | | Concurrent API serving, batching and parallel execution | vLLM | The exact model, kernel, hardware and request shape are supported together | | A serving or agent workload with a matching deployment recipe | SGLang | The recipe matches your artifact, platform, image and KV/attention settings | These are ways to narrow the first experiment. Capabilities overlap. An OpenAI-compatible endpoint does not make model loading, tool parsing, cache behavior or scheduling identical across runtimes. ## What the maintainers actually document **Ollama** documents platform installers, application integrations, a model library and a REST API for running and managing models. Its README now describes local and cloud access. If local execution is a requirement, select and verify an explicit local model and endpoint rather than inferring locality from the client application. Record the installed version and actual model identifier. [Pinned Ollama README](https://github.com/ollama/ollama/blob/36a6e994d2f6c4ab94b3555185154d069e5dba87/README.md#L90). **llama.cpp** documents C/C++ inference, Apple acceleration through Metal and related frameworks, CUDA, several other backends, and CPU/GPU hybrid execution. Its current quickstart uses a GGUF artifact and exposes CLI and OpenAI-compatible server entry points. It is a useful starting point when the artifact and backend controls are central to your decision. Verify architecture support and the exact binary build before planning offload or context capacity. [Pinned llama.cpp README](https://github.com/ggml-org/llama.cpp/blob/547208dca250ceb3e4c8615598d043f2b602a527/README.md#L60). **vLLM** documents continuous batching, chunked prefill, prefix caching, several parallelism modes and API serving. Its quantization list includes GGUF as well as multiple other formats; its hardware list also includes plugin paths. These lists are a starting point for compatibility checks, not proof that every listed model, format and accelerator combination works. Evaluate its serving features against your real queue and prompt distribution. [Pinned vLLM README](https://github.com/vllm-project/vllm/blob/da9134bc89364465f68aa679564435936d157a14/README.md#L28). **SGLang** describes serving, agentic and rollout workloads and links model/hardware recipes. Its supported-hardware table includes both DGX Spark and Apple Silicon through Metal/MLX. Avoid treating an older platform summary as the current compatibility boundary. Use the matching platform guide and cookbook recipe, then preserve the image/build, attention backend, KV dtype and speculative mode. A platform entry does not validate every checkpoint on that platform. [Pinned SGLang README](https://github.com/sgl-project/sglang/blob/e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0/README.md#L21). ## Keep document revisions separate from installed versions | Maintainer document | Reviewed repository revision | Installed version tested here | |---|---|---| | Ollama README | `36a6e994d2f6` | Unknown; no installation test | | llama.cpp README | `547208dca250` | Unknown; no installation test | | vLLM README | `da9134bc8936` | Unknown; no installation test | | SGLang README | `e5dc10e73b0c` | Unknown; no installation test | The full revisions and SHA-256 checksums are in the source ledger and JSON. A repository revision identifies the reviewed document tree. It does not identify the package installed on your machine, the binary build options, or an executed container image. Record those separately when evaluating a runtime. Avoid treating a moving image tag or an unpinned installation command as an immutable reproduction recipe. ## Compare a deployment contract before comparing a rate Write a small acceptance contract before trying the shortlist. Name the checkpoint and revision, tokenizer and chat template, weight format, KV dtype, hardware and backend, engine/build, context cap and allocated cache. Add required API behavior: streaming, tool calls, structured output, embeddings or multimodal inputs, as applicable. A successful plain chat request does not validate all of these features. Next, choose request shapes that resemble the application. Include a short interactive prompt and a representative longer prompt, with explicit output lengths and concurrency. Preserve prefix-cache and speculative-decoding state. Report first-token waiting and whole-request served rate separately; a fast decode segment can coexist with a long prefill. Use the [metric guide](/guides/llm-speed-metrics/) and retain correctness checks alongside timing. Memory must also be observed at the intended shape. A weight file that fits does not establish KV capacity, concurrency or long-context correctness. The [KV and speculation guide](/guides/kv-cache-speculative-decoding/) shows why configured ceilings, allocated pools and active request demand need separate fields. ## If the target is a pair of Sparks Treat GB10 architecture support, the attention path, container transport and per-node memory as explicit requirements. A recipe for a different checkpoint or GPU generation is a lead to investigate. Before a long load, inspect what the chosen build supports; after launch, verify the transport and actual cache allocation. The [GB10 incident guide](/guides/gb10-long-context-failures/) explains how a dtype assertion, a backend rejection and a shared-memory limit can look like one generic serving failure while requiring different responses. For AI readers, the adjacent JSON retains four `runtime_options`, each with documented capability, operator checks, source line and document digest. Its `tested_engine_version` is `null`. Keep the source fact and the editorial shortlist together when recommending a path. ## Original sources and conditions - ollama: https://github.com/ollama/ollama/tree/d1ed3fd13ac73f60cff00bbead8af24a68c4200d - Document revision: d1ed3fd13ac73f60cff00bbead8af24a68c4200d - SHA-256: 9e5a7d406950fdd2742084065ab38a3ea09b5aab5e2f13ef7d64147229ae0d8b - ggml-org: https://github.com/ggml-org/llama.cpp/tree/f74e2fa6a31cc33ab4ee4ae516641cfe62df3924 - Document revision: f74e2fa6a31cc33ab4ee4ae516641cfe62df3924 - SHA-256: dc2d34687c844ecf929d9e9a9c9c78e8023b52abf3b2b3d039600c79016673b6 - vllm-project: https://github.com/vllm-project/vllm/tree/ce49c47cbb241b59a32ef110079a5261dbd08e49 - Document revision: ce49c47cbb241b59a32ef110079a5261dbd08e49 - SHA-256: 4dc7399323b9660117f8cc1226afb7a81cdbb912dbb4ae74b66d89052417d11c - sgl-project: https://github.com/sgl-project/sglang/tree/2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e - Document revision: 2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e - SHA-256: 7c99b50fa5ef9f4a666a2f73860f746d3fc259e7e7912d4bff88dffdc1dc4c5e - ollama: https://github.com/ollama/ollama/blob/36a6e994d2f6c4ab94b3555185154d069e5dba87/README.md - Document revision: 36a6e994d2f6c4ab94b3555185154d069e5dba87 - SHA-256: 9e5a7d406950fdd2742084065ab38a3ea09b5aab5e2f13ef7d64147229ae0d8b - ggml-org: https://github.com/ggml-org/llama.cpp/blob/547208dca250ceb3e4c8615598d043f2b602a527/README.md - Document revision: 547208dca250ceb3e4c8615598d043f2b602a527 - SHA-256: dc2d34687c844ecf929d9e9a9c9c78e8023b52abf3b2b3d039600c79016673b6 - vllm-project: https://github.com/vllm-project/vllm/blob/da9134bc89364465f68aa679564435936d157a14/README.md - Document revision: da9134bc89364465f68aa679564435936d157a14 - SHA-256: 4dc7399323b9660117f8cc1226afb7a81cdbb912dbb4ae74b66d89052417d11c - sgl-project: https://github.com/sgl-project/sglang/blob/e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0/README.md - Document revision: e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0 - SHA-256: 7c99b50fa5ef9f4a666a2f73860f746d3fc259e7e7912d4bff88dffdc1dc4c5e Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [], "conditions": { "comparison_type": "documentation-informed editorial shortlist", "installed_versions_tested": null, "model_benchmarks_run": 0, "speed_ranking_allowed": false }, "runtime_options": [ { "id": "ollama", "name": "Ollama", "evidence_status": "maintainer_documented_capability", "document_revision": "36a6e994d2f6c4ab94b3555185154d069e5dba87", "tested_engine_version": null, "shortlist_when": "A packaged model workflow, app integrations, and a model-management REST API are the main requirements.", "operator_checks": [ "Select an explicit local model and endpoint if local execution is required; the README also offers cloud models.", "Record the actual Ollama version, model identifier and artifact digest.", "Validate the app API, prompt template and memory behavior with the intended request." ], "source_document_id": "ollama-readme-20261018", "source_document_sha256": "9e5a7d406950fdd2742084065ab38a3ea09b5aab5e2f13ef7d64147229ae0d8b", "source_line": 92, "evidence_url": "https://github.com/ollama/ollama/blob/36a6e994d2f6c4ab94b3555185154d069e5dba87/README.md#L92" }, { "id": "llama-cpp", "name": "llama.cpp", "evidence_status": "maintainer_documented_capability", "document_revision": "547208dca250ceb3e4c8615598d043f2b602a527", "tested_engine_version": null, "shortlist_when": "You want a GGUF-oriented starting point, direct CLI/server control, or documented CPU, Metal and CUDA backend choices.", "operator_checks": [ "Check architecture and quantization support for the exact GGUF artifact.", "Record the binary/build revision and selected backend.", "Measure GPU offload, KV allocation and request concurrency rather than inferring them from file size." ], "source_document_id": "llama-cpp-readme-20261018", "source_document_sha256": "dc2d34687c844ecf929d9e9a9c9c78e8023b52abf3b2b3d039600c79016673b6", "source_line": 72, "evidence_url": "https://github.com/ggml-org/llama.cpp/blob/547208dca250ceb3e4c8615598d043f2b602a527/README.md#L72" }, { "id": "vllm", "name": "vLLM", "evidence_status": "maintainer_documented_capability", "document_revision": "da9134bc89364465f68aa679564435936d157a14", "tested_engine_version": null, "shortlist_when": "A serving workload needs documented batching, prefix caching, parallel execution, and compatible API surfaces.", "operator_checks": [ "Verify the exact architecture, quantization, kernel and hardware combination; listed support is not a full compatibility matrix.", "Record the installed package/commit, wheel or image digest, and launch flags.", "Evaluate TTFT and served rate under representative concurrent requests." ], "source_document_id": "vllm-readme-20261018", "source_document_sha256": "4dc7399323b9660117f8cc1226afb7a81cdbb912dbb4ae74b66d89052417d11c", "source_line": 32, "evidence_url": "https://github.com/vllm-project/vllm/blob/da9134bc89364465f68aa679564435936d157a14/README.md#L32" }, { "id": "sglang", "name": "SGLang", "evidence_status": "maintainer_documented_capability", "document_revision": "e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0", "tested_engine_version": null, "shortlist_when": "A supported serving or agent workload has a matching model-and-hardware recipe you can maintain.", "operator_checks": [ "Start from the matching cookbook/platform recipe and pin it.", "Its README includes Apple Metal/MLX and DGX Spark; confirm per-model/backend limitations.", "Record image/build, KV dtype, attention backend and speculative mode separately." ], "source_document_id": "sglang-readme-20261018", "source_document_sha256": "7c99b50fa5ef9f4a666a2f73860f746d3fc259e7e7912d4bff88dffdc1dc4c5e", "source_line": 21, "evidence_url": "https://github.com/sgl-project/sglang/blob/e5dc10e73b0c4ccae7a9c4f454411b399e83a4f0/README.md#L21" } ] } ``` --- # GB10 Long Context: Read the Failure Before Changing Flags A long prompt can expose a kernel path that a short chat never exercises. On a pair of GB10 Sparks, first identify the failed layer: dtype handling, architecture support, per-block shared memory, correctness or container transport. Preserve the exact build and request shape before changing a setting. This note interprets historical records from [WeZZard’s dgx-spark-bench](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md), pinned at `25e8dbf1b6113dd0b3f12d27a670fb284944d260`. The relevant incidents are from September 2026; the documents were checked on 18 October. They do not establish the behavior of today’s upstream releases. Run AI Here did not reproduce these model runs. ![Conceptual incident review from observation through isolation to verification, with distinct evidence states.](/assets/a08-batch3-local-spark.webp) Figure: a conceptual incident-review diagram generated as SVG by the local Spark model and reviewed for publication. It contains no measured timing or hardware topology. ## Four evidence states, four different conclusions | Case | What the pinned source supports | What you should preserve | |---|---|---| | Qwen FP8 chunked prefill | A single-user failure was reproduced by the source | Probe version, KV dtype, chunk size and prompt shape | | QSA long-context corruption warning | A warning and launcher guard; the proposed ladder remains unperformed | Unknown threshold and missing correctness results | | GLM backend and tilelang failures | Different attempted paths failed; a tuned bf16 path served | The successful image, kernel tuning and later follow-up | | Hidden container RDMA devices | A source-verified transport incident | Running-container transport evidence, separate from host bandwidth | ## Case 1: a single user can trigger the FP8 KV crash The Qwen probe used `RadixArk/Qwen3.8-Flash-Next-NVFP4`, revision `7b719225242aacd3dbd3f9407468c2ee9a9d2594`, with `sglang-qwen38fn:sm121-ours`. Its log identifies SGLang `0.0.0.dev1+gd91c3682b`, `float8_e4m3fn` KV and a 310,656-token allocated pool. A short single-user control completed 64 output tokens and retrieved its needle. A 30,301-token single-user prompt then hit `Unsupported rhs dtype fp8e4nv` and terminated the scheduler. [Probe and diagnosis](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/qwen-qsa.md#L142). The source traces this to nonzero prefix lengths in chunked prefill. With an 8,192-token chunk size, a longer request reaches a path reading the FP8 paged pool instead of fresh bf16 K/V. Concurrency alone is therefore the wrong explanation. The document proposes a compatible build or dtype conversion; it supplies no verified fix commit. Raising the chunk size moves the boundary. It does not demonstrate a corrected kernel. The failed probe’s output count and the image digest remain unknown. Do not borrow the 332,480-token pool from an earlier campaign: it is a different observation in the same document. ## Case 2: the 120,000-token guard is not a correctness result The QSA document carries a warning about corruption beyond roughly 120K and contrasts it with a recipe requesting 262,144 tokens. The author inspected the image, did not find the named successor attention path, and capped the launcher at 120,000. The proposed 131K/262K retrieval ladder is explicitly still pending in this snapshot. [Warning, guard and unperformed experiment](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/qwen-qsa.md#L3). Preserve the distinction between an operating guard and a measured maximum safe context. No completed ladder, corruption threshold or upstream fix verification is available here. The named `pr36845.diff` is a successor reference, not proof of a merged or validated fix. A future retrieval pass would be useful evidence for that task; it would not establish universal long-context correctness. ## Case 3: unified RAM and kernel shared memory are different limits The GLM records concern `LibertAIDAI/GLM-5.3-Flash-NVFP4`, revision `aa28e1f54130286c95fee10d0705c74ce8743734`. FP8 attempts encountered an architecture allowlist rejection and a separate TensorRT-LLM unsupported-architecture error. A stock bf16 tilelang path requested 169,984 bytes of dynamic shared memory per block against a reported 101,376-byte device limit. This is not the machine’s 121 GiB unified-memory pool. [Attempt sequence](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/glm-dsa.md#L130). The source reports success with bf16 KV in `sglang-glm53:gb10-tilelang`, using `block_I=32`, `num_stages=1`, `threads=128`. The early run allocated 800,448 KV tokens and served 26,535-token prompts with eight users. The [pinned community build patch](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docker/patch-tilelang-gb10.py#L40) identifies the modified v1 path; it is not an upstream release-fix claim. Read the chronology. Although the early GLM document says DFlash remains untested, the [later speculation document](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L248) reports clean eight-token draft-verify capture for batch sizes one through eight on the tuned image. Keep that follow-up separate from the early no-speculation pool. Do not present the earlier warning as the final outcome. ## Case 4: verify transport inside the running container The source’s September 4 Qwen investigation found working host fabric and visible container HCA entries in sysfs, while `/dev/infiniband` was absent. NCCL used sockets. Mapping the required verbs devices, setting IPC_LOCK/memlock and selecting the connected HCA led to channel logs showing `NET/IB`. [Container evidence and remediation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md#L8). A host link test cannot confirm the serving container’s transport. Inspect the running service after startup; device visibility alone is also insufficient. This note does not promote the earlier sweep’s speed ratios to a current benchmark, and the original author still identifies remaining unexplained latency. ## A useful incident record Before tuning, save the first relevant error, artifact revision, engine version, image digest when known, attention backend, KV dtype and allocated pool. Add prompt/output lengths, concurrency, chunk size, cache state and speculative mode. An omitted runtime identifier stays unknown even when the documentation is pinned. Then write one falsifiable explanation and one change that tests it. Keep a short control and a representative longer correctness task. A server that starts proves startup; it does not prove long-context correctness. Preserve failed runs, successful follow-ups and unperformed proposals as separate records. The adjacent JSON exposes four `case_reports` with explicit `evidence_status`, per-case `conditions`, source pointers, remediation state and unresolved fields. Unknown values remain `null`; the warning is never exported as a completed experiment. Use the [two-Spark conditions guide](/guides/two-spark-deployment-conditions/) and [For AI reference](/for-ai/) alongside these cases. ## Original sources and conditions - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469 - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/qwen-qsa.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: b642874472f6727e05dc259ab38ff6bc7d78a80691436c65a9c34417abc201e8 - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/glm-dsa.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: a4b9448f5265f687504130bc1791ec3e7c1777d16c648851a94c3a3b4db99e32 - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 9e7904db6d48fc44691c9f170c03f402c77480dcbf54f66bc107ef57ef0ea94f - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f - WeZZard: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docker/patch-tilelang-gb10.py - Document revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - SHA-256: 88455f3811883262abc5ea292ca5db0dda91fd8fd6f6db7d12ce6d6a01b817ff Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [], "conditions": { "community_snapshot_revision": "25e8dbf1b6113dd0b3f12d27a670fb284944d260", "snapshot_date": "2026-09-06", "source_checked_on": "2026-10-18", "historical_cases_not_current_release_claims": true, "independent_site_model_runs": 0 }, "case_reports": [ { "id": "qwen-fp8-chunked-prefill", "title": "FP8 KV kills a single-request chunked prefill", "evidence_status": "source_reproduced_failure", "conditions": { "hardware": "2 x DGX Spark GB10 sm_121", "node_count": 2, "memory_per_node_gib": 121, "physical_interconnect_gbps": 200, "parallelism": "TP2", "engine": "SGLang", "image_digest": null, "runtime_execution_revision": null, "temperature": null, "top_p": null, "site_reproduced": false, "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "checkpoint_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "image_tag": "sglang-qwen38fn:sm121-ours", "experiment_date": "2026-09-04", "executed_engine_version": "0.0.0.dev1+gd91c3682b", "kv_dtype": "float8_e4m3fn", "kv_pool_tokens": 310656, "configured_max_total_tokens": 600000, "context_limit_tokens": 120000, "chunked_prefill_size": 8192, "concurrency": 1, "reported_long_prompt_tokens": 30301, "failed_probe_output_tokens": null, "short_control_workload": "agentic-4k", "short_control_prompt_tokens": null, "short_control_output_tokens": 64, "short_control_failed_requests": 0, "short_control_needle_hits": 1, "short_control_needle_trials": 1, "speculative_decoding": "same source launcher; exact probe flag tuple not fully logged" }, "symptom": "Unsupported rhs dtype fp8e4nv; scheduler exits during the longer probe.", "diagnosis": "The source traces nonzero prefix lengths to the _ck path, which consumes the FP8 paged KV pool; the no-prefix path consumes fresh bfloat16 tensors. Chunk splitting, rather than concurrent users alone, explains the single-user reproduction.", "remediation": { "state": "proposed_not_verified_in_this_probe", "proposal": "A compatible image/Triton path or a dtype conversion at the _ck call site; the later campaign used bfloat16 KV. Increasing chunk size only moves the failing boundary.", "verified_fix_commit": null }, "unresolved": [ "Failed-request generated-token count and image digest are not supplied.", "The probe does not establish the behavior of a current SGLang release." ], "observations": [ { "source_document_id": "spark-docs-qwen-qsa", "source_document_sha256": "b642874472f6727e05dc259ab38ff6bc7d78a80691436c65a9c34417abc201e8", "source_line": 152, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/qwen-qsa.md#L152" }, { "source_document_id": "spark-docs-qwen-qsa", "source_document_sha256": "b642874472f6727e05dc259ab38ff6bc7d78a80691436c65a9c34417abc201e8", "source_line": 146, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/qwen-qsa.md#L146" }, { "source_document_id": "spark-docs-qwen-qsa", "source_document_sha256": "b642874472f6727e05dc259ab38ff6bc7d78a80691436c65a9c34417abc201e8", "source_line": 136, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/qwen-qsa.md#L136" } ] }, { "id": "qsa-long-context-warning", "title": "The approximately 120K corruption warning is not a completed correctness test", "evidence_status": "source_warning_unperformed", "conditions": { "hardware": "2 x DGX Spark GB10 sm_121", "node_count": 2, "memory_per_node_gib": 121, "physical_interconnect_gbps": 200, "parallelism": "TP2", "engine": "SGLang", "image_digest": null, "runtime_execution_revision": null, "temperature": null, "top_p": null, "site_reproduced": false, "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "checkpoint_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "image_tag": "sglang-qwen38fn:sm121-ours", "experiment_date": null, "executed_engine_version": null, "kv_dtype": null, "kv_pool_tokens": null, "concurrency": null, "reported_prompt_tokens": null, "output_tokens": null, "source_recipe_context_limit_tokens": 262144, "launcher_context_limit_tokens": 120000, "proposed_test_contexts_tokens": [ 131072, 262144 ] }, "symptom": "A carried warning about QSA output corruption conflicts with a longer published context setting.", "diagnosis": "The author inspected the image for the successor attention path and chose a conservative launcher cap; the planned long-context retrieval ladder is explicitly still unperformed in this document.", "remediation": { "state": "guard_applied_correctness_ladder_unperformed", "proposal": "Pin and validate a matching successor build before claiming longer correct context. The document names pr36845.diff; its merged/fixed status is not established here.", "verified_fix_commit": null, "named_successor_artifact": "pr36845.diff" }, "unresolved": [ "120000 is a launcher guard, not a measured maximum safe context.", "No completed ladder result, corruption threshold or verified upstream fix is available in this snapshot.", "A retrieval check is useful but does not prove correctness for all tasks." ], "observations": [ { "source_document_id": "spark-docs-qwen-qsa", "source_document_sha256": "b642874472f6727e05dc259ab38ff6bc7d78a80691436c65a9c34417abc201e8", "source_line": 3, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/qwen-qsa.md#L3" }, { "source_document_id": "spark-docs-qwen-qsa", "source_document_sha256": "b642874472f6727e05dc259ab38ff6bc7d78a80691436c65a9c34417abc201e8", "source_line": 76, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/qwen-qsa.md#L76" } ] }, { "id": "glm-gb10-backend-shared-memory", "title": "Separate architecture rejection from per-block shared-memory failure", "evidence_status": "source_reproduced_remediation", "conditions": { "hardware": "2 x DGX Spark GB10 sm_121", "node_count": 2, "memory_per_node_gib": 121, "physical_interconnect_gbps": 200, "parallelism": "TP2", "engine": "SGLang", "image_digest": null, "runtime_execution_revision": null, "temperature": null, "top_p": null, "site_reproduced": false, "model_artifact": "LibertAIDAI/GLM-5.3-Flash-NVFP4", "checkpoint_revision": "aa28e1f54130286c95fee10d0705c74ce8743734", "weight_precision": "NVFP4", "image_tag": "sglang-glm53:gb10-tilelang", "executed_engine_version": null, "experiment_date": "2026-09-04", "kv_dtype": "bfloat16 on working tilelang path; early rejected attempts used fp8_e4m3", "kv_pool_tokens": 800448, "reported_prompt_tokens": 26535, "concurrency": 8, "output_tokens": null, "speculative_decoding": "off in the early success; later tuned DFlash verify is a separate follow-up", "reported_transport": "NCCL over RoCE", "stock_dynamic_shared_memory_bytes_per_block": 169984, "device_shared_memory_limit_bytes_per_block": 101376, "working_tuning": { "block_I": 32, "num_stages": 1, "threads": 128 } }, "symptom": "An architecture allowlist rejection, an sm121 unsupported-architecture failure, and a tilelang dynamic shared-memory failure occur in different attempted paths.", "diagnosis": "The first two are backend/build compatibility failures. The third exceeds the GPU per-block shared-memory limit; this is separate from unified system RAM and is not proof that long context alone caused an out-of-memory condition.", "remediation": { "state": "source_verified_for_named_tuned_image", "proposal": "The source selects bfloat16 KV and tuned tilelang. A pinned community build patch applies the smaller v1 kernel configuration when the device limit is below the stock request.", "verified_upstream_fix_commit": null, "community_patch": { "source_document_id": "spark-docker-patch-tilelang-gb10", "source_document_sha256": "88455f3811883262abc5ea292ca5db0dda91fd8fd6f6db7d12ce6d6a01b817ff", "source_line": 45, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docker/patch-tilelang-gb10.py#L45" } }, "follow_up": { "state": "source_verified_separate_follow_up", "observation": "The later speculation document reports clean DFlash draft-verify graph capture with eight tokens per request for batch sizes one through eight on the already tuned image; do not carry forward the early untested warning as the final state.", "conditions": { "image_tag": "sglang-glm53:gb10-tilelang", "kv_dtype": "bfloat16", "draft_tokens": 8, "graph_capture_batch_sizes_min": 1, "graph_capture_batch_sizes_max": 8, "earlier_kv_pool_tokens_not_reused_as_final_profile": true }, "source_document_id": "spark-docs-speculation", "source_document_sha256": "46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f", "source_line": 248, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L248" }, "unresolved": [ "No immutable image digest or executed engine version is supplied for the early success.", "This community patch is not evidence of an upstream release fix." ], "observations": [ { "source_document_id": "spark-docs-glm-dsa", "source_document_sha256": "a4b9448f5265f687504130bc1791ec3e7c1777d16c648851a94c3a3b4db99e32", "source_line": 95, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/glm-dsa.md#L95" }, { "source_document_id": "spark-docs-glm-dsa", "source_document_sha256": "a4b9448f5265f687504130bc1791ec3e7c1777d16c648851a94c3a3b4db99e32", "source_line": 92, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/glm-dsa.md#L92" } ] }, { "id": "container-rdma-hidden", "title": "A fast host link does not prove container RoCE", "evidence_status": "source_reproduced_transport_incident", "conditions": { "hardware": "2 x DGX Spark GB10 sm_121", "node_count": 2, "memory_per_node_gib": 121, "physical_interconnect_gbps": 200, "parallelism": "TP2", "engine": "SGLang", "image_digest": null, "runtime_execution_revision": null, "temperature": null, "top_p": null, "site_reproduced": false, "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "checkpoint_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594", "weight_precision": "NVFP4", "image_tag": "sglang-qwen38fn:sm121-ours", "experiment_date": "2026-09-04", "executed_engine_version": null, "kv_dtype": null, "kv_pool_tokens": null, "concurrency": [ 1, 6 ], "reported_prompt_range_tokens": "74 to approximately 30300 across the source sweep", "output_tokens": null }, "symptom": "The host fabric works and container sysfs lists HCAs, but verbs devices are absent and NCCL uses sockets.", "diagnosis": "Visible /sys/class/infiniband entries do not establish access to /dev/infiniband. Host bandwidth is not an observation of the running container transport.", "remediation": { "state": "source_verified_transport_change", "proposal": "The source maps the required verbs devices, sets IPC_LOCK/memlock and selects the connected HCA; actual NCCL channel logs then show NET/IB. Recheck transport after the service starts.", "verified_fix_commit": null }, "unresolved": [ "No speed ratio from the earlier sweep is promoted to a current benchmark here.", "This incident does not explain every remaining source latency." ], "observations": [ { "source_document_id": "spark-docs-interconnect", "source_document_sha256": "9e7904db6d48fc44691c9f170c03f402c77480dcbf54f66bc107ef57ef0ea94f", "source_line": 17, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md#L17" }, { "source_document_id": "spark-docs-interconnect", "source_document_sha256": "9e7904db6d48fc44691c9f170c03f402c77480dcbf54f66bc107ef57ef0ea94f", "source_line": 69, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md#L69" } ] } ] } ``` --- # DeepSeek 1017 on Two Sparks: Read the TensorFold Evidence There is now a community report that explicitly names DeepSeek-V4.1-Flash-1017 on a pair of DGX Sparks. Its useful contribution is a traceable model view, saved results and documented failure boundaries. Treat it as evidence for those particular configurations. It does not establish an engine-wide speed advantage or general model quality. This note reads [Elevatormusic’s pinned recipe](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/README.md), including the October 17 comparison and the separate October 18 follow-up. Run AI Here has not run this model or reproduced the author’s results. ![Selected 1017 source decode observations and separate final-setup prefill medians with five original row values.](/assets/deepseek-1017-source-conditions.webp) Figure: drawn directly from selected source values. The two decode arms and the later prefill cohort are separate. Prefill whiskers show an observed minimum and maximum, not a confidence interval. ## First verify which 1017 files were served The author combines `LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw` at `afc7aa0aab98ee735c0f54e0a122019af1ac08f1` with two Engram shards from `deepseek-ai/DeepSeek-V4.1-Flash-1017` at `1bae0e1088993b9b82de08f59de5be5a34e42170`. This identifies an EXL3 conversion and its separate base-model tables. It is not a measurement of the unconverted official checkpoint. [Model-view provenance](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/PROVENANCE.md#L7). The pinned limits document says that the measured conversion revision was already behind a newer card revision. The author did not compare them. Keep the measured revision when reusing a number; a current download is not automatically the same artifact. ## Keep one named build beside each decode value These selected observations use one request, thinking off and two prompts per workload. The suite streams with `max_tokens=700`, temperature 1.0 and top-p 1.0. Actual output lengths and exact prompt-token counts are not supplied in the selected result rows. Decode excludes the first-token wait: `(completion_tokens − 1) / (stream end − first content-bearing chunk)`. | October 17 arm | Prose decode, tokens/s | Code decode, tokens/s | |---|---:|---:| | vLLM kit, run 01 | 27.0 | 39.9 | | TensorFold, run 11 | 56.9 | 91.8 | Each cell is the source’s rounded mean of two prompt-level rates within one run. It has no supplied repeated-run uncertainty. [Original vLLM result](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json), [original TensorFold result](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json). The report’s headline median combines six different TensorFold builds. It is not six repetitions of this run-11 configuration. The selected values above deliberately identify one build instead of hiding that distinction. ## The comparison changes more than the engine Both arms use the same reported model view and Spark pair, but their serving settings differ. The vLLM kit uses an exllamav3 overlay, three draft tokens, one configured sequence, a 262,144-token cap and 1,536-token batching. TensorFold run 11 uses Python 0.6.6 at `fd2365042f56fcf726cf4071132c8b779ad2eae0`, five drafts, capacity for four requests, a 1,048,576-token cap and 2,048-token prefill chunks. The source also records a template patch, with its per-run use inferred from file times. [Comparison conditions](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/BENCHMARKS.md#L32). The baseline image digest is retained in JSON. The TensorFold executed image digest is unknown. The recipe now builds a later `92aa710b533d` revision prefix. Do not attach the October 17 decode cells to that later build or describe this as an isolated engine-only experiment. No multi-user vLLM comparison was measured. ## The newer long-prompt cohort answers a different question The October 18 final setup uses random-word prompts and records zero cached tokens. Five source rows give: | Prompt bucket | Median prefill, tokens/s | Observed range | |---|---:|---:| | 32K | 2,165 | 2,065–2,201 | | 128K | 1,961 | 1,851–1,986 | The denominator is prompt tokens divided by TTFT. These runs include a defined warm-up sequence, Engram page handling and host memory controls. They are separate from the older suite’s repeated-sentence prompts, where prefix reuse could not be ruled out. Each original row remains in the export. [Final setup and five rows](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L5). For deployment planning, inspect per-node disk, runtime cache and memory headroom separately. The author requires a substantial local model view and additional first-start cache. Host tuning affects the whole machine: read the original settings and their measured scope before adopting them. This article does not execute or prescribe them for your cluster. ## What the tests still leave open The author reports two near-million-token needle tests, with about 1.7 GiB available on the head at the memory low point. A retrieved fact does not validate general long-context reasoning, near-limit concurrent sessions or decode speed. Image input was not tested. Neither the task suite nor the weights’ separate perplexity claim establishes this engine’s general accuracy. Some slower states remain unexplained. [Known limits](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/KNOWN-LIMITS.md#L6). Use this recipe as a concrete reproduction candidate. Preserve the model view, exact arm, template, draft depth, cache state and request shape before judging a local result. The [JSON source record](/community/deepseek-1017-tensorfold-spark/index.json) and adjacent article exports keep those fields with the six selected measurements. ## Original sources and conditions - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/README.md - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: 546a0c3bfbbb6e84226ec812cf38927f6d993e53b73633a540c426eea5f13d29 - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/BENCHMARKS.md - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: b2d05a077ae8145a2cf2a98ac2ae2eabef87b992230c32398a60e0a2e9174861 - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/KNOWN-LIMITS.md - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: eeb6bc6f7a3fce20654f977581e9718b124e70fc01ab358efebefe9afd9dc942 - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/PROVENANCE.md - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: aa020bd6aa0fe840c9518c96dbc00f04bbdf6add834f2fe7bac2b076aa91a089 - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: 0cb7c61b5b321ba0b9eec76460e4d9d3d0e759bd09cfb03fc747ef66e7c2ea63 - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: d0ec8c99b0208ac4ddc272379d3cd71c796b5662d09481d6ba1e98fbbf0fa41c - deepseek-ai: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md - Document revision: 1bae0e1088993b9b82de08f59de5be5a34e42170 - SHA-256: b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "vllm-kit-run01 prose", "metric": "decode_rate", "value": 27.0, "unit": "tokens/s", "statistic": "one run; mean of two prompt-level decode rates; rounded by source", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "vllm-kit-run01", "engine": "vLLM + exllamav3 overlay", "engine_version": null, "runtime_execution_revision": null, "image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "kit_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "draft_tokens": 3, "configured_max_requests": 1, "context_limit_tokens": 262144, "prefill_chunk_tokens": 1536, "chat_template": "kit template; actual CHAT_TEMPLATE override not read by the source", "workload": "prose", "per_prompt_decode_rates": [ 24.7, 29.3 ], "per_prompt_TTFT_seconds": [ 0.32, 0.28 ], "prompt_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json#L6", "source_document_sha256": "0cb7c61b5b321ba0b9eec76460e4d9d3d0e759bd09cfb03fc747ef66e7c2ea63", "source_document_id": "ds1017-runs-2026-10-17-01_vllm-miaai-kit_suite-results_32of32.json", "source_line": 6, "source_record_id": "deepseek-1017-tensorfold-spark" }, { "label": "vllm-kit-run01 code", "metric": "decode_rate", "value": 39.9, "unit": "tokens/s", "statistic": "one run; mean of two prompt-level decode rates; rounded by source", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "vllm-kit-run01", "engine": "vLLM + exllamav3 overlay", "engine_version": null, "runtime_execution_revision": null, "image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "kit_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "draft_tokens": 3, "configured_max_requests": 1, "context_limit_tokens": 262144, "prefill_chunk_tokens": 1536, "chat_template": "kit template; actual CHAT_TEMPLATE override not read by the source", "workload": "code", "per_prompt_decode_rates": [ 40.4, 39.4 ], "per_prompt_TTFT_seconds": [ 0.47, 0.32 ], "prompt_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json#L20", "source_document_sha256": "0cb7c61b5b321ba0b9eec76460e4d9d3d0e759bd09cfb03fc747ef66e7c2ea63", "source_document_id": "ds1017-runs-2026-10-17-01_vllm-miaai-kit_suite-results_32of32.json", "source_line": 20, "source_record_id": "deepseek-1017-tensorfold-spark" }, { "label": "tensorfold-run11 prose", "metric": "decode_rate", "value": 56.9, "unit": "tokens/s", "statistic": "one run; mean of two prompt-level decode rates; rounded by source", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "tensorfold-run11", "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": "fd2365042f56fcf726cf4071132c8b779ad2eae0", "image_tag": "tensorfold-dsv41:fd2365042f56", "image_digest": null, "draft_tokens": 5, "configured_max_requests": 4, "context_limit_tokens": 1048576, "prefill_chunk_tokens": 2048, "chat_template": "patched kit template; source infers per-run use from file times", "workload": "prose", "per_prompt_decode_rates": [ 53.4, 60.4 ], "per_prompt_TTFT_seconds": [ 0.12, 0.12 ], "prompt_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json#L6", "source_document_sha256": "d0ec8c99b0208ac4ddc272379d3cd71c796b5662d09481d6ba1e98fbbf0fa41c", "source_document_id": "ds1017-runs-2026-10-17-11_tensorfold-fd2365042f56_suite-results_32of32.json", "source_line": 6, "source_record_id": "deepseek-1017-tensorfold-spark" }, { "label": "tensorfold-run11 code", "metric": "decode_rate", "value": 91.8, "unit": "tokens/s", "statistic": "one run; mean of two prompt-level decode rates; rounded by source", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "tensorfold-run11", "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": "fd2365042f56fcf726cf4071132c8b779ad2eae0", "image_tag": "tensorfold-dsv41:fd2365042f56", "image_digest": null, "draft_tokens": 5, "configured_max_requests": 4, "context_limit_tokens": 1048576, "prefill_chunk_tokens": 2048, "chat_template": "patched kit template; source infers per-run use from file times", "workload": "code", "per_prompt_decode_rates": [ 91.1, 92.5 ], "per_prompt_TTFT_seconds": [ 0.13, 0.16 ], "prompt_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json#L20", "source_document_sha256": "d0ec8c99b0208ac4ddc272379d3cd71c796b5662d09481d6ba1e98fbbf0fa41c", "source_document_id": "ds1017-runs-2026-10-17-11_tensorfold-fd2365042f56_suite-results_32of32.json", "source_line": 20, "source_record_id": "deepseek-1017-tensorfold-spark" }, { "label": "TensorFold final setup 32K prefill", "metric": "prefill_rate", "value": 2165, "unit": "tokens/s", "statistic": "median of five source rows; individually recoverable in per_run_values", "uncertainty": { "observed_min": 2065, "observed_max": 2201, "interval_kind": "observed range, not confidence interval" }, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-18", "concurrency": 1, "thinking": false, "temperature": null, "top_p": null, "max_output_tokens": null, "output_tokens": null, "input_tokens": null, "prefix_cache": false, "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "prompt_tokens / TTFT; source final setup and host tuning", "repetitions": 5, "checkpoint_is_latest_revision_verified": false, "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": null, "runtime_revision_prefix": "92aa710b533d", "image_tag": "tensorfold-dsv41:92aa710b533d", "image_digest": null, "reported_prompt_bucket": "32K", "cached_tokens": 0, "per_run_values": [ 2201, 2180, 2065, 2165, 2100 ], "run_ids": [ "hw5", "hw6", "ht1", "ht2", "ht3" ], "draft_tokens": 5, "context_limit_tokens": 1048576, "warmup": "one 8K prompt; subsequent 8K requests before 32K/64K/128K; random-word prompts", "host_tuning_scope": "author applies no engine swap and memory-reclaim/compaction controls; not a universal host-tuning recommendation" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L16", "source_document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "source_document_id": "ds1017-runs-2026-10-18-INDEX.md", "source_line": 16, "source_record_id": "deepseek-1017-tensorfold-spark" }, { "label": "TensorFold final setup 128K prefill", "metric": "prefill_rate", "value": 1961, "unit": "tokens/s", "statistic": "median of five source rows; individually recoverable in per_run_values", "uncertainty": { "observed_min": 1851, "observed_max": 1986, "interval_kind": "observed range, not confidence interval" }, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-18", "concurrency": 1, "thinking": false, "temperature": null, "top_p": null, "max_output_tokens": null, "output_tokens": null, "input_tokens": null, "prefix_cache": false, "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "prompt_tokens / TTFT; source final setup and host tuning", "repetitions": 5, "checkpoint_is_latest_revision_verified": false, "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": null, "runtime_revision_prefix": "92aa710b533d", "image_tag": "tensorfold-dsv41:92aa710b533d", "image_digest": null, "reported_prompt_bucket": "128K", "cached_tokens": 0, "per_run_values": [ 1986, 1981, 1914, 1961, 1851 ], "run_ids": [ "hw5", "hw6", "ht1", "ht2", "ht3" ], "draft_tokens": 5, "context_limit_tokens": 1048576, "warmup": "one 8K prompt; subsequent 8K requests before 32K/64K/128K; random-word prompts", "host_tuning_scope": "author applies no engine swap and memory-reclaim/compaction controls; not a universal host-tuning recommendation" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L16", "source_document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "source_document_id": "ds1017-runs-2026-10-18-INDEX.md", "source_line": 16, "source_record_id": "deepseek-1017-tensorfold-spark" } ], "conditions": { "source_reported": true, "independent_site_runs": 0, "same_pair_and_model_view_declared": true, "controlled_engine_only_AB": false, "old_six_build_median_is_not_repetition": true, "current_recipe_is_not_run11": true, "vision_tested": false, "perplexity_or_general_accuracy_tested": false }, "deployment_profiles": [ { "id": "vllm-kit-run01", "engine": "vLLM + exllamav3 overlay", "engine_version": null, "runtime_execution_revision": null, "image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "kit_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "draft_tokens": 3, "configured_max_requests": 1, "context_limit_tokens": 262144, "prefill_chunk_tokens": 1536, "chat_template": "kit template; actual CHAT_TEMPLATE override not read by the source" }, { "id": "tensorfold-run11", "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": "fd2365042f56fcf726cf4071132c8b779ad2eae0", "image_tag": "tensorfold-dsv41:fd2365042f56", "image_digest": null, "draft_tokens": 5, "configured_max_requests": 4, "context_limit_tokens": 1048576, "prefill_chunk_tokens": 2048, "chat_template": "patched kit template; source infers per-run use from file times" } ] } ``` --- # Qwen3.8-27B Chisel: Deployment, Speed and the Quality Gate The Chisel community recipe is a concrete Qwen3.8-27B deployment candidate for one DGX Spark. Its value is the pinned serving recipe and separate speed and quality records. Read the workload and quality gate before repeating a headline rate: the published results use different prompt shapes, cache settings and thinking modes. This note reads [yuanlin2004’s replay recipe](https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md), checked on 18 October 2026. The underlying local cohort is filed under October 15; some sessions cross UTC midnight. Run AI Here has not installed the recipe or run the model. ![Source Qwen quality differences and reported normal-approximation 95 percent intervals, with all four non-inferiority flags false.](/assets/qwen38-chisel-quality-source.webp) Figure: calculated positions use the source comparison JSON directly. Intervals describe paired task-score differences, not speed variation. Passing the author’s custom bar does not establish non-inferiority. ## Identify the target, drafter and actual image separately The target is `unsloth/Qwen3.8-27B-NVFP4`, reported with revision prefix `f0b7c9e7`. The full executed weight revision is not provided. The drafter is `yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter`, with its weight-file SHA-256 retained in the export. Ten draft tokens, bf16 GDN state and `chisel-passes 0.1.0` are part of the configuration. GDN state dtype does not fill an unreported attention-KV dtype. The environment names vLLM `0.31.1rc1.dev380`, driver `580.178.04` and a 2,000 MHz GPU cap. It explicitly says the main tests used a local build with the plugin preinstalled, while the published recipe installs it at startup. The base image has a digest; the saved `server.image` contains an empty array. Keep the executed local image digest unknown. [Environment](https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/ENVIRONMENT.md). The YAML configures TP1, memory utilization 0.60, a 262,144-token cap and up to four sequences. Those settings do not prove four simultaneous near-limit requests fit. The plugins change recurrent-state handling and the draft output head; the target verification path retains its full vocabulary according to the author. Source code for these recipe plugins is public, while the broader Chisel project is described as not yet public. [Pinned serving recipe](https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.yaml). ## Four numbers belong to two speed cohorts | Local test | Reported observation | Keep with the value | |---|---:|---| | Mixed prompts | 15.35 ms/token median decode latency | One user; thinking on, medium; at most 256 output tokens | | Same mixed prompts | 241 ms median TTFT | First-token waiting, separate from decode | | llama-benchy at 16K context | 30.0 tokens/s, one user | 2,048-token prompt stage, 128 output tokens, prefix cache on | | Same llama-benchy shape | 58.0 tokens/s total, two users | Aggregate across both users, not a per-user rate | The mixed test has 25 prompts: nine chat, nine code, six JSON-schema and one long story. Two fresh-start sessions each repeat that set three times. Decode latency excludes the first-token wait. The llama-benchy cohort uses version 0.4.0 and five runs per session. Its thinking override is not explicitly supplied in the method summary and remains unknown in the extraction. These are local runs of an arena-shaped benchmark, separate from the hosted arena entry. [Method](https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/METHOD.md). A median latency is not whole-request throughput. Nor can a two-user total be compared directly with the one-user latency. Use the [timing guide](/guides/llm-speed-metrics/) when deciding which measure fits an application. Avoid a “fastest” verdict across these conditions. ## Read the quality result beyond the pass flag The source compares against `Qwen/Qwen3.8-27B` BF16 at `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`, with bf16 KV. Quality evaluation uses lm-evaluation-harness 0.4.13, chat templates, greedy decoding and thinking off. The BF16 cohort is filed under October 9; the comparison JSON retains older original candidate paths. Candidate image, clock and plugin settings must not be silently assigned to the reference arm. | Task and sample count | BF16 score | NVFP4 replay score | Difference, points [reported 95% CI] | |---|---:|---:|---| | GSM8K 5-shot, 1,319 | 90.07 | 89.46 | −0.61 [−1.81, +0.60] | | IFEval strict, 541 | 80.96 | 81.89 | +0.92 [−1.56, +3.41] | | HumanEval chat pass@1, 164 | 95.12 | 94.51 | −0.61 [−2.68, +1.46] | | NarrativeQA F1, 200 | 34.83 | 35.24 | +0.40 [−1.64, +2.45] | The custom bar requires at least 97% recovery, a permitted answer-length ratio and no task flagged clearly worse. It passes. A stricter flag requires the lower interval bound to exceed a loss of one point; all four `non_inferior` flags are false. Both results must survive an AI summary. [Exact comparison JSON](https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json). The [comparison code](https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/tools/compare_quality.py#L68) computes paired item differences and a normal-approximation interval, mean ± 1.96 standard errors. These are not bootstrap intervals. Intervals crossing zero do not prove equivalence, and scores on four tasks do not guarantee general quality preservation. ## A useful local acceptance plan Before adopting the recipe, identify the quantized artifact, drafter, plugins and executed image. Check the required API features and memory allocation at your actual context and concurrency. Then evaluate both timing and correctness on representative prompts. Keep speed thinking-on results separate from thinking-off quality tests. The [source record](/community/qwen38-chisel-dflash-replay/) and adjacent JSON/Markdown retain the twelve selected observations, four paired comparisons and missing identifiers. ## Original sources and conditions - yuanlin2004: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md - Document revision: 84feafddff1e834bd16b849c827eed11676e5934 - SHA-256: 5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321 - yuanlin2004: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.yaml - Document revision: 84feafddff1e834bd16b849c827eed11676e5934 - SHA-256: 028f2badf76407546be67b8bd93e2e81aae8ad5d00076e20b8bd8835021a1ede - yuanlin2004: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/ENVIRONMENT.md - Document revision: 84feafddff1e834bd16b849c827eed11676e5934 - SHA-256: 349c8de9b51a1df3ebba677db79d4e62c9a652a6dc225965f7515b80583d471a - yuanlin2004: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/METHOD.md - Document revision: 84feafddff1e834bd16b849c827eed11676e5934 - SHA-256: 072fb07c8e46bb6e1e7a4c8a9416b94a87863cfc60dfde60f127e9243363dd8e - yuanlin2004: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/server.image - Document revision: 84feafddff1e834bd16b849c827eed11676e5934 - SHA-256: 37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 - yuanlin2004: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json - Document revision: 84feafddff1e834bd16b849c827eed11676e5934 - SHA-256: 46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb - yuanlin2004: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/tools/compare_quality.py - Document revision: 84feafddff1e834bd16b849c827eed11676e5934 - SHA-256: ee52180fce64fa6948d7516ac42f7f5fd0ef143584d2c284d6aa19f6fdecb783 - Qwen: https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md - Document revision: 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 - SHA-256: 57e4bdb258ee1a7d2635c5174ebd4e56abe392505cdb5f8bbb356b0dc4293641 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "Chisel replay mixed-prompt median decode latency", "metric": "decode_latency", "value": 15.35, "unit": "ms/token", "statistic": "source-reported median over two fresh-start sessions", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "thinking": true, "reasoning_effort": "medium", "greedy": true, "temperature": 0, "top_p": null, "concurrency": 1, "input_tokens": null, "max_output_tokens": 256, "output_tokens": null, "prompt_count": 25, "prompt_mix": { "chat": 9, "code": 9, "JSON_schema": 6, "long_story": 1 }, "repeats_per_session": 3, "fresh_start_sessions": 2, "measurement_denominator": "(last streamed token time - first) / (output tokens - 1); median latency, not whole-request served rate" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L9", "source_document_sha256": "5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321", "source_document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.md", "source_line": 9, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "Chisel replay mixed-prompt median TTFT", "metric": "ttft", "value": 241, "unit": "ms", "statistic": "source-reported median on the same prompt cohort", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "thinking": true, "reasoning_effort": "medium", "greedy": true, "temperature": 0, "top_p": null, "concurrency": 1, "input_tokens": null, "max_output_tokens": 256, "output_tokens": null, "prompt_count": 25, "prompt_mix": { "chat": 9, "code": 9, "JSON_schema": 6, "long_story": 1 }, "repeats_per_session": 3, "fresh_start_sessions": 2, "measurement_denominator": "request start to first streamed token" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L10", "source_document_sha256": "5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321", "source_document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.md", "source_line": 10, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "Local llama-benchy 16K C1", "metric": "generation_throughput", "value": 30.0, "unit": "tokens/s", "statistic": "source-reported summary; separate from mixed-prompt median latency", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "benchmark": "llama-benchy 0.4.0, local execution of spark-arena-v2 shape; not arena scores", "concurrency": 1, "input_tokens": 2048, "retained_context_tokens": 16384, "output_tokens": 128, "prefix_cache": true, "repeats_per_session": 5, "fresh_start_sessions": 2, "thinking": null, "measurement_denominator": "source local llama-benchy generation throughput; total across users for C2" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L11", "source_document_sha256": "5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321", "source_document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.md", "source_line": 11, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "Local llama-benchy 16K C2", "metric": "generation_throughput", "value": 58.0, "unit": "tokens/s", "statistic": "source-reported summary; separate from mixed-prompt median latency", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "benchmark": "llama-benchy 0.4.0, local execution of spark-arena-v2 shape; not arena scores", "concurrency": 2, "input_tokens": 2048, "retained_context_tokens": 16384, "output_tokens": 128, "prefix_cache": true, "repeats_per_session": 5, "fresh_start_sessions": 2, "thinking": null, "measurement_denominator": "source local llama-benchy generation throughput; total across users for C2" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L11", "source_document_sha256": "5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321", "source_document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.md", "source_line": 11, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "gsm8k BF16 reference", "metric": "task_score", "value": 90.06823351023503, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "Qwen/Qwen3.8-27B", "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reported_model_revision_prefix": null, "system": null, "node_count": null, "driver": null, "GPU_clock_cap_MHz": null, "engine": null, "executed_engine_version": null, "runtime_execution_revision": null, "recipe_base_image_digest": null, "executed_local_image_digest": null, "executed_image_boundary": "Reference model and bf16 KV are reported; candidate-specific image, plugin and clock settings are not imputed to this arm.", "plugin": null, "draft_model": null, "draft_weight_sha256": null, "draft_tokens": null, "GDN_state_dtype": null, "KV_dtype": null, "configured_context_limit_tokens": null, "configured_max_requests": null, "GPU_memory_utilization": null, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "BF16 reference", "task": "gsm8k", "n": 1319, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "5-shot accuracy", "paired_difference_normal_CI95_points": [ -1.8137451343135735, 0.6007049523575464 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L8", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 8, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "gsm8k NVFP4 replay", "metric": "task_score", "value": 89.46171341925702, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "NVFP4 replay", "task": "gsm8k", "n": 1319, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "5-shot accuracy", "paired_difference_normal_CI95_points": [ -1.8137451343135735, 0.6007049523575464 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L9", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 9, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "ifeval BF16 reference", "metric": "task_score", "value": 80.96118299445472, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "Qwen/Qwen3.8-27B", "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reported_model_revision_prefix": null, "system": null, "node_count": null, "driver": null, "GPU_clock_cap_MHz": null, "engine": null, "executed_engine_version": null, "runtime_execution_revision": null, "recipe_base_image_digest": null, "executed_local_image_digest": null, "executed_image_boundary": "Reference model and bf16 KV are reported; candidate-specific image, plugin and clock settings are not imputed to this arm.", "plugin": null, "draft_model": null, "draft_weight_sha256": null, "draft_tokens": null, "GDN_state_dtype": null, "KV_dtype": null, "configured_context_limit_tokens": null, "configured_max_requests": null, "GPU_memory_utilization": null, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "BF16 reference", "task": "ifeval", "n": 541, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "prompt-level strict", "paired_difference_normal_CI95_points": [ -1.560611007417151, 3.409039842906985 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L26", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 26, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "ifeval NVFP4 replay", "metric": "task_score", "value": 81.88539741219964, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "NVFP4 replay", "task": "ifeval", "n": 541, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "prompt-level strict", "paired_difference_normal_CI95_points": [ -1.560611007417151, 3.409039842906985 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L27", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 27, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "humaneval_chat BF16 reference", "metric": "task_score", "value": 95.1219512195122, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "Qwen/Qwen3.8-27B", "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reported_model_revision_prefix": null, "system": null, "node_count": null, "driver": null, "GPU_clock_cap_MHz": null, "engine": null, "executed_engine_version": null, "runtime_execution_revision": null, "recipe_base_image_digest": null, "executed_local_image_digest": null, "executed_image_boundary": "Reference model and bf16 KV are reported; candidate-specific image, plugin and clock settings are not imputed to this arm.", "plugin": null, "draft_model": null, "draft_weight_sha256": null, "draft_tokens": null, "GDN_state_dtype": null, "KV_dtype": null, "configured_context_limit_tokens": null, "configured_max_requests": null, "GPU_memory_utilization": null, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "BF16 reference", "task": "humaneval_chat", "n": 164, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "chat pass@1", "paired_difference_normal_CI95_points": [ -2.6839968720184646, 1.4644846768965132 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L44", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 44, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "humaneval_chat NVFP4 replay", "metric": "task_score", "value": 94.51219512195121, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "NVFP4 replay", "task": "humaneval_chat", "n": 164, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "chat pass@1", "paired_difference_normal_CI95_points": [ -2.6839968720184646, 1.4644846768965132 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L45", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 45, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "longbench_narrativeqa BF16 reference", "metric": "task_score", "value": 34.83234254995091, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "Qwen/Qwen3.8-27B", "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reported_model_revision_prefix": null, "system": null, "node_count": null, "driver": null, "GPU_clock_cap_MHz": null, "engine": null, "executed_engine_version": null, "runtime_execution_revision": null, "recipe_base_image_digest": null, "executed_local_image_digest": null, "executed_image_boundary": "Reference model and bf16 KV are reported; candidate-specific image, plugin and clock settings are not imputed to this arm.", "plugin": null, "draft_model": null, "draft_weight_sha256": null, "draft_tokens": null, "GDN_state_dtype": null, "KV_dtype": null, "configured_context_limit_tokens": null, "configured_max_requests": null, "GPU_memory_utilization": null, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "BF16 reference", "task": "longbench_narrativeqa", "n": 200, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "F1, full stories", "paired_difference_normal_CI95_points": [ -1.6421295670363527, 2.4483306144283086 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L62", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 62, "source_record_id": "qwen38-chisel-dflash-replay" }, { "label": "longbench_narrativeqa NVFP4 replay", "metric": "task_score", "value": 35.235443073646884, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "NVFP4 replay", "task": "longbench_narrativeqa", "n": 200, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "F1, full stories", "paired_difference_normal_CI95_points": [ -1.6421295670363527, 2.4483306144283086 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L63", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 63, "source_record_id": "qwen38-chisel-dflash-replay" } ], "conditions": { "source_reported": true, "independent_site_runs": 0, "speed_thinking": true, "quality_thinking": false, "arena_and_local_cohorts_merged": false, "source_all_non_inferior": false, "source_custom_bar_passed": true, "current_recipe_is_complete_executed_image_proof": false }, "quality_comparisons": [ { "task": "gsm8k", "n": 1319, "reference_score": 90.06823351023503, "candidate_score": 89.46171341925702, "diff_points": -0.6065200909780136, "ci95_points": [ -1.8137451343135735, 0.6007049523575464 ], "non_inferior": false, "length_ratio": 1.0086485431409682, "flips_right_to_wrong": 37, "flips_wrong_to_right": 29, "recovery": 0.9932659932659933, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L6", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 6 }, { "task": "ifeval", "n": 541, "reference_score": 80.96118299445472, "candidate_score": 81.88539741219964, "diff_points": 0.9242144177449169, "ci95_points": [ -1.560611007417151, 3.409039842906985 ], "non_inferior": false, "length_ratio": 1.001384198457369, "flips_right_to_wrong": 21, "flips_wrong_to_right": 26, "recovery": 1.0114155251141552, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L24", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 24 }, { "task": "humaneval_chat", "n": 164, "reference_score": 95.1219512195122, "candidate_score": 94.51219512195121, "diff_points": -0.6097560975609756, "ci95_points": [ -2.6839968720184646, 1.4644846768965132 ], "non_inferior": false, "length_ratio": 1.0066489055410106, "flips_right_to_wrong": 2, "flips_wrong_to_right": 1, "recovery": 0.9935897435897434, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L42", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 42 }, { "task": "longbench_narrativeqa", "n": 200, "reference_score": 34.83234254995091, "candidate_score": 35.235443073646884, "diff_points": 0.403100523695978, "ci95_points": [ -1.6421295670363527, 2.4483306144283086 ], "non_inferior": false, "length_ratio": 0.9618899475014583, "recovery": 1.011572593003698, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L60", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 60 } ] } ``` --- # Qwen Flash-Next: Quantization, Memory and Long Context A smaller quantized file, a lower RSS reading and a longer configured window describe different capabilities. For Qwen3.8-Flash-Next, the useful evidence here comes from two separate deployments: a one-Spark NVFP4/MXFP8 recipe with measured cold-prefix latency, and an RTX PRO 6000 experiment that changes how the n-gram PLE table is stored. Read their hardware, allocations and request shapes before estimating whether a setup fits. ![One-Spark cold-prefix first-token latency at 16K, 64K and 128K, beside a separate PLE host-RSS comparison on RTX PRO 6000.](/assets/qwen-flash-next-memory-context-source.webp) *The left panel shows original means and between-boot SD. The right panel retains source-labelled GB and distinguishes reclaimable mapped pages. The panels describe different machines; neither is whole-model peak RAM.* ## Start with the complete single-Spark artifact The [one-Spark recipe](https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/README.md) serves `ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE` at `03f4a0570496bbe741189f4a2a7d611679e85406`. It also reads an MXFP8 prefill shard from `local-inference-lab/Qwen3.8-Flash-Next-NVFP4` at `7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd`. Record both identities. “NVFP4” alone omits the dense, attention and prefill paths, as well as the refitted four-draft MTP head. The selected experiment is the v2.4.0 promotion arm on one 128 GB unified-memory DGX Spark, using vLLM and b12x. Its saved [experimental recipe](https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/arm-p1/recipe.yaml) fixes the checkpoint and image tag. The [build report](https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/BENCHMARKS.md) describes the subsequent release image with an added compiled-cache layer. That release digest is useful provenance, but it is not silently substituted for an unreported digest of the exact cold-test image. Engine commit prefixes likewise remain prefixes when the complete executed revision is unavailable. ## Read the actual 16K–128K cold-prefix workload The [source capability CSV](https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv) selects C1, temperature zero and one requested output token. Each value averages four boots with three cold prompts per boot. The uncertainty is SD between boot means, not a confidence interval or the spread of every individual request. | Actual input tokens | First-token latency, mean ± SD | Prefill, mean ± SD | |---:|---:|---:| | 16,384 | 7.104 ± 0.016 s | 2,306.4 ± 5.2 tokens/s | | 65,536 | 30.723 ± 0.060 s | 2,133.2 ± 4.2 tokens/s | | 131,072 | 70.000 ± 0.137 s | 1,872.5 ± 3.7 tokens/s | These observations describe the cost of reading a fresh prefix. They do not measure a long answer after that prefix. “Cold prefix” excludes a matching prompt-cache hit; it does not say that weights, filesystem pages or compiled kernels were cold. The selected fields do not supply every request's actual completion count or all sampling parameters, so those gaps stay visible in JSON. The author reports a separate retrieval gate through 128K and explicitly leaves larger one-Spark depths unmeasured in this snapshot. Keep a retrieval gate, this latency curve and the configured window as separate evidence. None proves general answer quality over the whole configured window. ## Keep the KV pool separate from the request ceiling The experimental recipe sets FP8 KV and an explicit 15,032,385,536-byte pool, or 14 GiB. The build report records an allocated capacity of 993,754 tokens. The per-request ceiling is 262,144 tokens and the scheduler accepts up to eight request slots. Those three values are not a claim that eight full windows fit at once. Admission also depends on shared pool use, state and request scheduling. A token-capacity figure is not measured peak process memory. The recipe's `gpu_memory_utilization=0.01` is a startup guard in conjunction with explicit KV bytes, as its own comment explains; it does not mean the model consumes one percent of unified memory. The PLE table is paged from NVMe, introducing another distinct working set. ## Compare PLE memory on its own hardware The [PLE sidecar report](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md) uses one 96 GB RTX PRO 6000 Blackwell, 176 GB host RAM and local NVMe. Its main table uses the mixed NVFP4/FP8 checkpoint, 8K input, 512 requested output tokens and no prefix-cache hits. The reported launch window is 32,768 with 36 request slots. These conditions are not the Spark experiment's conditions. Two selected memory rows make the distinction concrete. The BF16 PLE table is labelled 95.4 GB, with approximately 95 GB host RSS. The INT4 group-16 sidecar is labelled 32 GB, with 32.9 GB host RSS. Its [format metadata](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/ples_int4/META.json) specifies group-16 INT4 with FP16 scales. Mapped RSS contains resident file pages that the operating system can reclaim. It is not a permanent anonymous allocation, and table size is not whole-model memory use. The source's GB labels are preserved; the site does not infer their exact byte convention or silently equate them with GiB. The full checkpoint revision, executed image digest and timing of the RSS sample remain unreported. ## Separate first-request residence from warm throughput An optional experiment drops page cache before each arm and compares INT4 mmap with an anonymous resident copy. The source reports 19.0 tokens/s on the first mmap request and 74.3 on the first resident request without MTP. These are first-request observations from a different cohort, whose exact request shape is not given; the main table's 8K/512 shape is not copied into it. The resident sidecar adds 29.8 GiB of anonymous memory above roughly 8 GiB already held by the worker. This is another allocation, not the INT4 mmap RSS restated in different units. Anonymous memory also has different reclaim and swap behaviour; it should not be described as universally unreclaimable. Within this optional cohort, both residence modes retain the same reported KV allocation for each MTP setting: 426,639 tokens without MTP and 134,192 with three drafts. The observation does not isolate a universal memory formula. The source also reports much closer warm rates and limited quality/sanity checks, so neither a cold-request advantage nor a smaller sidecar establishes general quality equivalence. ## What should accompany a deployment decision? Keep the exact main checkpoint, extra prefill weights, PLE format, drafter and executed runtime together. Name the memory measurement: file bytes, mapped RSS, anonymous allocation, KV capacity or minimum free memory. Record actual input/output length, concurrency, cache state and whether variation is within one boot or between boots. For automated reading, [the two complete source records](/for-ai/) are available through `qwen-flash-next-memory`; the adjacent article JSON retains the same conditions. Missing values remain null. Use the [runtime choice guide](/guides/inference-runtime-choice/) and [KV/speculation explanation](/guides/kv-cache-speculative-decoding/) to frame a follow-up test. Run AI Here has not executed either deployment or independently measured these models. ## Original sources and conditions - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/README.md - Document revision: 348ede7e28a01f6aa6b501aaf2392d20a286a012 - SHA-256: bb618a879867f416c51cfc3fa8b699da427d28ec324b14af7d1279a2501ab37f - primitive-ai: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md - Document revision: a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 - SHA-256: 0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9 - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv - Document revision: 348ede7e28a01f6aa6b501aaf2392d20a286a012 - SHA-256: 4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205 - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/BENCHMARKS.md - Document revision: 348ede7e28a01f6aa6b501aaf2392d20a286a012 - SHA-256: ac2128626b607b2e4571bdb45fb7e2639e5c65c83841a8f28367bb5692986fc0 - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/VERSIONS.md - Document revision: 348ede7e28a01f6aa6b501aaf2392d20a286a012 - SHA-256: c513523883d95b716707697a0633fb9c5e63c9cf748aeaf37a38d4f771913abd - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/arm-p1/recipe.yaml - Document revision: 348ede7e28a01f6aa6b501aaf2392d20a286a012 - SHA-256: 33ecbeea74e85987bd4dfa7418543438e98ff93ea2270c103deccab44a43fa7e - primitive-ai: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/ples_int4/META.json - Document revision: a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 - SHA-256: e9a1863008450b7ac329d50a3b6b908f21122b8bf5da999bdeff05004161884a - primitive-ai: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/ples_fp8/META.json - Document revision: a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 - SHA-256: de757efa51030ac1e4edb6acdb29507521f8ace860a3be511661a26e5017b632 - primitive-ai: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/ples_nvfp4/META.json - Document revision: a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 - SHA-256: 5aa4d53703f67c8c664d674c95f16c29c8ce7a94a2d7d9ac27ae8afe4c0b56ef Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "1x v2.4.0 16384-token ttft", "metric": "ttft", "value": 7.104, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 0.016 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 16384, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1262, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1262", "source_record_id": "qwen38-flash-next-single-spark-context" }, { "label": "1x v2.4.0 16384-token prefill_rate", "metric": "prefill_rate", "value": 2306.4, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 5.2 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 16384, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1263, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1263", "source_record_id": "qwen38-flash-next-single-spark-context" }, { "label": "1x v2.4.0 65536-token ttft", "metric": "ttft", "value": 30.723, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 0.06 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 65536, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1264, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1264", "source_record_id": "qwen38-flash-next-single-spark-context" }, { "label": "1x v2.4.0 65536-token prefill_rate", "metric": "prefill_rate", "value": 2133.2, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 4.2 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 65536, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1265, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1265", "source_record_id": "qwen38-flash-next-single-spark-context" }, { "label": "1x v2.4.0 131072-token ttft", "metric": "ttft", "value": 70.0, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 0.137 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 131072, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1266, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1266", "source_record_id": "qwen38-flash-next-single-spark-context" }, { "label": "1x v2.4.0 131072-token prefill_rate", "metric": "prefill_rate", "value": 1872.5, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 3.7 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 131072, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1267, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1267", "source_record_id": "qwen38-flash-next-single-spark-context" }, { "label": "1x v2.4.0 allocated KV pool", "metric": "kv_pool_tokens", "value": 993754, "unit": "tokens", "statistic": "boot allocation observed on every reported boot", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": null, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": null, "thinking": null, "output_tokens": null, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": null, "memory_kind": "allocated_KV_token_pool_not_measured_peak_RAM", "max_context_simultaneously_validated": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-BENCHMARKS.md", "source_document_sha256": "ac2128626b607b2e4571bdb45fb7e2639e5c65c83841a8f28367bb5692986fc0", "source_line": 106, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/BENCHMARKS.md#L106", "source_record_id": "qwen38-flash-next-single-spark-context" }, { "label": "BF16 in RAM table size", "metric": "artifact_size", "value": 95.4, "unit": "GB", "statistic": "source-reported PLE table size, not whole model or peak RAM", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "BF16 in RAM", "memory_kind": "PLE_artifact_size", "memory_backing": "in RAM" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 96, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L96", "source_record_id": "qwen38-flash-next-ple-memory" }, { "label": "BF16 in RAM host RSS", "metric": "host_rss", "value": 95, "unit": "GB", "statistic": "approximate source host RSS", "uncertainty": { "approximate": true }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "BF16 in RAM", "memory_kind": "host_RSS_not_necessarily_anonymous", "memory_backing": "in RAM" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 96, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L96", "source_record_id": "qwen38-flash-next-ple-memory" }, { "label": "INT4 group-16 mmap table size", "metric": "artifact_size", "value": 32, "unit": "GB", "statistic": "source-reported PLE table size, not whole model or peak RAM", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16 mmap", "memory_kind": "PLE_artifact_size", "memory_backing": "reclaimable mapped file pages" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 98, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L98", "source_record_id": "qwen38-flash-next-ple-memory" }, { "label": "INT4 group-16 mmap host RSS", "metric": "host_rss", "value": 32.9, "unit": "GB", "statistic": "source host RSS including reclaimable file pages", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16 mmap", "memory_kind": "host_RSS_not_necessarily_anonymous", "memory_backing": "reclaimable mapped file pages" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 98, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L98", "source_record_id": "qwen38-flash-next-ple-memory" }, { "label": "mmap cold first request", "metric": "decode_rate", "value": 19.0, "unit": "tokens/s", "statistic": "first-request observation; warm asymptote is a separate quantity", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": 1, "cache_state": "page cache dropped before each arm", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": null, "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "workload": "first request, exact request shape unreported; optional resident-vs-mmap cohort, not main 8K/512 table", "memory_backing": "NVMe mmap" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 68, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L68", "source_record_id": "qwen38-flash-next-ple-memory" }, { "label": "resident cold first request", "metric": "decode_rate", "value": 74.3, "unit": "tokens/s", "statistic": "first-request observation; warm asymptote is a separate quantity", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": 1, "cache_state": "page cache dropped before each arm", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": null, "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "workload": "first request, exact request shape unreported; optional resident-vs-mmap cohort, not main 8K/512 table", "memory_backing": "anonymous resident copy" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 69, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L69", "source_record_id": "qwen38-flash-next-ple-memory" }, { "label": "INT4 resident sidecar anonymous allocation", "metric": "anonymous_memory", "value": 29.8, "unit": "GiB", "statistic": "sidecar allocation only, on top of roughly 8 GiB held by worker", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "anonymous_resident_sidecar_not_total_RSS" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 81, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L81", "source_record_id": "qwen38-flash-next-ple-memory" }, { "label": "optional residence cohort KV pool, MTP 0", "metric": "kv_pool_tokens", "value": 426639, "unit": "tokens", "statistic": "reported identical mmap/resident allocation for this MTP setting", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "allocated_KV_token_pool_not_measured_RAM" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 80, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L80", "source_record_id": "qwen38-flash-next-ple-memory" }, { "label": "optional residence cohort KV pool, MTP 3", "metric": "kv_pool_tokens", "value": 134192, "unit": "tokens", "statistic": "reported identical mmap/resident allocation for this MTP setting", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 3, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "allocated_KV_token_pool_not_measured_RAM" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 80, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L80", "source_record_id": "qwen38-flash-next-ple-memory" } ], "conditions": { "cross_hardware_memory_profiles_merged": false, "native_window_is_measured_full_window": false, "one_Spark_cold_prefill_validated_through_tokens": 131072, "general_quality_equivalence_established": false, "independent_site_runs": 0 }, "deployment_profiles": [ { "id": "qwen-1x-v2.4-cold", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "checkpoint_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "engine": "vLLM + b12x", "image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "context_limit_tokens": 262144, "measured_cold_input_tokens": [ 16384, 65536, 131072 ], "configured_max_requests": 8, "KV_pool_bytes": 15032385536, "KV_pool_tokens": 993754, "MTP_draft_tokens": 4, "PLE_file_reported_GiB": 26.8, "all_eight_full_window_requests_proven": false, "cold_prefix_is_not_cold_page_cache": true } ], "memory_observations": [ { "record_id": "qwen38-flash-next-ple-memory", "kind": "source_main_table_host_RSS", "BF16_in_RAM_GB_approx": 95, "INT4_mmap_GB": 32.9, "mapped_pages_reclaimable": true, "anonymous_memory_inferred_from_RSS": false, "GB_byte_convention_verified": false }, { "record_id": "qwen38-flash-next-ple-memory", "kind": "optional_anonymous_residence", "INT4_sidecar_GiB": 29.8, "additional_worker_GiB_approx": 8, "same_cohort_as_main_table": false } ] } ``` --- # DeepSeek 1017: Quantization, Memory and Context Evidence DeepSeek 1017 has several usable community evidence trails, but they answer different questions. A dual-Spark EXL3 matrix measures short endpoint requests. Two near-million-token runs retrieve one string and return seven tokens. A VQ conversion reports file layout and a loader check while explicitly leaving 1017 serving speed unmeasured. Keep those states separate before choosing a quantization or budgeting memory. ![DeepSeek 1017 short-run EXL3 decode, separate near-million-token needle latency and a VQ artifact-size receipt with unknown runtime performance.](/assets/deepseek-1017-quant-memory-context-source.webp) *The chart preserves three source states. Short decode rates are not near-1M generation rates, and VQ file bytes are not a runtime RAM requirement. No independent site model benchmark was performed.* ## Identify the EXL3 model and matching Engrams The [dual-Spark deployment](https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/README.md) uses `LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw` at `afc7aa0aab98ee735c0f54e0a122019af1ac08f1`. Its [manifest](https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/manifest.json) separately pins official Engram tables to `deepseek-ai/DeepSeek-V4.1-Flash-1017` at `1bae0e1088993b9b82de08f59de5be5a34e42170`. A document commit, converted-weight commit and Engram commit identify different objects. This deployment uses two 128 GB GB10 systems and TP2 over RoCE, with checkpoint reads through a read-only NFS export and rank-local Engram lookup. The TensorFold path is patched Python 0.6.0; the vLLM arm uses an EXL3 overlay. These are concrete integrations, not an instruction that an arbitrary current installation of either engine accepts the files unchanged. ## What do the short-run rates actually measure? The [method](https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/benchmark-method.md) uses deterministic ledger prompts: 520 input tokens with thinking disabled and 546 at reasoning effort 75. Selected C1 runs produce 64 output tokens. Each is one pass; no run-to-run variance estimate is available. | C1 arm | Thinking off | Effort 75 | |---|---:|---:| | TensorFold Python 0.6.0 | 58.1 tokens/s | 77.5 tokens/s | | vLLM with EXL3 overlay | 36.9 tokens/s | 41.4 tokens/s | The table rounds the [TensorFold CSV](https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv) and [vLLM CSV](https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv) to one decimal; JSON retains the original numeric precision. Decode divides completion tokens after the first by the first-to-last streamed-delta interval. Initial prefill is excluded. Batch wall throughput is a separate quantity and must not be replaced by this decode rate. An effort-75 response capped at 64 tokens can finish its budget inside thinking. A faster short interval therefore does not establish better answers or sustained long-generation capacity. The text matrix also predates the explicit native-image-routing flag. A later red-image smoke check in the deployment report is a serving-path check, not an image-understanding benchmark. ## A shared pool is an admission setting The [TensorFold deployment description](https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/deployment.md) reports eight slots, a 524,288-token shared pool and a 262,144-token per-request ceiling. Its [settings](https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/tensorfold-knobs.env.example) configure 5 GiB and 4 GiB memory floors, full prefill, and disabled session/prepared-cache writes. These are this profile's controls, not peak RAM measurements. They do not demonstrate eight simultaneous maximum-length requests. They also do not govern the separate near-1M experiment below. Comparing its lower free-memory reading with these floors would incorrectly combine different builds, launch settings and memory policies. ## Read the near-1M result as one specific retrieval task The [18 October run index](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md) and two saved JSON records describe one exact-string needle at 63 percent depth in random filler. Both runs found their requested string, with zero cached prompt tokens and seven completion tokens. | Saved run | Actual input tokens | Output tokens | First token | |---|---:|---:|---:| | Earlier build, `fd2365042f56` | 1,034,753 | 7 | 830.4 s | | Final setup, `92aa710b533d` | 1,035,020 | 7 | 817.7 s | The [earlier record](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json) and [final record](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json) retain exact counts and timestamps. These are different build/setup observations, not two unchanged-configuration repetitions. Runtime identifiers are reported prefixes; exact image digests and sampling values remain unknown. Zero reasoning tokens in the output is preserved without inventing an unlogged thinking flag. The result supports a narrow conclusion: each named setup accepted that prompt and found that string. It does not establish broad reasoning over a million tokens, multiple-needle recall, long-answer stability, multimodal context or concurrent full-window requests. The seven-token completion matters as much as the large input count. ## Keep the small memory margin attached to the result During the final fill, minimum host `MemAvailable` was 1,779 MiB on the head and 2,542 MiB on the worker. These are observed minimum available-memory readings, not the model's allocated memory or a recommended spare-RAM budget. The source describes swap restrictions, host reclaim/compaction settings and Engram page dropping. Its earlyoom threshold was 1,536 MiB during the reported run and was later changed to 1,024 MiB. This is a tuned, narrow-margin case. The [known limits](https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/KNOWN-LIMITS.md) and run index also retain slower prefill states and later final-image runs below earlier results. The site does not turn those whole-host changes into general deployment defaults. ## VQ conversion: file inspection before speed claims The [1017 VQ artifact card](https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md) reports a 113,556,639,424-byte base and a 971,066,016-byte vision sidecar. The base excludes the two Engram files. A 1,000-tensor loader binding check passed, but this snapshot explicitly has no 1017-specific serving speed, peak runtime RAM or model-quality result. Its [recipe](https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/RECIPE.md) pins the quantizer and vision converter separately. Its [provenance](https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/SOURCE.md) explains why matching names and sizes are insufficient: 1017 Engram and vision contents differ from older V4.1 files. Preserve revision and the supplied [SHA-256 identities](https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/SHA256SUMS), including when a loader accepts an older file without error. The native 1,048,576-token metadata field is not a measured VQ serving result. Older V4.1 speed/fidelity examples in the card remain older-checkpoint examples. They cannot fill the missing 1017 values, and file bytes cannot account for all runtime state, buffers, cache residency or service overhead. ## Use the evidence without extending it The [previous TensorFold explanation](/guides/deepseek-1017-tensorfold-community/) discusses other named configurations; it does not make their settings interchangeable with these cases. For AI access, `deepseek-1017-long-context` preserves both narrow retrieval cases and headroom, while `deepseek-1017-quantization` returns the short EXL3 report beside the VQ inspection. See [For AI](/for-ai/) for executable examples. Choose the evidence type needed for the next decision: verified file identity, endpoint behaviour, memory observation or task correctness. Keep actual input/output tokens, cache state, runtime and weight identities beside every result. Where the source supplies no execution result, the corresponding field stays null rather than becoming zero or an inherited benchmark. ## VQ follow-up checked on October 18 The publisher card was rechecked at revision `f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5`. It still explicitly leaves 1017-specific GB10 speed and quality unmeasured. The selected [runtime guide](https://github.com/Baekpica/ds4-dfm-rs/blob/c9425237669057937178e0598b3114ed9f8a15ec/docs/deepseek41-flash.md) and [performance document](https://github.com/Baekpica/ds4-dfm-rs/blob/c9425237669057937178e0598b3114ed9f8a15ec/docs/performance.md) do not supply a 1017 VQ run that fills those gaps. The Hub discussion listing was empty at this check. This is a review of those named sources, not proof that no report exists anywhere. File identity remains useful; old V4.1 speed, quality and residency figures remain outside the 1017 measurement set. No new VQ serving result is added. ## Further VQ source check on October 18 A later bounded check retains the same [1017 VQ card revision](https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md). The runtime has advanced to `9feb2d79fd0f84066d5c06ace4f4423e49fbda99`, but its [model guide](https://github.com/Baekpica/ds4-dfm-rs/blob/9feb2d79fd0f84066d5c06ace4f4423e49fbda99/docs/deepseek41-flash.md) and [performance document](https://github.com/Baekpica/ds4-dfm-rs/blob/9feb2d79fd0f84066d5c06ace4f4423e49fbda99/docs/performance.md) still contain no 1017-specific report. Hub discussions remain empty in this snapshot. No new 1017 VQ throughput, quality score or peak runtime RAM can be added from these checked sources; those fields remain null. Older V4.1 performance and the unrelated EXL3 results are not transferred. This describes the named-source review, not an exhaustive absence claim across the community. ## Original sources and conditions - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/README.md - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: 839509cc2365ea030d94360454437e66165c1087182ce6ccb85d23d85d5bb66d - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e - Baekpica: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md - Document revision: f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5 - SHA-256: b9f85bae000a8aa0e270555d17b1f9bd859a79bc3f5f2d7fde7eb00607cc990f - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/manifest.json - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: 6b3da5443112253502befe2a866983a31d188af234aef5e03865a724dbe9b65a - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/tensorfold-knobs.env.example - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: e26dc4761458be238d048c5a03b524993e7c632c27f8457527e3ba0030bc58c0 - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/vllm-knobs.env.example - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: ee4eb1535b45d28c47fdd5b136e4b04268a7955d2ac9ca83174afeb35c0d62d8 - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/deployment.md - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: 72fb210773604c15560e73d6db8ada144b0e953354201e42b3b7aebd4c018799 - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/benchmark-method.md - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: 33d83fe761ea2eaafaadee937fa212ffcec19d69a2528614776ad7fb1139ba3b - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/README.md - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: 380393cc1e86d175f21af1c216b613a2aad741ed90c851b7e8f491bed5231acf - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: 3005b3be00581a962f9a269211eef2d598a6358d83c41e565ee1549b28e03c33 - x00byte: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv - Document revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - SHA-256: befbf211d4f143798e12053ec21291cc77e0893284ccdb8d5a0aa19ef4722c6e - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: 9b81971c1378ab5b0bef466219c3a7aa7e2f78315bddbfc5bd0837cfec2fde40 - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: c476006b019d46c68a334ad3763f3e8e0f0532d01dfc6fb800c043c225709ab1 - Elevatormusic: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/KNOWN-LIMITS.md - Document revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - SHA-256: eeb6bc6f7a3fce20654f977581e9718b124e70fc01ab358efebefe9afd9dc942 - Baekpica: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/RECIPE.md - Document revision: f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5 - SHA-256: 1b66bfc44386bd6b708b1d4250cdd0dd825d2b45eee18c74d2bebfbff9e62486 - Baekpica: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/SOURCE.md - Document revision: f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5 - SHA-256: d039fb41a620380e352de9794271a6b2855e5d6931979d43695f4d3a236d443e - Baekpica: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/SHA256SUMS - Document revision: f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5 - SHA-256: 6b98a0a8923558f69ad53a9d60c29332b984a59f8109a6115c7c88e684763b19 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "tensorfold C1 thinking off", "metric": "decode_rate", "value": 58.05409336065285, "unit": "tokens/s", "statistic": "one pass; no run-to-run uncertainty estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "TensorFold Python 0.6.0 patched", "runtime_execution_revision": "c4646171139ee8a3c38103eaa1699dad226ec12b", "overlay_source_revision": "2dcce0303451da2a59f80e6c7413da8b3b0f7a57", "executed_image_digest": null, "thinking": false, "reasoning_effort": null, "input_tokens": 520, "actual_output_tokens": 64, "batch_wall_seconds": 2.4011132870218717, "end_to_end_output_wall_tps": 26.65430254620761, "TTFT_seconds": 1.313945789996069 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsdual-results-tensorfold-quick.csv", "source_document_sha256": "3005b3be00581a962f9a269211eef2d598a6358d83c41e565ee1549b28e03c33", "source_line": 2, "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv#L2", "source_record_id": "deepseek-1017-dual-spark-quick" }, { "label": "tensorfold C1 thinking 75", "metric": "decode_rate", "value": 77.49652509362433, "unit": "tokens/s", "statistic": "one pass; no run-to-run uncertainty estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "TensorFold Python 0.6.0 patched", "runtime_execution_revision": "c4646171139ee8a3c38103eaa1699dad226ec12b", "overlay_source_revision": "2dcce0303451da2a59f80e6c7413da8b3b0f7a57", "executed_image_digest": null, "thinking": true, "reasoning_effort": 75, "input_tokens": 546, "actual_output_tokens": 64, "batch_wall_seconds": 2.141774678020738, "end_to_end_output_wall_tps": 29.881761446140466, "TTFT_seconds": 1.326543439005036 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsdual-results-tensorfold-quick.csv", "source_document_sha256": "3005b3be00581a962f9a269211eef2d598a6358d83c41e565ee1549b28e03c33", "source_line": 10, "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv#L10", "source_record_id": "deepseek-1017-dual-spark-quick" }, { "label": "vllm C1 thinking off", "metric": "decode_rate", "value": 36.8753802051545, "unit": "tokens/s", "statistic": "one pass; no run-to-run uncertainty estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "vLLM + EXL3 overlay", "runtime_execution_revision": null, "overlay_source_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "executed_image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "thinking": false, "reasoning_effort": null, "input_tokens": 520, "actual_output_tokens": 64, "batch_wall_seconds": 2.5652991679962724, "end_to_end_output_wall_tps": 24.948357212460998, "TTFT_seconds": 0.8555283379973844 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsdual-results-vllm-quick.csv", "source_document_sha256": "befbf211d4f143798e12053ec21291cc77e0893284ccdb8d5a0aa19ef4722c6e", "source_line": 2, "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv#L2", "source_record_id": "deepseek-1017-dual-spark-quick" }, { "label": "vllm C1 thinking 75", "metric": "decode_rate", "value": 41.447290817386964, "unit": "tokens/s", "statistic": "one pass; no run-to-run uncertainty estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "vLLM + EXL3 overlay", "runtime_execution_revision": null, "overlay_source_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "executed_image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "thinking": true, "reasoning_effort": 75, "input_tokens": 546, "actual_output_tokens": 64, "batch_wall_seconds": 2.4199117650277913, "end_to_end_output_wall_tps": 26.447245277666145, "TTFT_seconds": 0.8985815939959139 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsdual-results-vllm-quick.csv", "source_document_sha256": "befbf211d4f143798e12053ec21291cc77e0893284ccdb8d5a0aa19ef4722c6e", "source_line": 10, "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv#L10", "source_record_id": "deepseek-1017-dual-spark-quick" }, { "label": "fd2365042f56 near-1M first token", "metric": "ttft", "value": 830.4, "unit": "s", "statistic": "single prompt, no variance estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "fd2365042f56", "input_tokens": 1034753, "output_tokens": 7, "cached_tokens": 0, "cache_state": "zero prefix-cache hit tokens; not a cold OS page-cache guarantee", "experiment_date": "2026-10-18T01:51:54Z", "needle_found": true, "reasoning_tokens": 0, "total_seconds": 835.8 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsneedle-runs-2026-10-18-needle-1m.json", "source_document_sha256": "9b81971c1378ab5b0bef466219c3a7aa7e2f78315bddbfc5bd0837cfec2fde40", "source_line": 8, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json#L8", "source_record_id": "deepseek-1017-million-token-needle" }, { "label": "92aa710b533d near-1M first token", "metric": "ttft", "value": 817.7, "unit": "s", "statistic": "single prompt, no variance estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cached_tokens": 0, "cache_state": "zero prefix-cache hit tokens; not a cold OS page-cache guarantee", "experiment_date": "2026-10-18T04:47:07Z", "needle_found": true, "reasoning_tokens": 0, "total_seconds": 825.9 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsneedle-runs-2026-10-18-needle-1m-final.json", "source_document_sha256": "c476006b019d46c68a334ad3763f3e8e0f0532d01dfc6fb800c043c225709ab1", "source_line": 8, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json#L8", "source_record_id": "deepseek-1017-million-token-needle" }, { "label": "near-1M final head minimum MemAvailable", "metric": "minimum_memavailable", "value": 1779, "unit": "MiB", "statistic": "minimum observed in this one fill, not a safe capacity guarantee", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cache_state": "same final needle; cached_tokens 0", "experiment_date": "2026-10-18", "memory_kind": "minimum_host_MemAvailable_during_prefill", "node": "head", "earlyoom_limit_MiB_at_measurement": 1536, "earlyoom_later_limit_MiB": 1024, "OS_memory_controls_changed": true, "ordinary_host_baseline_inferred": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsneedle-runs-2026-10-18-INDEX.md", "source_document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "source_line": 59, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L59", "source_record_id": "deepseek-1017-million-token-needle" }, { "label": "near-1M final worker minimum MemAvailable", "metric": "minimum_memavailable", "value": 2542, "unit": "MiB", "statistic": "minimum observed in this one fill, not a safe capacity guarantee", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cache_state": "same final needle; cached_tokens 0", "experiment_date": "2026-10-18", "memory_kind": "minimum_host_MemAvailable_during_prefill", "node": "worker", "earlyoom_limit_MiB_at_measurement": 1536, "earlyoom_later_limit_MiB": 1024, "OS_memory_controls_changed": true, "ordinary_host_baseline_inferred": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsneedle-runs-2026-10-18-INDEX.md", "source_document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "source_line": 59, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L59", "source_record_id": "deepseek-1017-million-token-needle" }, { "label": "1017 VQ base GGUF", "metric": "artifact_size", "value": 113556639424, "unit": "bytes", "statistic": "source-reported file bytes; not a RAM requirement", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "model_revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "base_model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "quantizer_revision": "7fb26cc332d740416792204a29c7d13d2986caf5", "vision_converter_revision": "e4ea07889144d85acd33b2490ec4f44032644fb0", "quantization": "YoungAi vq8sh14-q4k-mtpnative; routed-expert VQ, q4_K skeleton; native MTP towers", "measurement_kind": "publisher artifact layout inspection, not runtime evaluation", "experiment_date": "2026-10-17 status in card", "system": null, "engine": "target ds4-dfm-rs; loader check passed, serving unmeasured on 1017", "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "context_limit_tokens": 1048576, "context_limit_origin": "model metadata, not measured serving result", "speed_1017_measured": false, "quality_1017_measured": false, "peak_runtime_memory_measured": false, "base_model_speed_inherited": false, "filename": "DeepSeek-V4.1-Flash-vq8sh14-q4k-mtpnative.gguf", "memory_kind": "artifact_disk_file_bytes", "Engram_included": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "ds1017vq-readme", "source_document_sha256": "b9f85bae000a8aa0e270555d17b1f9bd859a79bc3f5f2d7fde7eb00607cc990f", "source_line": 97, "evidence_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md#L97", "source_record_id": "deepseek-1017-vq-artifact-inspection" }, { "label": "1017 BF16 vision sidecar", "metric": "artifact_size", "value": 971066016, "unit": "bytes", "statistic": "source-reported file bytes; not a RAM requirement", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "model_revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "base_model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "quantizer_revision": "7fb26cc332d740416792204a29c7d13d2986caf5", "vision_converter_revision": "e4ea07889144d85acd33b2490ec4f44032644fb0", "quantization": "YoungAi vq8sh14-q4k-mtpnative; routed-expert VQ, q4_K skeleton; native MTP towers", "measurement_kind": "publisher artifact layout inspection, not runtime evaluation", "experiment_date": "2026-10-17 status in card", "system": null, "engine": "target ds4-dfm-rs; loader check passed, serving unmeasured on 1017", "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "context_limit_tokens": 1048576, "context_limit_origin": "model metadata, not measured serving result", "speed_1017_measured": false, "quality_1017_measured": false, "peak_runtime_memory_measured": false, "base_model_speed_inherited": false, "filename": "DeepSeek-V4.1-Flash.vision.gguf", "memory_kind": "artifact_disk_file_bytes", "Engram_included": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "ds1017vq-readme", "source_document_sha256": "b9f85bae000a8aa0e270555d17b1f9bd859a79bc3f5f2d7fde7eb00607cc990f", "source_line": 98, "evidence_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md#L98", "source_record_id": "deepseek-1017-vq-artifact-inspection" } ], "conditions": { "one_needle_is_general_long_context_quality": false, "EXL3_and_VQ_results_interchangeable": false, "VQ_1017_speed_measured": false, "VQ_1017_quality_measured": false, "independent_site_runs": 0 }, "deployment_profiles": [ { "id": "dsdual-python-profile", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "checkpoint_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "engine": "TensorFold Python 0.6.0 patched", "runtime_revision": "c4646171139ee8a3c38103eaa1699dad226ec12b", "context_limit_tokens": 262144, "configured_shared_pool_tokens": 524288, "configured_request_slots": 8, "configured_MemAvailable_floor_GiB": 5, "configured_hard_floor_GiB": 4, "memory_floors_are_measured_peaks": false, "session_cache": false, "prepared_cache_writes": false, "source_current_recipe_has_native_image_routing": true, "text_matrix_predates_image_flag": true } ], "memory_observations": [ { "record_id": "deepseek-1017-million-token-needle", "kind": "minimum_host_MemAvailable_during_final_fill", "head_MiB": 1779, "worker_MiB": 2542, "host_tuning_required_in_source": true, "safe_capacity_guarantee": false }, { "record_id": "deepseek-1017-vq-artifact-inspection", "kind": "file_inventory_not_RAM", "base_bytes": 113556639424, "vision_bytes": 971066016, "Engram_shard_bytes": [ 101535150936, 101537926640 ], "actual_peak_RAM": null, "speed_and_quality_1017": null } ], "long_context_cases": [ { "id": "dsneedle-runs-2026-10-18-needle-1m.json", "classification": "source_reported_single_needle_success", "configured_limit_tokens": 1048576, "actual_input_tokens": 1034753, "actual_output_tokens": 7, "needle_depth_fraction": 0.63, "exact_match": true, "TTFT_seconds": 830.4, "cached_tokens": 0, "runtime_revision_prefix": "fd2365042f56", "executed_image_digest": null, "sampling": null, "quality_scope": "one needle at one depth; not broad reasoning, multi-needle, long-response or concurrency validation", "source_document_id": "dsneedle-runs-2026-10-18-needle-1m.json", "source_document_sha256": "9b81971c1378ab5b0bef466219c3a7aa7e2f78315bddbfc5bd0837cfec2fde40", "source_line": 3, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json#L3" }, { "id": "dsneedle-runs-2026-10-18-needle-1m-final.json", "classification": "source_reported_single_needle_success", "configured_limit_tokens": 1048576, "actual_input_tokens": 1035020, "actual_output_tokens": 7, "needle_depth_fraction": 0.63, "exact_match": true, "TTFT_seconds": 817.7, "cached_tokens": 0, "runtime_revision_prefix": "92aa710b533d", "executed_image_digest": null, "sampling": null, "quality_scope": "one needle at one depth; not broad reasoning, multi-needle, long-response or concurrency validation", "source_document_id": "dsneedle-runs-2026-10-18-needle-1m-final.json", "source_document_sha256": "c476006b019d46c68a334ad3763f3e8e0f0532d01dfc6fb800c043c225709ab1", "source_line": 3, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json#L3" } ] } ``` --- # Qwen Flash-Next: Concurrency Has a Latency Cost Qwen3.8-Flash-Next can serve more requests while each user waits longer. The useful question is which request mix was actually tested. A single-Spark SGLang study publishes a ShareGPT concurrency sweep; a separate two-Spark vLLM/b12x study publishes task throughput and cold-prefix latency. Their different checkpoints, cache policies and hardware prevent a controlled engine ranking. A third operator's memory failure explains why a configured window and a scheduler limit are insufficient capacity evidence. ![Two distinct Spark benchmark cohorts with different request shapes, cache settings and repeat counts.](/assets/qwen-flash-next-spark-concurrency-source.webp) ## Single Spark: read latency beside aggregate throughput The [nabe2030 recipe](https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md) uses `senfu/Qwen3.8-Flash-Next-NVFP4` at `5d37b3b3711d8406174b96ff950c0aa16324b266`, SGLang commit `4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b`, and NVMe file-backed PLE. Experts and the LM head use NVFP4; the dense path uses FP8. Those details distinguish it from the other Flash-Next conversions on this site. Its [aggregated result table](https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md) uses `vllm bench serve` with 200 ShareGPT prompts, thinking off, MTP off and radix caching disabled. One warmup request is discarded. The C1 summary is the median of three trials; the higher-concurrency points are single trials. All selected senfu cases completed 200 requests without errors. Individual prompt/output distributions, temperature and per-request cache history are not supplied. | Concurrent requests | Aggregate output, tokens/s | Median first token, ms | Median time per output token, ms | |---:|---:|---:|---:| | 1 | 26.97 | 221.23 | 36.14 | | 8 | 67.68 | 526.78 | 113.46 | | 16 | 77.48 | 686.27 | 198.00 | | 32 | 89.35 | 1,017.98 | 343.57 | ![Single-Spark median TTFT increases from 221.23 ms at C1 to 1017.98 ms at C32.](/assets/qwen-sglang-ttft-concurrency.webp) For a shared endpoint, more total output does not preserve the interactive experience. Keep first-token wait and token spacing beside aggregate throughput when deciding how many users to admit. Do not estimate an individual stream's speed by dividing aggregate tokens/s by concurrency: the timing definitions and request completion patterns need not match. Single higher-load trials also provide no run-to-run error bar. These results are useful measurements of this workload, rather than a latency guarantee for a different team. ## Two Sparks: preserve the older concurrency cohort The [TP2 repository](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/README.md) contains several generations of measurements. The selected k76 concurrency series belongs to recipe v2.0.0 on October 10, using two directly connected Sparks and `llama-benchy 0.4.1.dev4+g0d4de4271`. It requests 2,048 input and 512 output tokens, temperature 1.0, top-p 0.95, top-k 20 and thinking on, with prefix caching enabled. It reports three runs within one boot. The [version ledger](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/VERSIONS.md) binds v2.0.0 to GDN-MSE checkpoint `16c9bd54788d12390838a65ce4a4ecda97fa5f1d`. The later v2.1.0/v2.2.0 checkpoint ends in `...85406` and includes a different MTP drafter. Keeping the old checkpoint identity prevents newer recipe defaults from silently rewriting historical results. The ledger's release-image digest is retained separately from the unreported digest of the exact executed test container. | Concurrency | Aggregate output, mean ± run SD | Per-request decode, mean ± run SD | |---:|---:|---:| | 1 | 88.697 ± 2.562 | 88.697 ± 2.562 | | 4 | 156.645 ± 4.558 | 48.631 ± 1.533 | | 8 | 172.213 ± 16.348 | 33.236 ± 0.977 | | 16 | 231.294 ± 4.910 | 22.142 ± 0.362 | Both columns use tokens/s. The [machine-readable source CSV](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv) retains the exact values and original run paths. Aggregate throughput divides all output by elapsed wall time including prompt processing. Per-request decode describes one stream after its first token. Their denominators differ; neither column can be reconstructed by simply multiplying or dividing the other by the number of users. ![Dual-Spark per-request decode falls from 88.697 to 22.142 tokens per second across C1 to C16.](/assets/qwen-tp2-per-request-concurrency.webp) The larger C8 aggregate SD remains visible. It describes dispersion in those three runs; it does not identify a cause, quantify between-boot variation or establish a confidence interval. Reporting mean and SD does not itself require a normally distributed population. A scheduling explanation would need more evidence than the summary cells provide. ## Long-context latency comes from a separate release The v2.2.0 cold-prefix arm uses the later checkpoint, C1, temperature zero and one requested output token. It averages four boots with three cold prompts per boot. SD is between boot means. The [method and build notes](https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/BENCHMARKS.md) separate this prefill work from the older concurrency campaign. | Actual input tokens | First-token latency, mean ± boot SD | Prefill, mean ± boot SD, tokens/s | |---:|---:|---:| | 16,384 | 5.487 ± 0.045 s | 2,986.0 ± 24.6 | | 65,536 | 24.041 ± 0.057 s | 2,726.1 ± 6.4 | | 131,072 | 54.037 ± 0.256 s | 2,425.6 ± 11.5 | ![Dual-Spark cold-prefix TTFT is 5.487, 24.041 and 54.037 seconds at the three measured input lengths.](/assets/qwen-tp2-cold-context-latency.webp) A cold prefix means there was no matching prefix-cache hit. It does not establish cold weight pages, cold compilation caches or a cold operating-system page cache. One requested output token measures the route to the first answer token, rather than long-answer generation. These points cannot prove that sixteen simultaneous 128K prompts fit or retain the same latency. Read the earlier [memory/context explanation](/guides/qwen-flash-next-memory-context/) for the distinction between a shared KV pool and a request ceiling. ## Memory failure is useful evidence too In a separate [first-person dual-GB10 operator report](https://forums.developer.nvidia.com/t/385731/1), jcagle describes vLLM 0.30.0 with MiaAI overlays, FP8 KV, three MTP drafts, 16 sequence slots and capped GPU clocks. This is a third configuration, rather than another repetition of k76. The report is post version 3, updated October 13; its public URL is mutable, so a dated snapshot hash is retained in the case export instead of claiming an immutable commit URL. The operator reports an eight-request deep-prefill burst that reached 2.4 GiB head-node MemAvailable at GMU 0.80 before earlyoom killed the engine. With GMU 0.78, the reported eight-by-64K-class load completed at 2,510 combined prefill tokens/s and 117 seconds TTFT, with 3.8 GiB available on the head. Exact individual token counts, output lengths, sampling and repeated-run dispersion are missing. These observations demonstrate the operator's outcome; they do not make 0.78 a universal safe setting or isolate GMU as the only causal variable. The single-Spark SGLang recipe separately lists a 49,152-token per-request window, 393,216 total tokens and eight slots for one configuration. That configuration is not evidence that every slot was simultaneously filled. Its 3.04 GB KV figure at 128K is a source-labelled allocation, with byte convention unverified, rather than whole-model peak RAM. The [KV and speculation guide](/guides/kv-cache-speculative-decoding/) explains why those distinctions matter. ## A practical reading order Choose the source matching your checkpoint, runtime and hardware. Next, compare the tested prompt/output mix with your actual workload. Then inspect aggregate output, per-request decode, first-token delay and host memory together. Finally, mark configured capacities that the source did not actually fill. This is an editorial method for choosing what to verify, not a request to reproduce these benchmarks automatically. ### Does C32 throughput prove C32 long-context serving? No. The single-Spark C32 ShareGPT mix does not publish a simultaneous full-window fill. Its prompt distribution is also absent. ### Can the two Spark articles rank SGLang against vLLM? No controlled engine comparison is available here. Checkpoints, nodes, cache behavior, MTP and request shapes differ. ### Which missing fields should an AI retain? Keep null execution digests, unknown request lengths, trial counts and memory scope. Use the [For AI filters](/for-ai/) to retrieve complete reports and their conditions rather than a speed-only fragment. ## Original sources and conditions - nabe2030: https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md - Document revision: 8a161f2bee2e2d85b689a2d457ae68bead3929a4 - SHA-256: 8a27445102511702ff42fa60cfcc469583de37c4c0bd031c43cd40db7c0db9f4 - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/README.md - Document revision: b384f4a0e3ff0d563980befe52eb3e6964a5b6b1 - SHA-256: f3bd45cb1f01c287ef00f97b98e0d5c5d66576d5be34baaefce5c108242ad1cf - nabe2030: https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md - Document revision: 8a161f2bee2e2d85b689a2d457ae68bead3929a4 - SHA-256: cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297 - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv - Document revision: b384f4a0e3ff0d563980befe52eb3e6964a5b6b1 - SHA-256: a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/BENCHMARKS.md - Document revision: b384f4a0e3ff0d563980befe52eb3e6964a5b6b1 - SHA-256: 5645ff4e129b3e2766672e092c65333b073b5550ad89aa6374cbd81ce1da7776 - ursuciprian: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/VERSIONS.md - Document revision: b384f4a0e3ff0d563980befe52eb3e6964a5b6b1 - SHA-256: eb9ec6435802430a1be7302b445c67ceb48da9cf5e1ba32e939b5f7aaf9b100e - jcagle: https://forums.developer.nvidia.com/t/385731/1 - Document revision: 3 - SHA-256: 95ae1019d350e10f941b22110e09c3d65372413fa3139ae2b676b7eb244267a0 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "single Spark senfu C1 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 26.97, "unit": "tokens/s", "statistic": "median of three trials", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 9, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C1 median_ttft", "metric": "median_ttft", "value": 221.23, "unit": "ms", "statistic": "median of three trials", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 9, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C1 median_time_per_output_token", "metric": "median_time_per_output_token", "value": 36.14, "unit": "ms", "statistic": "median of three trials", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 9, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C8 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 67.68, "unit": "tokens/s", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 10, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C8 median_ttft", "metric": "median_ttft", "value": 526.78, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 10, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C8 median_time_per_output_token", "metric": "median_time_per_output_token", "value": 113.46, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 10, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C16 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 77.48, "unit": "tokens/s", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 11, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C16 median_ttft", "metric": "median_ttft", "value": 686.27, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 11, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C16 median_time_per_output_token", "metric": "median_time_per_output_token", "value": 198.0, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 11, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C32 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 89.35, "unit": "tokens/s", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 12, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C32 median_ttft", "metric": "median_ttft", "value": 1017.98, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 12, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "single Spark senfu C32 median_time_per_output_token", "metric": "median_time_per_output_token", "value": 343.57, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 12, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12", "source_record_id": "qwen38-flash-next-sglang-concurrency" }, { "label": "TP2 v2.0.0 C1 2048 per_request_decode_rate", "metric": "per_request_decode_rate", "value": 88.697, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 2.562, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 1, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1142, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1142", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.0.0 C4 2048 per_request_decode_rate", "metric": "per_request_decode_rate", "value": 48.631, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 1.533, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 4, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1146, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1146", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.0.0 C8 2048 per_request_decode_rate", "metric": "per_request_decode_rate", "value": 33.236, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 0.977, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 8, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1148, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1148", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.0.0 C16 2048 per_request_decode_rate", "metric": "per_request_decode_rate", "value": 22.142, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 0.362, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 16, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1150, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1150", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.0.0 C1 2048 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 88.697, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 2.562, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 1, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1152, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1152", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.0.0 C4 2048 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 156.645, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 4.558, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 4, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1156, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1156", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.0.0 C8 2048 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 172.213, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 16.348, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 8, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1158, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1158", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.0.0 C16 2048 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 231.294, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 4.91, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 16, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1160, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1160", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.2.0 C1 16384 ttft", "metric": "ttft", "value": 5.487, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 0.045, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 16384, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1262, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1262", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.2.0 C1 16384 prefill_rate", "metric": "prefill_rate", "value": 2986.0, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 24.6, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 16384, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1263, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1263", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.2.0 C1 65536 ttft", "metric": "ttft", "value": 24.041, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 0.057, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 65536, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1264, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1264", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.2.0 C1 65536 prefill_rate", "metric": "prefill_rate", "value": 2726.1, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 6.4, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 65536, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1265, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1265", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.2.0 C1 131072 ttft", "metric": "ttft", "value": 54.037, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 0.256, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 131072, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1266, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1266", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" }, { "label": "TP2 v2.2.0 C1 131072 prefill_rate", "metric": "prefill_rate", "value": 2425.6, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 11.5, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 131072, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1267, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1267", "source_record_id": "qwen38-flash-next-tp2-concurrency-context" } ], "conditions": { "cross_engine_controlled_comparison": false, "concurrency_sweep_is_full_window_test": false, "one_boot_SD_is_boot_SD": false, "independent_site_runs": 0 }, "case_reports": [ { "id": "qwen-dual-gmu080", "classification": "source_reported_host_oom", "model": "Qwen3.8-Flash-Next", "engine": "vLLM 0.30.0 plus MiaAI overlays", "node_count": 2, "model_revision": null, "executed_image_digest": null, "GMU": 0.8, "concurrency": 8, "prompt_tokens_per_request_reported": "64K-class deep-prefill burst; exact individual counts absent", "minimum_head_MemAvailable_GiB": 2.4, "outcome": "earlyoom killed engine", "source_post_version": 3, "source_updated_at": "2026-10-13T22:17:54.906Z", "source_url": "https://forums.developer.nvidia.com/t/385731/1", "source_snapshot_sha256": "95ae1019d350e10f941b22110e09c3d65372413fa3139ae2b676b7eb244267a0", "immutable_public_revision_URL": false }, { "id": "qwen-dual-gmu078", "classification": "source_reported_deep_prefill_completion", "model": "Qwen3.8-Flash-Next", "engine": "vLLM 0.30.0 plus MiaAI overlays", "node_count": 2, "model_revision": null, "executed_image_digest": null, "GMU": 0.78, "concurrency": 8, "input_length_source_label": "64K each; exact token accounting absent", "aggregate_prefill_tokens_per_second": 2510, "TTFT_seconds": 117, "minimum_head_MemAvailable_GiB": 3.8, "output_tokens": null, "thinking": null, "temperature": null, "repetitions": null, "source_post_version": 3, "source_updated_at": "2026-10-13T22:17:54.906Z", "source_url": "https://forums.developer.nvidia.com/t/385731/1", "source_snapshot_sha256": "95ae1019d350e10f941b22110e09c3d65372413fa3139ae2b676b7eb244267a0", "immutable_public_revision_URL": false } ], "memory_observations": [ { "kind": "source_reported_KV_allocation_at_context", "context_limit_tokens": 131072, "KV_source_GB": 3.04, "GB_byte_convention_verified": false, "whole_model_peak_RAM": null, "source_document_id": "qwen-sglang-readme", "source_document_sha256": "8a27445102511702ff42fa60cfcc469583de37c4c0bd031c43cd40db7c0db9f4", "source_line": 19, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md#L19" }, { "kind": "source_recipe_slots_not_a_completed_long_batch", "per_request_window_tokens": 49152, "shared_total_tokens": 393216, "request_slots": 8, "all_slots_full_simultaneously_proven": false, "source_document_id": "qwen-sglang-readme", "source_document_sha256": "8a27445102511702ff42fa60cfcc469583de37c4c0bd031c43cd40db7c0db9f4", "source_line": 134, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md#L134" } ] } ``` --- # Qwen on RTX 5090: What the 128K Recipe Tested A Qwen3.8-Flash-Next server can advertise a 131,072-token window while its longest published request contains 104,805 tokens. That is useful long-context evidence with a precise boundary. The original RTX 5090 WSL2 recipe also publishes memory telemetry from a shorter 32K gate, rejects MTP promotion and leaves ngram speculation unrun. Preserving these separate outcomes makes the report more useful than a single “fits on 32 GB” headline. ![The configured 131072-token window, completed 104805-token retrieval and 32K-only peak-memory telemetry are separate claims.](/assets/qwen-flash-next-5090-context-memory-source.webp) ## Start with the CPU offload and the exact artifact The [original kutaelee repository](https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md) describes one RTX 5090 reporting 32,607 MiB, a 16-core/32-thread desktop CPU, a 128 GiB-class host and Ubuntu under WSL2 with a 96 GiB VM ceiling. The selected model is `jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF` at `4a768bf4458a83e1c80667b4d818ec721080546e`. Its file is 126,846,153,856 bytes, with a publisher-provided SHA-256 retained in the JSON conditions. Run AI Here inspected the public documents; it did not download or execute this weight file. The tested engine is `ik_llama.cpp` merge commit `d854dfd36a7262fc46cb7542716b04a9f6d4b47d`, including the pinned MTP patch. A repository documentation revision and a runtime execution revision identify different things, so both are retained. Later changes at a repository's moving main branch are not automatically part of this test. Most importantly, the [baseline configuration](https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/configs/ik-llama-baseline-128k.example.sh) keeps 38 expert layers on the CPU, defers experts, enables mmap and uses Q8_0 KV. It allows one request and one server slot. This is partial CPU expert offload, not a claim that the whole model resides in the GPU's memory. A Q8 n-gram embedding table is part of the model artifact; ngram speculative decoding is a different feature. Keeping speculation off does not mean the model's embedding table is absent or unused. ## Read the actual request lengths The three [reported baseline gates](https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md) use different prompts. Throughput comes from server timing fields, according to the [methodology](https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/methodology.md). It is not a client-side elapsed-time estimate. | Completed gate | Configured window | Actual prompt tokens | Prompt processing, tokens/s | Decode, tokens/s | |---|---:|---:|---:|---:| | 32K exact needle | 32,768 | 26,253 | 271.64 | 16.53 | | 128K server smoke | 131,072 | 61 | 17.02 | 15.26 | | Longer exact needle | 131,072 | 104,805 | 245.72 | 13.94 | ![Three completed 5090 gates use 26253, 61 and 104805 prompt tokens, with distinct prompt-processing rates.](/assets/qwen-5090-actual-input-gates.webp) The 61-token request confirms that a server configured for 128K completed a short smoke test. It does not test a 128K input. The longer gate is more informative about retrieval at depth, while still falling short of a completely filled 131,072-token input. Actual completion lengths, needle depth, per-gate sampling and repeat counts are not supplied in the selected table. Do not silently insert the runtime's current defaults into those empty fields. These three rows are not a controlled context-scaling experiment. Request content, startup state and task differ, and no uncertainty distribution accompanies these gate values. It would be misleading to fit a latency law or predict a maximum memory requirement from the table. A request passing also does not establish that several such requests can run together: every published row uses C1. ## Keep the 32K memory telemetry in its own cohort The source reports 25.15 seconds of startup and the following observations during the completed 32K monitoring gate: | Quantity | Original reported value | What it describes | |---|---:|---| | Peak host RSS | 53,901,956 KiB | Resident process memory in the reported monitor scope | | Minimum MemAvailable | 86,701,016 KiB | An available-memory sample whose precise boundary is unspecified | | Maximum swap use | 63,708 KiB | Reported swap use during that gate | | Peak GPU memory | 25,813 MiB | GPU allocation observation for that gate | ![The 32K telemetry retains peak RSS, minimum MemAvailable, swap and peak GPU memory in original KiB or MiB.](/assets/qwen-5090-32k-memory-telemetry.webp) There is no paired peak-RAM, swap or peak-GPU series for the 104,805-token needle in the selected documents. Those longer-test fields remain null. Do not relabel the shorter gate's peak as a “128K memory requirement,” and do not treat absent long-context swap data as zero swap. The reported MemAvailable scope deserves particular care. The source names a 96 GiB WSL ceiling but does not establish that its available-memory sample and that ceiling describe the same boundary. The site therefore preserves the original number without calculating a supposed spare VM budget from it. Host RSS, mapped file pages, GPU allocations and a VM limit should not simply be added together. Units remain KiB or MiB rather than silently becoming decimal GB. ## mmap success addresses a specific allocation problem The source explains that the default CUDA-host override attempted a 98.61 GiB anonymous pinned allocation during validation. The accepted baseline sets `GGML_CUDA_NO_PINNED=1` and retains mmap. That observation explains the author's configuration choice; it does not guarantee identical memory behavior on another operating system, cache state or runtime build. The [limitations document](https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/limitations.md) identifies cold expert-page faults and CPU expert work as important constraints for this workstation. Those are source interpretations with a clear configuration attached. They do not establish that every 5090 deployment is CPU limited, nor that every alternative conversion has the same behavior. The [Spark memory article](/guides/qwen-flash-next-memory-context/) describes different hardware and PLE backing experiments; those measurements cannot substitute for missing WSL telemetry. ## Retain the failed and unrun speculation paths The repository reports that MTP n=4 preserved eight correctness checks, but two completed comparable prose samples reached only 2.52 and 3.14 tokens/s. Canonical MTP n=1 also preserved tested correctness and was slower across its completed short gates. Its peak GPU memory reached 31,775 MiB. The memory point is a separate MTP short-gate observation, not the baseline's long-context peak. The author consequently rejected MTP promotion for this exact MXFP4 conversion and 38-layer CPU-expert split. Ngram speculative decoding was not run because the protocol required a beneficial MTP lane first. “Not run” provides no measured ngram speed, memory improvement or failure rate. The n-gram embedding table remains a separate model component throughout this decision. ![Baseline is retained, MTP speed promotion was rejected, and ngram was not run because its prerequisite failed.](/assets/qwen-5090-mtp-promotion-gates.webp) This is a useful operational result: correctness, speed and headroom can disagree. Passing a small correctness gate does not compel deployment of a slower speculative path. It also does not prove general quality equivalence. Read the broader [KV/speculation explanation](/guides/kv-cache-speculative-decoding/) before transferring a favorable MTP result from a different checkpoint or engine. ## What a useful follow-up report would add For this configuration, the most valuable missing evidence would pair actual prompt and output counts with telemetry from the same long request. It would identify whether MemAvailable was sampled inside the VM or on the host, retain the cache state, and repeat an unchanged workload before summarizing variation. Any concurrency claim would need completed multi-request runs with their scheduling and memory outcomes. These are editorial suggestions for future evidence, not measurements performed by this site. ### Has the full 128K input been validated here? The selected source validates an exact needle in 104,805 input tokens under a 131,072-token configured ceiling. It does not publish a fully filled 131,072-token input gate. ### Does the 104K pass establish repository-analysis quality? No. Exact retrieval of one embedded answer is narrower than multi-document reasoning, trustworthy code changes or long output generation. ### Is ngram missing because the embedding table was disabled? No. The skipped experiment is speculative decoding. The Q8 n-gram embedding table belongs to the model artifact and is a separate mechanism. The [For AI reference](/for-ai/) keeps these states distinct in the exported conditions. ## Original sources and conditions - kutaelee: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md - Document revision: fca75dd108ce04b6a79db3b45934ed58bba893bb - SHA-256: 0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0 - kutaelee: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/methodology.md - Document revision: fca75dd108ce04b6a79db3b45934ed58bba893bb - SHA-256: 54d61a37de6873bcebe34c2ad34978a9b0a7191232fb865c364540cdd3e4c706 - kutaelee: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/limitations.md - Document revision: fca75dd108ce04b6a79db3b45934ed58bba893bb - SHA-256: 029f3cfc9a4b8e2e03eba92137e4df3ca722782b13af4dd990295a8fe1c270d2 - kutaelee: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/configs/ik-llama-baseline-128k.example.sh - Document revision: fca75dd108ce04b6a79db3b45934ed58bba893bb - SHA-256: fc5de996c4e03b80de59f21597ae543a37ff829822641bf2117426b9c6f09f28 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "32K needle gate prefill_rate", "metric": "prefill_rate", "value": 271.64, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "workload": "32K needle gate", "memory_cohort": "completed 32K monitoring" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 53, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L53", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "32K needle gate decode_rate", "metric": "decode_rate", "value": 16.53, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "workload": "32K needle gate", "memory_cohort": "completed 32K monitoring" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 53, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L53", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "128K server smoke prefill_rate", "metric": "prefill_rate", "value": 17.02, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 61, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K server smoke", "memory_cohort": "no paired 128K telemetry published" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 54, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L54", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "128K server smoke decode_rate", "metric": "decode_rate", "value": 15.26, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 61, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K server smoke", "memory_cohort": "no paired 128K telemetry published" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 54, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L54", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "128K long-context needle prefill_rate", "metric": "prefill_rate", "value": 245.72, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 104805, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K long-context needle", "memory_cohort": "no paired 128K telemetry published" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 55, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L55", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "128K long-context needle decode_rate", "metric": "decode_rate", "value": 13.94, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 104805, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K long-context needle", "memory_cohort": "no paired 128K telemetry published" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 55, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L55", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "32K telemetry startup", "metric": "startup", "value": 25.15, "unit": "s", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "startup_duration", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "32K telemetry host_rss", "metric": "host_rss", "value": 53901956, "unit": "KiB", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "peak_host_RSS", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "32K telemetry minimum_memavailable", "metric": "minimum_memavailable", "value": 86701016, "unit": "KiB", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "minimum_MemAvailable_in_reported_monitor_scope", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "32K telemetry swap_use", "metric": "swap_use", "value": 63708, "unit": "KiB", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "maximum_swap_use", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "32K telemetry GPU_memory", "metric": "GPU_memory", "value": 25813, "unit": "MiB", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "peak_GPU_memory", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "source_record_id": "qwen38-flash-next-5090-offload-gates" }, { "label": "canonical MTP n1 peak GPU memory", "metric": "GPU_memory", "value": 31775, "unit": "MiB", "statistic": "maximum in completed MTP short gates", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 1, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": null, "context_limit_tokens": 131072, "cache_state": null, "memory_cohort": "canonical MTP n1 short gates; not baseline or long-context peak", "memory_kind": "peak_GPU_memory" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 61, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L61", "source_record_id": "qwen38-flash-next-5090-offload-gates" } ], "conditions": { "concurrency": 1, "hardware_sample_size": 1, "full_131072_input_tested": false, "all_gate_memory_peaks_measured": false, "MTP_promoted": false, "ngram_state": "NOT_RUN_BY_GATE", "independent_site_runs": 0 }, "memory_observations": [ { "kind": "unpaired_memory_telemetry", "measured_context_limit_tokens": 32768, "measured_input_tokens": 26253, "longer_needle_input_tokens": 104805, "longer_needle_peak_RAM": null, "host_MemAvailable_scope": null, "CPU_expert_layers": 38, "source_GGUF_bytes": 126846153856 } ], "long_context_cases": [ { "classification": "source_reported_exact_needle_success", "actual_input_tokens": 26253, "configured_limit_tokens": 32768, "actual_output_tokens": null, "exact_match": true, "needle_depth_fraction": null, "general_reasoning_quality_proven": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 53, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L53" }, { "classification": "source_reported_short_smoke", "actual_input_tokens": 61, "configured_limit_tokens": 131072, "actual_output_tokens": null, "exact_match": true, "needle_depth_fraction": null, "general_reasoning_quality_proven": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 54, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L54" }, { "classification": "source_reported_exact_needle_success", "actual_input_tokens": 104805, "configured_limit_tokens": 131072, "actual_output_tokens": null, "exact_match": true, "needle_depth_fraction": null, "general_reasoning_quality_proven": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 55, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L55" } ] } ``` --- # Qwen Long Context: Concurrency, Memory and Failure Evidence ![Penny Royal original tests, a four-GPU matrix, and a dated scheduler incident.](/assets/batch7-qwen-long-hero.webp) *Original reports only. No new Run AI Here model execution.* Long context changes the question a deployment report needs to answer. A server can accept a large window, submit several requests and report a high token rate without keeping every long document resident at once. Useful evidence records the actual request shape, overlapping execution, queue, prefix cache, completion length and memory sampling window together. This note reads original Qwen3.8-Flash-Next reports with those distinctions intact. Run AI Here has not repeated their model tests. ## One-card long batches: what actually overlapped The [Penny Royal original results](https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md) contain September 11 v2.5.0 observations on one RTX PRO 6000 Blackwell, TP1. The relevant configuration uses RadixArk ModelOpt NVFP4, RAM PLE, FP8 KV, FR-Spec/native NEXTN, a 524,288-token request limit, an 824,384-token pool, 24 Mamba slots and 32 GiB HiCache/NIXL. “Online FP8” changes selected projections; it does not mean every model component changes precision. | Source-labelled workload | Option off, aggregate tokens/s | Online FP8, aggregate tokens/s | Online-arm output tokens / group seconds | |---|---:|---:|---| | 128K, four requests | 367.127 | 422.037 | 15,898 / 37.670 | | 490K, four requests | 236.678 | 329.010 | 15,127 / 45.977 | Each value is one batch observation. Aggregate means all output tokens divided by synchronized group makespan, including first-token waiting. Requests end at natural stop or a 4,096-token cap, so the arms produce different totals. These are contextual batch rates, with no repeated-run dispersion. The online arm records four-way overlap for 30.152 seconds at 128K and 24.850 seconds at 490K. The source does not establish four fully distinct, full-length resident prefix copies. Exact per-request input counts and cache sharing are missing in this summary. ![Penny Royal v2.5.0 aggregate rates at 128K and 490K, with distinct off and online-FP8 arms.](/assets/batch7-qwen-long-rates.webp) *One batch per cell. These aggregate rates include synchronized group wall time and different completion totals.* The matching online-FP8 C1 observations use exactly 1,024 output tokens: 195.634 tokens/s after the first token at 128K, and 172.644 at 490K. TTFT is 12.906 and 63.296 seconds respectively. Those long observations are single samples. The source explicitly says cold TTFT did not improve against its option-off samples. A higher generation rate therefore need not shorten the complete request by the same proportion. Several release-state changes also accompany the precision option, preventing an isolated FP8 causal claim. ## The newer release is a separate experiment October 10 [Penny Royal 3.0 release metadata](https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.json) identifies another checkpoint, adaptive MTP and a 1,067,072-token pool. Its cold inputs are exactly 63,921 and 489,915 tokens; reported native mean prefill is 17,235.90 and 8,839.75 tokens/s, with TTFT 4.71 and 60.37 seconds. Three-key recall passes in the longer input. Completion length for those cold cases is not supplied here. The 3.0 regular screens instead use 128 input and 1,024 output tokens, three repeats, temperature 0.8, top-p 0.95, top-k 40, min-p 0.05 and xhigh reasoning. Native means are 218.63 tokens/s at C1 and 489.35 aggregate at C4. They weight complete active reporting intervals, excluding idle/prefill-crossing intervals and partial tails. They do not measure complete agent-task wall time or replace the older long C4 cells. The accepted public core and the local checkout with private diagnostics have separate full identities. A document commit, public release image and actual native execution are separate provenance fields. ## A four-card matrix provides a real campaign peak The [CS Technology Services original report](https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell) uses four RTX PRO 6000 Max-Q cards, TP4, PCIe Gen4 without NVLink, vLLM `0.1.dev20073+g8e685d198` and harness v0.4.29. At nominal 128K, C1/C8/C16 aggregate rates are 128.5/505.0/729.9 tokens/s. Each cell lasts 30 seconds with `ignore_eos=true` and an 8,192-token output cap. Effective concurrency was checked. C32 is omitted for capacity, not an observed crash. Its 374,976 MiB sampled peak is summed used VRAM across the **whole matrix**, over 501 samples in 1,206 seconds. Per-cell peak and host peak RAM are absent. Matrix TTFT is warm-prefix/queue time; cold C1 takes 24.25 seconds at 128,021 actual input tokens. Per-user rates in that report are derived by division. Image digest, weight revision and full launch command were not retained. This is a different deployment, without a controlled comparison to Penny Royal. ![Do not attach a peak from one campaign to an unrelated latency cell.](/assets/batch7-qwen-memory-scope.webp) *GiB and MiB are retained. Free, used, KV occupancy and host RAM are different quantities.* Penny Royal's 7.52 GiB free after graph capture is startup headroom. Its 1,897 MiB sampled minimum free belongs to ten GPU-media checks. Neither is the long C4 peak used memory. Keep free versus used, one versus four cards, startup versus workload and GPU versus host measurements separate. Pool occupancy is another counter again; percentages cannot be converted into whole-process bytes without an established mapping. ## Sustained submissions can still queue The [credited community follow-up](https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md) describes a separate operator's v2.5.0 configuration with a 1,114,304-token pool. Ten rounds submit four approximately 240K requests with 8,192 reported output tokens each. The author reports 93–107 seconds per round, 40/40 HTTP 200 and no OOMs, retractions or aborts. Queuing is observed and pool occupancy peaks at 57 percent. This supports sustained operation for that workload; it does not establish four running throughout, near-full-pool stress, exact retrieval correctness, peak RAM or a changed runtime default. The approximate input label remains approximate. ## A state-cache failure below the request ceiling An [original SGLang incident comment](https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113) records approximately 140K cold input, a 262K-class window, FP8 E4M3 KV, no speculation, 4,096-token prefill chunks, 51 Mamba slots and a 12-request ceiling. HiCache uses write-through. Only two requests are running when Mamba usage jumps from 0.18 to 1.00 while full-token usage is 0.46. The scheduler raises `Can not alloc mamba cache`; its child exits and the container reload interrupts service. The reporter mentions one queued request, while the cited failure log shows zero at that instant. Both observations are retained. ![Original SGLang incident: approximately 140K cold prompt, HiCache write-through.](/assets/batch7-qwen-mamba-failure.webp) *Source-reported outage. The proposed fallback and CPU tests do not prove every GPU workload fixed.* The related [PR37619](https://github.com/sgl-project/sglang/pull/37619) remains open and unmerged at this check. Its author reports two CPU tests and no rerun of the GPU reproducer. The proposed fallback skips an optional unfinished checkpoint when slots cannot be allocated. That scope does not establish a universal fix or a new speed result. Request slots, KV capacity and held Mamba states are separate constraints. For deployment selection, first match hardware, exact checkpoint, runtime, cache policy and requested output. Then ask whether the source observes simultaneous execution and measures memory over the same interval. Preserve a null where that evidence is missing. This produces a more useful capacity decision than combining the fastest token rate, largest configured window and smallest memory number from different runs. ## Original sources and conditions - jpezzulli: https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md - Document revision: ca9be12f9be920e916d02954a41702a981b18823 - SHA-256: 51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d - jpezzulli: https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/README.md - Document revision: ca9be12f9be920e916d02954a41702a981b18823 - SHA-256: 7645e7297eb1326393e8e4106fb73334980dfe44540b45e51655bc1dadd295ad - jpezzulli: https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md - Document revision: ca9be12f9be920e916d02954a41702a981b18823 - SHA-256: 2bbe9428f0701dc597dd0417193c12e72157dff2aca3ee6481b3104755a1301c - jpezzulli: https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.json - Document revision: ca9be12f9be920e916d02954a41702a981b18823 - SHA-256: 1799eca8496bf143fc48bbdc83a4f6ccc34b8773c43083b47e29721b1ceb3dcf - jpezzulli: https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.2.json - Document revision: ca9be12f9be920e916d02954a41702a981b18823 - SHA-256: 4843d621fa7f735a4c95d6bc79c9582d465cfaecef6b473689158d8c190920e2 - CS Technology Services: https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell - Document revision: snapshot-2026-10-18 - SHA-256: b81878ba45f2f6b7a6de06b0bf930bd1dfa3cb497202ed3fcd368efe66065930 - leehom0123: https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113 - Document revision: comment-5606493113 - SHA-256: d36cc21fa7b9577474ba20c9fe26becbb3c61bab9b9c4a5c421447b9d905c3f4 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "128K C4 option off", "metric": "aggregate_output_rate", "value": 367.127, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K four-request three-key retrieval", "input_length_source_label": "128K", "concurrency": 4, "precision_option": "option off", "actual_output_tokens_group": 15725, "group_wall_seconds": 42.833, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": null, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 231, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L231", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "128K C4 online FP8", "metric": "aggregate_output_rate", "value": 422.037, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K four-request three-key retrieval", "input_length_source_label": "128K", "concurrency": 4, "precision_option": "online FP8", "actual_output_tokens_group": 15898, "group_wall_seconds": 37.67, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": 30.152, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 231, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L231", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "490K C4 option off", "metric": "aggregate_output_rate", "value": 236.678, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K four-request three-key retrieval", "input_length_source_label": "490K", "concurrency": 4, "precision_option": "option off", "actual_output_tokens_group": 15455, "group_wall_seconds": 65.3, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": null, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 232, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L232", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "490K C4 online FP8", "metric": "aggregate_output_rate", "value": 329.01, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K four-request three-key retrieval", "input_length_source_label": "490K", "concurrency": 4, "precision_option": "online FP8", "actual_output_tokens_group": 15127, "group_wall_seconds": 45.977, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": 24.85, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 232, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L232", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "128K online FP8 C1 decode_rate", "metric": "decode_rate", "value": 195.634, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "128K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 229, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L229", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "128K online FP8 C1 ttft", "metric": "ttft", "value": 12.906, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "128K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 229, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L229", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "490K online FP8 C1 decode_rate", "metric": "decode_rate", "value": 172.644, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "490K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 230, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L230", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "490K online FP8 C1 ttft", "metric": "ttft", "value": 63.296, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "490K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 230, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L230", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "Post-graph free VRAM", "metric": "available_GPU_memory", "value": 7.52, "unit": "GiB", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "concurrency": null, "memory_kind": "post_graph_available_not_peak_used", "memory_cohort": "startup", "paired_long_text_batch_peak": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 243, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L243", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "Media-window minimum free VRAM", "metric": "available_GPU_memory", "value": 1897, "unit": "MiB", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "concurrency": null, "memory_kind": "sampled_minimum_free_not_per_long_text_batch_peak", "memory_cohort": "ten selected GPU-media checks; ordinary/large JPEG, concurrent images and video history", "paired_long_text_batch_peak": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 263, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L263", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 cold 64K prefill_rate", "metric": "prefill_rate", "value": 17235.9, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 63921, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "native mean input rate" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 44, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L44", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 cold 64K ttft", "metric": "ttft", "value": 4.71, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 63921, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "client first-token latency" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 44, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L44", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 cold 490K prefill_rate", "metric": "prefill_rate", "value": 8839.75, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 489915, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "native mean input rate" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 45, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L45", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 cold 490K ttft", "metric": "ttft", "value": 60.37, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 489915, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "client first-token latency" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 45, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L45", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 short C1 native mean", "metric": "native_decode_rate", "value": 218.63, "unit": "tokens/s", "statistic": "native mean; three regular repeats, no SD", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": 0.8, "top_p": 0.95, "thinking": "xhigh", "repetitions": 3, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 128, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "regular warmed screens", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "top_k": 40, "min_p": 0.05, "metric_denominator": "elapsed-weighted complete active native log intervals; excludes idle/prefill crossing and partial tail; aggregate if C4" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 26, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L26", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 short C4 native mean", "metric": "native_decode_rate", "value": 489.35, "unit": "tokens/s", "statistic": "native mean; three regular repeats, no SD", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": 0.8, "top_p": 0.95, "thinking": "xhigh", "repetitions": 3, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 128, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "regular warmed screens", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 4, "top_k": 40, "min_p": 0.05, "metric_denominator": "elapsed-weighted complete active native log intervals; excludes idle/prefill crossing and partial tail; aggregate if C4" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 27, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L27", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" } ], "conditions": { "independent_site_runs": 0, "controlled_cross_version_speedup": false, "per_cell_peak_RAM_known": false }, "case_reports": [ { "id": "qwen-cstech-128k-matrix", "classification": "source_reported_duration_matrix", "source_url": "https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell", "source_snapshot_sha256": "b81878ba45f2f6b7a6de06b0bf930bd1dfa3cb497202ed3fcd368efe66065930", "public_URL_is_immutable": false, "source_published_on": "2026-08-26", "model": "Qwen/Qwen3.8-Flash-Next-FP8", "model_revision": null, "engine": "vLLM 0.1.dev20073+g8e685d198", "executed_image_digest": null, "runtime_execution_revision": null, "harness": "llm-inference-bench v0.4.29", "system": "4 RTX PRO 6000 Blackwell Max-Q, 300 W/card, Threadripper, PCIe Gen4, no NVLink", "parallelism": "TP4", "context_limit_tokens": 262144, "KV_pool_tokens": 2662752, "configured_max_requests": 256, "draft_tokens": 3, "temperature": null, "top_p": null, "thinking": null, "input_length_source_label": "128k nominal matrix context; exact per-cell prompt token counts unreported", "duration_seconds_per_cell": 30, "output_tokens_cap": 8192, "ignore_eos": true, "effective_concurrency_validated_by_source": true, "cache_state": "duration-mode integrated scout warms shared prefix; matrix-start TTFT includes cache hit and queueing", "cells": [ { "concurrency": 1, "aggregate_output_tokens_per_second": 128.5, "TTFT_seconds": 1.41 }, { "concurrency": 8, "aggregate_output_tokens_per_second": 505.0, "TTFT_seconds": 2.02 }, { "concurrency": 16, "aggregate_output_tokens_per_second": 729.9, "TTFT_seconds": 3.42 } ], "omitted_cell": { "concurrency": 32, "status": "not_run_capacity", "aggregate_output_tokens_per_second": null, "observed_runtime_crash": false }, "memory": { "kind": "sampled_peak_summed_GPU_used_over_entire_matrix", "value": 374976, "unit": "MiB", "sample_count": 501, "campaign_seconds": 1206, "paired_per_cell_peak": null, "host_peak_RAM": null }, "cold_C1": { "actual_input_tokens": 128021, "TTFT_seconds": 24.25, "prefill_tokens_per_second": 5279 }, "per_user_table_is_derived_aggregate_divided_by_streams": true }, { "id": "qwen-mamba-long-prefill-crash", "classification": "source_reported_scheduler_assertion", "source_url": "https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113", "source_snapshot_sha256": "d36cc21fa7b9577474ba20c9fe26becbb3c61bab9b9c4a5c421447b9d905c3f4", "source_updated_at": "2026-09-09T18:05:35Z", "public_URL_is_immutable": false, "model": "Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "engine": "SGLang 0.0.0.dev1+g4ccff141d", "runtime_execution_revision": null, "executed_image_digest": null, "weight_precision": "modelopt_mixed", "kv_dtype": "fp8_e4m3", "context_length_source_label": "262k", "actual_prompt_tokens": null, "prompt_length_source_label": "approximately 140k", "chunked_prefill_size": 4096, "mamba_track_interval": 64, "mamba_slots": 51, "configured_max_requests": 12, "observed_running_requests": 2, "observed_queued_requests_reporter": 1, "queued_requests_at_cited_log_line": 0, "mamba_usage_before": 0.18, "mamba_usage_at_failure": 1.0, "full_token_usage_at_failure": 0.46, "HiCache": true, "hicache_ratio": 2.0, "write_policy": "write_through", "speculation": false, "error": "AssertionError: Can not alloc mamba cache", "outcome": "scheduler child exited; container auto-restarted, service outage during reload", "fix_reference": "https://github.com/sgl-project/sglang/pull/37619", "fix_PR_state": "open", "fix_PR_merged": false, "fix_PR_head_revision": "cd530ac217ac0a67b913f7083679c0890670d228", "author_GPU_reproducer_rerun": false, "universal_fix_verified": false, "peak_RAM": null } ], "memory_observations": [ { "kind": "sampled_peak_summed_GPU_used_over_entire_matrix", "value": 374976, "unit": "MiB", "sample_count": 501, "campaign_seconds": 1206, "paired_per_cell_peak": null, "host_peak_RAM": null }, { "kind": "v2.5.0_GPU_media_window_minimum_free", "value": 1897, "unit": "MiB", "long_text_C4_peak": null, "source_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md" } ], "long_context_cases": [ { "classification": "source_reported_four_submitted_stress", "recipe_release": "v2.5.0", "contributor": "BESTIA / Prometeus field report; separate contributor from runtime author", "source_url": "https://www.reddit.com/r/BlackwellPerformance/comments/1weaxdz/comment/p9i76fb/", "source_summary_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md", "context_limit_tokens": 524288, "KV_pool_tokens": 1114304, "input_tokens": null, "input_length_source_label": "approximately 240K", "submitted_concurrency": 4, "simultaneous_running_concurrency": null, "rounds": 10, "total_requests": 40, "output_tokens_reported": 8192, "round_seconds_range": [ 93, 107 ], "HTTP_200": 40, "OOMs_reported": 0, "retractions_reported": 0, "aborts_reported": 0, "peak_pool_occupancy_percent": 57, "peak_memory_bytes": null, "queuing_observed": true, "retrieval_correctness": null } ] } ``` --- # GLM-5.3-Flash on Two Sparks: Speed Meets Quality Gates ![Two DGX Sparks / NVIDIA NVFP4 artifact / Marlin W4A16 compute / text only.](/assets/batch7-glm-hero.webp) *Four sustained prompts per arm, one boot per arm. No production-readiness score.* A deployment can finish long requests and generate faster while still failing the contract an application needs. Weschera's [GLM-5.3-Flash experiment](https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/README.md) is useful because it preserves that outcome. On two DGX Sparks, a seven-token DFlash2 arm improves the source's median estimated decode rate, but strict output failures remain. A faster three-token diagnostic is rejected before code execution. None of the three arms receives a clean overall quality pass. These are original community observations, not new Run AI Here measurements. ## Pin the target, draft and execution environment separately The target is `nvidia/GLM-5.3-Flash-NVFP4` at `423acf37583782c51c142d145aef733d72943d93`. The DFlash2 draft is a different artifact at `bf582e4eacc1810f76656d1811693ff6c6737d2a`. The community vLLM image has a recorded digest ending `abe85a6`; its complete digest is retained in the adjacent JSON and original manifest. A document pin does not replace any of these identities. The [source manifest](https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/SOURCE_MANIFEST.json) and [reproduction document](https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/REPRODUCE.md) describe two 128 GB unified-memory GB10 nodes using TP2 over RoCE. NVIDIA NVFP4 is the storage format, while Marlin uses W4A16 MoE compute. Do not describe the result as a native W4A4 throughput test. The target export lacks an MTP head; DFlash2 is the tested speculation path. The report is text only, with image and video inputs disabled. The selected setup uses eager execution, an explicit 6 GiB FP8 E4M3 KV allocation, a 131,072-token window, one request slot, a 128-token prefill batch and disabled prefix caching. An overlay supplies Transformers 5.16.1 and tokenizers 0.23.1. These details make a useful reproduction inventory. They do not establish portability to a stock upstream installation or a second GB10 software stack, and this is a community recipe rather than an official NVIDIA production qualification. ## Read the speed denominator before the headline The [saved comparison JSON](https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json) includes four naturally stopping sustained requests per arm: two English technical tutorials and two Korean essays. Prompt counts are 78 or 92 tokens. Outputs vary greatly, rather than holding one identical response length constant. The saved English request uses temperature zero, top-p one and seed 37. Its output cap is the remaining serving window. The source does not accept a context-limit finish as a natural benchmark stop. | Arm | Draft tokens | Median estimated decode, tokens/s | Overall quality gate | |---|---:|---:|---| | Baseline | 0 | 15.18 | false | | DFlash2 selected | 7 | 20.50 | false | | DFlash2 diagnostic | 3 | 24.78 | false | Estimated decode divides completion tokens by wall time after subtracting measured TTFT. It is different from end-to-end request throughput. There is one boot per arm and four different prompts, not four independent boot replications of an identical task. No uncertainty interval is supplied. Ratios of those medians describe these outputs under these settings; they do not establish a universal improvement or the same reduction in application completion time. ![Three GLM decode medians are shown alongside a false overall quality flag for every arm.](/assets/batch7-glm-speed-gates.webp) *The selected depth-7 arm passed executed semantic checks but retained two strict output-format failures.* For example, the baseline English outputs contain 4,356 and 10,853 tokens, while the depth-7 English outputs contain 7,396 and 2,292. The amount and shape of generated work differ. Those facts matter when deciding whether a faster token stream improves a particular application. Short ACK latency, sustained prose generation, code correctness and long retrieval should remain separate checks. ## Preserve the selected arm's strict failures The selected depth-7 arm passes the executed semantic checks, including the source's fixed and generated code cases. It nevertheless fails two strict requirements. One probe asks for Python without a Markdown fence and receives fenced code. A separate thinking-on probe requires final content exactly `703` and receives reasoning plus the answer. These are observable response-contract failures. The arithmetic answer being present does not turn the strict final-content requirement into a pass. All arms have `quality_passed=false` in the raw comparison. A selected serving candidate is therefore not synonymous with a fully qualified deployment. The distinction matters for JSON consumers, code runners and tools that accept a constrained output format. An editor should not strip unwanted material, run the repaired answer and retrospectively mark the original response compliant. A downstream repair workflow would need its own explicitly defined evaluation. ## Why the fastest arm was rejected The baseline and depth-3 diagnostic fail the early one-code-block contract in the short quality gate. The source stops before later code execution checks in that branch. This means later checks are **skipped**, not that the underlying algorithm was measured and proven wrong. The failed response structure is enough to reject that arm for the source's contract, but it does not justify a broader claim about every code answer from the model. Likewise, the comparison's `semantic_checks_passed=false` must be read with the detailed gate. It is a campaign outcome containing an early contract failure. A single Boolean cannot explain which tests executed. The adjacent JSON retains both the flag and strict-failure descriptions, so people and AI can inspect the actual reason rather than infer it from a speed ranking. ![120,124 actual input tokens under a 131,072-token configured window.](/assets/batch7-glm-context.webp) *All three arms recovered three markers; this is neither full-window proof nor broad quality equivalence.* ## The long input is an exact retrieval check All three arms recover beginning, middle and end markers from 120,124 actual input tokens. The configured window is 131,072. The check demonstrates that particular retrieval under that serving configuration; it does not prove a completed full-window input, arbitrary document reasoning or concurrent long-context admission. These requests run at C1, and no higher-concurrency performance sweep is supplied. The 6 GiB KV allocation is a configuration value. It is not whole-model peak unified memory. The source does not provide a paired total peak-RAM trace for every sustained or long-context request. Weight-file bytes are another quantity and should not be added to an allocation to invent an observed peak. Peak memory, repeated-boot dispersion and concurrency scaling stay unknown in this interpretation. ## Treat the custom thinking template as its own condition The report's forced-OFF template uses an empty thinking prefix while retaining maximum-effort system wording. Planning can leak into final text. The author therefore does not qualify it as native thinking OFF. The separate thinking-on strict failure should not be attributed to that template without an isolated comparison. Output structure, reasoning controls and draft depth vary across the probes, and the saved failures do not identify one universal cause. ![A pinned NVIDIA artifact does not make a community GB10 recipe an official NVIDIA runtime.](/assets/batch7-glm-identity.webp) *DFlash2 draft revision, target revision and image digest remain separate source fields.* Finally, the selected server's restart is followed by a health probe. That is a service-readiness observation, not a repeated quality suite on the final boot. Anyone reusing this recipe should retain the target revision, draft revision, image digest, template and overlay in their own inventory, then validate the response contracts they actually need. The published experiment gives concrete starting conditions and useful failures; it leaves production readiness, broader quality and workload-specific peak memory to further measured evidence. ## Original sources and conditions - Weschera: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/README.md - Document revision: 281dc9a982bca17e944b322746658250a670ae73 - SHA-256: 76327764bf18694718fbfac4d253c5525bf98e0b1d8d914f35765acfd34904cf - Weschera: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/docs/RESULTS.md - Document revision: 281dc9a982bca17e944b322746658250a670ae73 - SHA-256: ff3e50dd4f7a65232e37c01e4dd79ca5bee10c9ba66dbbec0f2cf1a7b5a4c3d5 - Weschera: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/REPRODUCE.md - Document revision: 281dc9a982bca17e944b322746658250a670ae73 - SHA-256: 89dd3e9334b41901f2ae8dc1d92ec8437e2ed90c143e4026c28c0978242f13c2 - Weschera: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/SOURCE_MANIFEST.json - Document revision: 281dc9a982bca17e944b322746658250a670ae73 - SHA-256: 7065d7de93ef78feb6023048fefde4506c097148cfc2733bfe65257bcbac1692 - Weschera: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json - Document revision: 281dc9a982bca17e944b322746658250a670ae73 - SHA-256: 36dc46ce255d484340558a64b6b45ce2b79f644471d91cbe6dc0e7a82de2d461 - Weschera: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/dflash2-7/quality-report.json - Document revision: 281dc9a982bca17e944b322746658250a670ae73 - SHA-256: 239adbba20e74bd853f04af15301d6c5b4254512b5ee9e4b79a532b64bac0957 - Weschera: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/dflash2-7/speed-prose-0-request.json - Document revision: 281dc9a982bca17e944b322746658250a670ae73 - SHA-256: 46e917cee3ce542c4b710c92cdb0f22fa98281177d0405f94e57af3f6fd89ff6 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "baseline median decode", "metric": "decode_rate", "value": 15.181243819412654, "unit": "tokens/s", "statistic": "median of four different prompts in one arm boot; not repeated identical task", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 0, "drafter": null, "drafter_revision": null, "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 4356, 2355, 10853, 2238 ], "all_speed_requests_natural_stop": true, "quality_passed": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-weschera-extra3", "source_document_sha256": "36dc46ce255d484340558a64b6b45ce2b79f644471d91cbe6dc0e7a82de2d461", "source_line": 18, "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L18", "source_record_id": "glm53-flash-nvfp4-dflash-quality" }, { "label": "dflash2-7 median decode", "metric": "decode_rate", "value": 20.50006471628756, "unit": "tokens/s", "statistic": "median of four different prompts in one arm boot; not repeated identical task", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 7, "drafter": "incoai/GLM-5.3-Flash-DFlash2", "drafter_revision": "bf582e4eacc1810f76656d1811693ff6c6737d2a", "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 7396, 2272, 2292, 2158 ], "all_speed_requests_natural_stop": true, "quality_passed": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-weschera-extra3", "source_document_sha256": "36dc46ce255d484340558a64b6b45ce2b79f644471d91cbe6dc0e7a82de2d461", "source_line": 98, "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L98", "source_record_id": "glm53-flash-nvfp4-dflash-quality" }, { "label": "dflash2-3 median decode", "metric": "decode_rate", "value": 24.77753188791761, "unit": "tokens/s", "statistic": "median of four different prompts in one arm boot; not repeated identical task", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 3, "drafter": "incoai/GLM-5.3-Flash-DFlash2", "drafter_revision": "bf582e4eacc1810f76656d1811693ff6c6737d2a", "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 2822, 2194, 6331, 2105 ], "all_speed_requests_natural_stop": true, "quality_passed": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-weschera-extra3", "source_document_sha256": "36dc46ce255d484340558a64b6b45ce2b79f644471d91cbe6dc0e7a82de2d461", "source_line": 173, "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L173", "source_record_id": "glm53-flash-nvfp4-dflash-quality" } ], "conditions": { "independent_site_runs": 0, "production_ready": false, "native_thinking_OFF_qualified": false }, "quality_comparisons": [ { "variant": "baseline", "speculative_tokens": 0, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": false, "long_context_passed": true, "strict_failures": [ { "probe": "short-quality", "category": "semantic", "passed": false, "error": "Expected one code block" } ] }, { "variant": "dflash2-7", "speculative_tokens": 7, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": true, "long_context_passed": true, "strict_failures": [ { "probe": "code", "requirement": "Only Python code, no Markdown fence", "passed": false }, { "probe": "thinking-on", "requirement": "Final content exactly 703", "passed": false, "observed": "**Reasoning:** The sum of the first n positive integers is given by the formula n(n+1)/2. For n = 37:\n\nSum = 37 × 38 / 2 = 37 × 19 = 703\n\n**Answer:** 703" } ] }, { "variant": "dflash2-3", "speculative_tokens": 3, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": false, "long_context_passed": true, "strict_failures": [ { "probe": "short-quality", "category": "semantic", "passed": false, "error": "Expected one code block" } ] } ], "memory_observations": [ { "kind": "explicit_KV_allocation", "value": 6, "unit": "GiB", "whole_model_peak_RAM": null } ], "long_context_cases": [ { "classification": "source_reported_three_marker_success", "actual_input_tokens": 120124, "configured_limit_tokens": 131072, "arms": [ "baseline", "dflash2-7", "dflash2-3" ], "exact_all_three_markers": true, "full_window_tested": false, "general_long_context_reasoning_proven": false } ] } ``` --- # MiniMax-M3 on Two Sparks: The Measured W4A16 Lane ![July 5, 2026 measurements / W4A16 GPTQ / EAGLE3 depth 3 / b12x NVFP4 KV.](/assets/batch7-minimax-hero.webp) *The model family is current; these particular tests retain their July date.* MiniMax-M3 has several community deployment recipes, but their evidence should not be collapsed into a single “best” configuration. The [tonyd2wild report](https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/README.md) distinguishes a lane independently reproduced on its own two Sparks from two alternative lanes still under test. This note retains that distinction, the original July measurement date, short-probe timing and a negative transport experiment. It offers a concrete deployment reference without turning unfinished recipes into completed benchmarks. ## Keep the measurement date attached to the model Lane A is verified on July 5, 2026. Reading it today as a MiniMax-family deployment reference does not make its results October measurements. Its hardware is two DGX Sparks with 128 GB unified memory each, connected by direct 200G RoCE and serving through TP2. This is distinct from a three-Spark layer split or an offload configuration on a discrete GPU workstation. The target is `Sebesky/MiniMax-M3-W4A16-GPTQ`, with a separate `Sebesky/MiniMax-M3-EAGLE3-RTN-INT4` draft. The report describes the full, unpruned model and a vLLM+b12x path. It names an execution image tag and a short source reference, but does not retain a full executed image digest or full target/draft weight revisions. Those fields remain null. Looking up a current model repository today would not establish which exact files ran in July. The source credits a3refaat's upstream recipe and clearly labels where its own reproduction begins. Attribution is important here: a reference number quoted from the upstream author is not an additional independent measurement by the second reporter. The catalog points below are restricted to the reporter's measured lane A. ## The measured lane uses one stream The [default-configuration document](https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/DEFAULT-CONFIG.md) selects EAGLE3 depth three, b12x NVFP4 KV, a 196,608-token configured context and a 208,128-token KV pool. All published lane-A speed observations use `max_num_seqs=1`. A configured large context does not mean those short probes contain that many input tokens or test the whole window. The [original result document](https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md) specifies 256-token probes, temperature zero, one stream and a warm engine. The first request after boot is excluded because b12x compilation warmup can take about a minute. Actual per-probe input counts, completion counts and repeated-run dispersion are not supplied. Although the source table says “decode,” its stated denominator is `completion_tokens / wall`, including request wall time. The article and JSON keep that definition. | Measured workload | Thinking disabled, tokens/s | Adaptive, tokens/s | |---|---:|---:| | JSON | 36.6 | 35.4 | | Code | 31.8 | 29.2 | These four observations are useful for their own prompt classes. They are not a concurrency curve, a full-context curve or a confidence interval. Thinking disabled and adaptive can also produce different work, so the rate difference alone does not establish the same change in complete task time. No general quality score accompanies this experiment. ![MiniMax-M3 request rates for JSON and code with thinking off and adaptive modes.](/assets/batch7-minimax-rates.webp) *The denominator is whole request wall time, despite the source table calling the values decode speed.* For a latency-sensitive application, the next relevant evidence would record actual input and completion lengths, first-token latency, the complete request time and successful output checks under the intended mode. The existing table cannot supply those missing measurements. Keeping them explicit is more useful than filling them from another MiniMax report on a different engine or date. ## The allocation bracket explains a reproducibility trap The reporter finds that GMU 0.93 fails a startup free-memory check by 0.02 GiB. Reducing to 0.925 starts the engine but leaves a KV pool below the desired context requirement. The chosen 0.929 reaches the target pool with approximately 100 MB startup margin. The source preserves the bracket because one seemingly reasonable rounded parameter can change whether the recipe starts and whether it has enough configured capacity. That margin is measured at startup in the reporter's boot state. It is not a sampled worst-case workload margin and does not show safe headroom for simultaneous long prefills. File-cache state, other resident processes and temporary runtime allocations can differ across systems. The published bracket does not justify a universal GMU recommendation or a guarantee that the same fraction works on every Spark pair. ![GMU brackets are observations from the reporter's own boot state.](/assets/batch7-minimax-capacity.webp) *The approximately 100 MB startup margin is not a workload safety allowance.* The 208,128-token pool is measured/configured KV capacity, in tokens. The 196,608-token ceiling is a per-request setting. They are not memory in bytes, and neither value is whole-model peak unified RAM. The source does not provide a paired peak-memory time series for a long request or multi-user load. Those unknowns remain null rather than being estimated from weight-file size or a nominal machine total. ## A transport change that made this workload slower The report tests `NCCL_PROTO=LL` with `NCCL_MAX_NCHANNELS=2`. In the code workload, stock NCCL gives 31.8 tokens/s and the LL/two-channel setting gives 28.8. The JSON workload also declines in the source's observed range. This negative result is retained rather than selecting only the fastest headline. The author suggests that a larger MoE's communication pattern may explain why a setting useful on another model loses here. That is a hypothesis, not an isolated mechanism test. The evidence supports a narrower conclusion: this exact change did not improve the tested lane-A workloads on this pair. It does not identify a universally best NCCL protocol, prove a model-class law or replace measurements on another interconnect. ![Lane A, stock NCCL versus LL protocol plus two channels.](/assets/batch7-minimax-negative.webp) *Source hypothesis about model class is interpretation, not a separately isolated mechanism.* ## Why the other two lanes stay references Lane C proposes a different KVarN KV path plus EAGLE3 for speed, and lane B proposes a drafterless context-oriented path. Their numbers in this repository are attributed to the upstream author's own hardware. The second reporter labels both lanes under test and leaves its own result checkboxes incomplete. This article does not publish their quoted speeds as reproduced lane-A data or add them to the selected result points. The same restriction applies to proposed concurrency and repetition-penalty experiments. A checklist item is a plan. It cannot be turned into a success case because the surrounding recipe is detailed. Any later measured lane should retain its own boot evidence, KV pool, exact runtime identity, prompts, output limits, timing definitions and validation outcomes before joining this series. For deployment selection, lane A is the supported starting reference in this source. Use its original reproduction inventory to understand the required kernel path, two-node transport and draft model, then account for missing identities and workload telemetry. The useful result is a bounded, reproducible question: whether the same lane satisfies a specified application's response and memory requirements. The published four short rates alone cannot answer the broader questions of maximum concurrency, long-context reliability, peak memory or standardized model quality. ## Original sources and conditions - tonyd2wild: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/README.md - Document revision: a8101fa423cac8140016e45110a2ccc8a0a72162 - SHA-256: 98f2ec691a993fb8f143ca6f31d19ad9c2090de979e577b8a4dfbc4c8039a6b2 - tonyd2wild: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/DEFAULT-CONFIG.md - Document revision: a8101fa423cac8140016e45110a2ccc8a0a72162 - SHA-256: 04d26375babfc741c02dac6d1d8c0f0004152284f4843c2ca1efe997d2282287 - tonyd2wild: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md - Document revision: a8101fa423cac8140016e45110a2ccc8a0a72162 - SHA-256: 9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "JSON thinking False", "metric": "request_output_rate", "value": 36.6, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "JSON", "thinking": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 38, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L38", "source_record_id": "minimax-m3-tp2-lane-a" }, { "label": "JSON thinking adaptive", "metric": "request_output_rate", "value": 35.4, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "JSON", "thinking": "adaptive" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 38, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L38", "source_record_id": "minimax-m3-tp2-lane-a" }, { "label": "code thinking False", "metric": "request_output_rate", "value": 31.8, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "code", "thinking": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 39, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L39", "source_record_id": "minimax-m3-tp2-lane-a" }, { "label": "code thinking adaptive", "metric": "request_output_rate", "value": 29.2, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "code", "thinking": "adaptive" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 39, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L39", "source_record_id": "minimax-m3-tp2-lane-a" }, { "label": "Lane A KV pool", "metric": "KV_pool", "value": 208128, "unit": "tokens", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "thinking": null }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 31, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L31", "source_record_id": "minimax-m3-tp2-lane-a" } ], "conditions": { "independent_site_runs": 0, "measurement_date": "2026-07-05", "only_lane_A_reproduced_by_this_reporter": true, "concurrency_measured_above_one": false }, "case_reports": [ { "id": "minimax-lane-a-startup-bracket", "classification": "source_reported_startup_capacity_failure", "GMU_failed": 0.93, "shortfall_GiB": 0.02, "GMU_boots_but_KV_below_requested": 0.925, "GMU_selected": 0.929, "startup_margin_source_label": "approximately 100 MB", "runtime_peak_RAM": null, "source_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/DEFAULT-CONFIG.md" } ], "memory_observations": [ { "kind": "KV_pool_tokens_not_peak_RAM", "value": 208128, "unit": "tokens", "configured_limit_tokens": 196608, "whole_model_peak_RAM": null } ] } ``` --- # Kimi IQ1_S on Three Sparks: Scores, Retries and Missing Pins ![Publisher-labelled K3 / three Spark layer-split RPC / exact upstream checkpoint unpinned.](/assets/batch7-kimi-hero.webp) *Small source subsets and unequal cloud settings do not establish lossless quantization.* Small local quality reports can be informative when they preserve the questions tested, failure history and sampling controls. They become misleading when a composite score is presented as a first-pass result or matching percentages become a claim of lossless quantization. The [chishiki37 report](https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md) provides deployment details and raw subset results for a publisher-labelled Kimi K3 IQ1_S on three Sparks. This note keeps its useful observations and the limits visible together. ## Start with the identity that is actually known The August 7 report calls the model Kimi K3/Kimi-Linear IQ1_S, with nine GGUF shards and an approximately 330 GB file total. It names a shard pattern but does not pin the exact upstream repository revision, weight digest or complete runtime commit. Run AI Here therefore retains the publisher's model label without certifying that it uniquely identifies a current official checkpoint. File size and a plausible filename cannot establish that identity. Three DGX Sparks supply 128 GB unified memory each. A pwilkin llama.cpp fork uses layer-split RPC with two remote workers. This is not tensor parallelism across three GPUs. The fabric overview says MTU9000, while the actual deployment configuration says MTU4096, inline316 and receive-depth24. Those are distinct source statements, and the JSON preserves both instead of quietly choosing the larger value. The configuration uses f16 KV, load mode `none` and one request slot. The source reports that Q8 KV is incompatible with its V-head dimension and block constraint in this build. That is a specific runtime/artifact condition, not a claim that every Kimi model or future llama.cpp version lacks quantized KV support. The original launch line is a reproduction reference; this site has not executed it. ## A configured million-token window is not a measured full input The configured window is 1,048,576 tokens. The performance summary reports approximately 2.9 tokens/s steady decode and the following prefill values. Exact per-run prompt counts, output lengths, timing traces and repeated-run dispersion are not supplied in that table. | Original input label | Reported prefill, tokens/s | |---|---:| | 8K | 79.6 | | 32K | 43.9 | | 131K | 48.5 | ![Kimi IQ1_S source prefill rates at 8K, 32K and 131K nominal prompt lengths.](/assets/batch7-kimi-prefill.webp) *A configured 1,048,576-token window is separate from these performance observations.* The longest listed performance label is 131K. A source statement that a server loads with a 1M window is not enough to establish a completed 1,048,576-token input, correct recall throughout that input or acceptable concurrency at that length. The selected sources do not supply the corresponding full-input payload and paired telemetry. Those fields stay unknown rather than borrowing a test from a different Kimi configuration. ## Read the GSM8K score as a fixed subset The [local GSM8K raw summary](https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json) reports 49 correct of 50, or 98 percent, on the selected canonical test questions. The source's cloud Kimi and cloud GLM-5.2 comparisons also report 49/50 and the same failed item. These are source subsets, not full standardized leaderboard evaluations and not independent Run AI Here measurements. The controls are not matched across services. Local Kimi uses temperature0.2 and an 8,192-token output cap, while cloud Kimi forces temperature1 and uses a 2,048-token GSM8K cap. The GLM comparison raises its cap after two token-exhaustion failures. Exact API model versions and a matched unquantized local baseline are missing. Equal subset percentages under those conditions do not establish equivalence or isolate the quality cost of IQ1_S quantization. ## HumanEval: the first pass was 14/20 The [initial local HumanEval file](https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json) records 14 passes out of 20, or 70 percent. Four requests time out at the 600-second client limit: IDs1,6,10 and17. Two other tasks, IDs2 and7, fail assertions. Timeouts are retained as first-pass failures rather than silently removed from the denominator. The [timeout-only rerun](https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json) evaluates those four timed-out tasks with higher limits and passes all four. Its 4/4 result is a selected retry subset. Combined with the fourteen initial passes, it explains the README's final 18/20, or 90 percent. That composite is not a fresh first-pass evaluation of twenty problems and should never replace the initial 14/20 in a reliability discussion. ![Kimi local HumanEval subset rises from 14 of 20 initially to 18 of 20 after four timeout retries.](/assets/batch7-kimi-retry-scores.webp) *The four successful retries are a selected subset; final 18/20 must retain this history.* One retry takes 995.8 seconds and another 1,180.1. This matters operationally: raising a timeout can recover a correct answer while increasing how long a client or worker remains occupied. The source's final aggregate timing summary does not give a clean first-pass service-latency distribution. It is unsuitable for an application percentile claim, and the selected retries do not estimate the rate at which arbitrary future timeout failures recover. The remaining failures concern `truncate_number` and `filter_by_substring`. The README groups both under floating-point edge cases, but substring filtering is not itself a floating-point task. This article retains the assertion failures without adopting that blanket diagnosis. Raw failure IDs and errors are evidence; a common root cause would require further analysis. No upstream code or generated candidate was executed during this editorial review. ## Memory occupancy is approximate and lacks a peak method The source gives about 119–120 GB per node and about 357 GB total, with small reported headroom. It does not define a peak sampling method, sampling cadence, process-versus-host boundary or paired context length. The GB label also lacks a verified byte convention. Those observations are approximate occupancy reports, not a measured peak-RAM curve. ![The report labels approximately 119–120 GB per node without a telemetry method.](/assets/batch7-kimi-memory.webp) *Do not infer a 1M full-input pass, safe peak headroom or quantization equivalence.* The reported `vmstat` swap-in/out counters are zero during observation. That does not prove no swap was allocated, that no future workload can swap or that all three machines have safe transient headroom at the configured maximum. A warm-cache load time and a memory occupancy snapshot answer different questions. They should not be added to weight-file bytes or transformed into a universal hardware sizing guarantee. ## What these scores can help decide This report offers a concrete three-node deployment reference, a slow but measured generation path and a transparent small-suite failure history. It can help frame a reproduction: pin the actual artifact and runtime, retain per-task sampling and output caps, record timeout policy, and measure memory alongside the intended request lengths. It does not support a global ranking against cloud models or another quantization format. The useful score is therefore the one that keeps its denominator and history. Read 49/50 with the subset and unequal controls; read 18/20 with the initial 14/20 and selected four-task rerun. Keep configured context separate from completed input and approximate occupancy separate from peak memory. Those distinctions allow both readers and AI systems to use the original evidence without turning a limited experiment into a broader capability claim. ## Original sources and conditions - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 2d297e3f59c7beb393fa72e6297edec8af0b6525139057ee0e26b425d071f2be - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 29a942e4a2194c737b8492b59ab752015896fc8e3180cc85135c33c6b249f590 - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 469ff6a70092ca219c70e3dc2d6967729133f74e97ed0b9cd56e68323b2f2053 - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/cloud-humaneval-logic-k3-cloud.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 8d42fd9033e794d5a8a7330d5c56707128b521f869c46be071a364c5f45fa22c - chishiki37: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-cloud.json - Document revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - SHA-256: 636e0ee2c31986d961ffe637927bcb95aa1d3d82d78dea08f7efe0807f29971b Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "steady decode", "metric": "decode_rate", "value": 2.9, "unit": "tokens/s", "statistic": "approximate source summary", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "steady decode", "input_length_source_label": null }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 43, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L43", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "8K prefill", "metric": "prefill_rate", "value": 79.6, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "8K prefill", "input_length_source_label": "8K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 44, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L44", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "32K prefill", "metric": "prefill_rate", "value": 43.9, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "32K prefill", "input_length_source_label": "32K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 45, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L45", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "131K prefill", "metric": "prefill_rate", "value": 48.5, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "131K prefill", "input_length_source_label": "131K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 46, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L46", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "GSM8K local subset", "metric": "accuracy", "value": 98.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "GSM8K canonical test subset", "correct": 49, "total": 50, "subset_ids": "50 source-indexed questions; fixed subset, not full benchmark", "timeout_seconds": 1800, "retry_policy": "GLM comparison changed output cap; local retries unreported" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra0", "source_document_sha256": "2d297e3f59c7beb393fa72e6297edec8af0b6525139057ee0e26b425d071f2be", "source_line": 5, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json#L5", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "HumanEval initial local subset", "metric": "accuracy", "value": 70.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval subset IDs 0 through 19", "correct": 14, "total": 20, "subset_ids": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "timeout_seconds": 600, "timeouts": 4, "retry_policy": "initial run; four timeouts retained as failures" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra1", "source_document_sha256": "29a942e4a2194c737b8492b59ab752015896fc8e3180cc85135c33c6b249f590", "source_line": 6, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json#L6", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "HumanEval timeout retries", "metric": "accuracy", "value": 100.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval timeout-only subset", "correct": 4, "total": 4, "subset_ids": [ 1, 6, 10, 17 ], "timeout_seconds": null, "retry_policy": "rerun only four timed-out problems with higher limits; not fresh 20-problem pass" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra2", "source_document_sha256": "469ff6a70092ca219c70e3dc2d6967729133f74e97ed0b9cd56e68323b2f2053", "source_line": 6, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json#L6", "source_record_id": "kimi-k3-iq1s-three-spark-scores" }, { "label": "HumanEval local after selected retries", "metric": "accuracy", "value": 90.0, "unit": "percent", "statistic": "source composite score after retries", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval subset IDs 0 through 19", "correct": 18, "total": 20, "subset_ids": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "timeout_seconds": null, "retry_policy": "14 initial passes plus 4 successful timeout-only retries; not first-pass 18/20", "failed_ids": [ 2, 7 ] }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 83, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L83", "source_record_id": "kimi-k3-iq1s-three-spark-scores" } ], "conditions": { "independent_site_runs": 0, "full_1048576_input_test_proven": false, "unquantized_local_control": false, "sampling_matched_to_cloud": false, "quantization_equivalence_proven": false, "upstream_checkpoint_identity_verified": false }, "quality_comparisons": [ { "benchmark": "GSM8K", "subset_size": 50, "local_correct": 49, "cloud_Kimi_correct": 49, "cloud_GLM52_correct": 49, "local_temperature": 0.2, "cloud_Kimi_temperature": 1.0, "local_output_cap": 8192, "cloud_Kimi_output_cap": 2048, "GLM_cap_changed_after_exhaustion": true, "matched_accuracy_proves_equivalence": false }, { "benchmark": "HumanEval", "subset_size": 20, "initial_local_correct": 14, "initial_timeouts": 4, "retry_subset_ids": [ 1, 6, 10, 17 ], "retry_correct": 4, "final_local_correct": 18, "remaining_failure_ids": [ 2, 7 ], "first_pass_18_of_20": false } ], "memory_observations": [ { "kind": "approximate_per_node_occupancy_unknown_measurement_method", "source_range_GB": [ 119, 120 ], "unit": "GB (source label; byte convention unspecified)", "node_count": 3, "is_measured_peak": false, "peak_RAM": null } ], "long_context_cases": [ { "classification": "configured_context_not_completed_full_window", "configured_limit_tokens": 1048576, "longest_performance_source_label": "131K", "actual_full_window_prompt_tokens": null } ] } ``` --- # Qwen Long Concurrency: Paired Memory Peaks and OOM Recovery ![Qwen: the concurrent peak did not fall. Both trees report +1,248 MiB. This is a paired concurrent delta, not total GPU or host RAM.](/assets/batch8-qwen-memory-hero.webp) *Both trees report +1,248 MiB. This is a paired concurrent delta, not total GPU or host RAM.* The most useful memory result in this Qwen3.8-Flash-Next report is a negative one. A change that cuts an isolated allocation and a 500K single-request peak does **not** reduce the measured peak in its two-request, approximately 250K-per-request production-like cell. Keeping that result prevents a deployment guide from turning a kernel improvement into an unsupported concurrency-capacity promise. Run AI Here has read the original measurements; it has not run these model experiments. ## First separate the two physical cards The [maintainer README](https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/README.md) distinguishes a production/reference RTX PRO 6000 Blackwell **Max-Q Workstation**, 300 W, PCIe Gen4 x16 and driver 610.43.02, from a rented **Server Edition**, 600 W, PCIe Gen5 x16 and driver 580.95.05 used for later validation. Both report 97,887 MiB device capacity. The host-KV, bounded-gather and recovery validation belongs to the Server Edition. The September 30 production crash belongs to the Max-Q lane. A shared product-family name does not make their latency or failure margins interchangeable, and no controlled cross-SKU scaling result is established. The model is a community NVFP4 Qwen3.8-Flash-Next artifact with FP8 KV and host-resident components. The selected report does not give a complete upstream weight revision for these cells. Its patched SGLang branches and a base prefix are documented, but the exact full executed tree and image digest for every arm are missing. The pinned document revision identifies what was read, rather than silently supplying those execution identities. ## The concurrent memory controls answer two questions In [section 3.15b of the original benchmark ledger](https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md), a cold two-by-250K prefill compares on-GPU KV against host KV on the same validation card. The control auto-sizes to a 451,840-token pool; host KV uses 1,310,720. The control pool was not intentionally pinned to match the production lane. This matters because placement and capacity change together. | During the concurrent cold prefill | On-GPU control | Host KV | |---|---:|---:| | Minimum free device memory, MiB | 3,414 | 6,962 | | Peak non-pool allocation above idle, MiB | 1,090 | 1,248 | The host-KV arm has more minimum free VRAM while its non-pool increment is slightly larger. Those observations are compatible because they describe different accounting quantities. Neither column supplies peak whole-host RAM. Subtracting a free-memory observation from an unrelated device capacity or combining different idle baselines would create a result the author did not measure. ![Host KV: concurrent prefill headroom. Cold 2 x 250K prefill: minimum free VRAM, not peak host RAM. Source section 3.15b.](/assets/batch8-qwen-headroom.webp) *Cold 2 x 250K prefill: minimum free VRAM, not peak host RAM. Source section 3.15b.* A separate old-versus-bounded-gather A/B keeps the host-KV setup and uses two cold approximately 250K requests. Both trees report **+1,248 MiB non-pool peak over idle**. The author explicitly declines an end-to-end peak improvement for this cell. The likely explanation that another allocation dominates is left as a hypothesis; the report does not isolate that allocation. A planner should therefore retain the equal peak when estimating this workload's headroom. ## Why the 500K result can still improve Section 3.16 traces full-context prefill gather temporaries that grow with the entire request, even when prefill is chunked. In an isolated 524,288-token FP8 kernel microbenchmark, its own transient allocation falls from 1,024.0 to 32.0 MiB, excluding the output. Twelve cases in each of two runs are bit-exact with zero differing values. That is a kernel-level check, not proof that complete generated answers are deterministic or that every inference path is equivalent. The in-engine A/B instead holds a 689,728-token pool and 94,661 MiB idle device usage. At 500,000 input tokens, old and new device-used peaks are 97,237 and 96,093 MiB. All nine needle cases across three lengths and three depths pass in both arms. This supports a lower peak for that single-request sweep, while leaving the paired concurrent no-improvement result intact. The smallest kernel case and the wide-int64 case also show why “always 32 MiB” would overstate the implementation's behavior. ![500K: a different cell does improve. In-engine device peaks at 500,000 tokens. Both arms pass 9/9 needles over the full three-length sweep.](/assets/batch8-qwen-500k-peak.webp) *In-engine device peaks at 500,000 tokens. Both arms pass 9/9 needles over the full three-length sweep.* Host-UVA timing has an unresolved reproduction problem. The first new-path run takes 341–376 ms, whereas later runs take 4.99–5.68 ms and report 159–189 MiB peaks. A first-touch explanation is suggested but not proven. The source excludes a host-UVA kernel-time improvement claim, and so does this guide. ## Prefix loss can hurt without an HTTP error The same-card two-main-agent test sends histories labelled 250K and 200K, then adds two subagents on turn three. Host KV retains prefixes of 249,984 and 200,000 tokens; the control evicts both. Host-KV first-token times are 1.2 and 1.0 seconds, against 22.9 and 41.1 seconds in the control. Every control request still completes. This is evidence of a cache-capacity failure mode rather than an HTTP failure or a general throughput comparison. The source's 95.56 percent multi-turn cached fraction belongs to six conversations growing from about 60K to 80K, not to the 250K–500K main agents. Its two-520K host-KV case runs, but the on-GPU control cell does not execute. Keep that missing comparison instead of inventing a speedup. ## The real production outage happened outside a full pool On September 30 at 16:44:17, the Max-Q production lane fails an eager-prefill allocation of 560 MiB with 280.94 MiB free. The process reports 93.92 GiB device usage including non-PyTorch memory; another process uses 676 MiB. The configured static fraction is 0.98 and the pool has 522,880 tokens. The final log has one running request, fourteen queued, 1,485,849 pending tokens, token usage 0.73 and Mamba usage 0.06. Scheduler termination interrupts service for 12 minutes 54 seconds. That snapshot shows why a token-pool percentage alone cannot bound every external allocation. It does not prove that power limit or PCIe generation caused the crash, nor establish a universal safe static fraction. The queue alert arrives 53 seconds before failure but does not feed admission. A monitoring signal and an enforced dispatch constraint are separate operational facts. ## Recovery counts must keep their branch and invalid attempts The development `oom-retract` battery runs with host KV **off**: one run has 39 injected and six real-pressure OOMs, with 45 recoveries; the second has 39 plus three, with 42. A scheduler suite initially has nine runner errors, then passes 100 tests after a harness correction. These are not forty-five ship-tree recoveries. ![Recovery evidence must keep its sample size. The large branch battery is not a 45-event ship-tree validation. Decode and driver errors remain fatal.](/assets/batch8-qwen-recovery.webp) *The large branch battery is not a 45-event ship-tree validation. Decode and driver errors remain fatal.* The first host-KV ship-tree injection attempt is invalid because request serialization fails: zero injections and zero recoveries occur. The valid rerun has two injections, two recoveries, correct needles and a live engine. Recovery requires the source's narrow overlap/EXTEND/TP1/PP1 conditions, the other exclusion predicates and a completed drain. After three retractions an unfit request gets 503; the next request succeeds. Decode OOM and driver failures stay fatal. Peer outputs are not bit-identical, including ordinary OOM-free repeats. Finally, a composite long-context gate has no NIAH cells in its own bundle and therefore remains false, despite nine separately passing needles. That is a harness-composition gap, not demonstrated model-quality failure. Useful next evidence would combine the real serving workload, repeated per-cell GPU and host peaks, exact executed artifacts and the assembled quality verdict. Those missing fields remain unknown in the JSON export. ## Original sources and conditions - nanogenomic: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md - Document revision: 11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2 - SHA-256: 7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076 - nanogenomic: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/README.md - Document revision: 11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2 - SHA-256: bcfd94276d2bebef681e8a7737444ecec9814d6c73dce9ecf95a0b9aa437a3ab - nanogenomic: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/COMMIT-MAP.tsv - Document revision: 11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2 - SHA-256: 7a6b188d895f6d99fe46ca650832e7d1d8f182aa0b2c59e1442bd7794042fa92 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "on-GPU control minimum free VRAM", "metric": "available_GPU_memory", "value": 3414, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 870, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "on-GPU control non-pool peak over idle", "metric": "non_pool_memory_over_idle", "value": 1090, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 871, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "host KV minimum free VRAM", "metric": "available_GPU_memory", "value": 6962, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 870, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "host KV non-pool peak over idle", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 871, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "524288 FP8 old gather isolated transient", "metric": "kernel_transient_memory", "value": 1024.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "old gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 954, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "524288 FP8 bounded gather isolated transient", "metric": "kernel_transient_memory", "value": 32.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "bounded gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 954, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "500K old gather device peak", "metric": "peak_device_used_memory", "value": 97237, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "old gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 972, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "500K bounded gather device peak", "metric": "peak_device_used_memory", "value": 96093, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "bounded gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 972, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "2x250K host-KV old gather non-pool peak", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "old gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 981, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "2x250K host-KV bounded gather non-pool peak", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "bounded gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 981, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "Production OOM failed_allocation", "metric": "failed_allocation", "value": 560.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 639, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L639", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "Production OOM available_GPU_memory", "metric": "available_GPU_memory", "value": 280.94, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 640, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L640", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "Production OOM process_device_memory", "metric": "process_device_memory", "value": 93.92, "unit": "GiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 642, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L642", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" } ], "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null }, "case_reports": [ { "id": "deep-queue-production-OOM", "classification": "source_reported_real_allocator_failure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 633, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L633", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "downtime_seconds": 774, "error": "torch.OutOfMemoryError in eager prefill; scheduler SIGQUIT and process-tree termination", "monitoring_alert_before_crash_seconds": 53, "admission_consumed_alert": false }, { "id": "recovery-branch-versus-ship", "classification": "source_reported_fault_injection_and_real_pressure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 999, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L999", "branch": "oom-retract, host KV off", "branch_pool_tokens": [ 451712, 451840 ], "runs": [ { "forced": 39, "real": 6, "recoveries": 45 }, { "forced": 39, "real": 3, "recoveries": 42 } ], "engine_alive": true, "scheduler_suite": { "initial_harness_errors": 9, "initial_error": "runner IndexError", "rerun_passed": 100 }, "ship_tree": { "initial_valid": false, "initial_error": "BatchEncoding not JSON serializable", "initial_injects": 0, "initial_recoveries": 0, "initial_verdict": false, "rerun_injections": 2, "rerun_recoveries": 2, "rerun_all_needles_correct": true, "engine_alive": true, "host_KV_pool_tokens": 1310720 }, "recovery_predicates": [ "overlap enabled", "EXTEND", "TP1", "PP1", "no MLP sync", "no disaggregation", "HiSparse off", "PDMux off", "no diffusion config", "device-wide drain completes" ], "max_retractions": 3, "exhaustion_status": 503, "next_request_status": 200, "decode_OOM_recovered": false, "driver_error_recovered": false, "peer_outputs_bit_identical": false }, { "id": "host-UVA-regression", "classification": "unresolved_nonreproducing_microbenchmark", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 992, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L992", "first_new_ms": [ 341, 376 ], "later_new_ms": [ 4.99, 5.68 ], "later_new_peak_MiB": [ 159, 189 ], "first_touch_explanation_proven": false, "kernel_time_improvement_claim": false }, { "id": "composite-gate-missing-same-bundle-NIAH", "classification": "harness_composition_failure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 873, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L873", "decode_path_gate": true, "standalone_needle_correct": 9, "standalone_needle_total": 9, "same_bundle_NIAH_cells": 0, "long_context_gate": false, "overall_gate": false, "model_quality_failure_established": false } ], "memory_observations": [ { "label": "on-GPU control minimum free VRAM", "value": 3414, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870" }, { "label": "on-GPU control non-pool peak over idle", "value": 1090, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871" }, { "label": "host KV minimum free VRAM", "value": 6962, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870" }, { "label": "host KV non-pool peak over idle", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871" }, { "label": "524288 FP8 old gather isolated transient", "value": 1024.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "old gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954" }, { "label": "524288 FP8 bounded gather isolated transient", "value": 32.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "bounded gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954" }, { "label": "500K old gather device peak", "value": 97237, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "old gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972" }, { "label": "500K bounded gather device peak", "value": 96093, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "bounded gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972" }, { "label": "2x250K host-KV old gather non-pool peak", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "old gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981" }, { "label": "2x250K host-KV bounded gather non-pool peak", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "bounded gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981" }, { "label": "Production OOM failed_allocation", "value": 560.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L639" }, { "label": "Production OOM available_GPU_memory", "value": 280.94, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L640" }, { "label": "Production OOM process_device_memory", "value": 93.92, "unit": "GiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L642" } ], "long_context_cases": [ { "id": "host-KV-two-mains-turn3", "input_length_source_labels": [ "250K", "200K" ], "additional_subagents": 2, "host_KV_pool_tokens": 1310720, "control_pool_tokens": 451840, "host_KV_TTFT_seconds": [ 1.2, 1.0 ], "control_TTFT_seconds": [ 22.9, 41.1 ], "host_KV_cache_hit_rate": 0.9949, "control_cache_hit_rate": 0, "control_requests_completed": true, "host_peak_RAM": null, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 833, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L833" } ] } ``` --- # Full GLM-5.3 on Four Sparks: Cache Layout, Speed and Fidelity ![Full GLM-5.3: four Sparks, two cache layouts. Source-reported HTTP rates include prompt time; exact concurrent token accounting is not retained.](/assets/batch8-glmfull-hero.webp) *Source-reported HTTP rates include prompt time; exact concurrent token accounting is not retained.* Full GLM-5.3 is a different deployment problem from GLM-5.3-Flash. The [BertholomusAI four-GB10 recipe](https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md) targets the 753B non-Flash model, all 79 layers plus its MTP layer, using EXL3 and TensorFold. Its value is a documented route to fitting and serving that full model, with explicit cache layouts and a fidelity panel. It does not provide a general model leaderboard, a production peak-memory trace or a claim that four long jobs each occupy the entire configured window. Run AI Here has not reproduced its GPU measurements. ## The hardware and quantization contract The report uses four DGX Spark/GB10 nodes with 128 GB unified memory each, one tensor-parallel rank per node, direct RoCE links and both 200G ports. Its runtime is TensorFold 0.6.0 plus the author's `glm-dsa-tp4` branch. The source credits TensorFold's original author separately from the model-specific recipe. Substituting a switched fabric, different rank layout or another engine would be another deployment requiring its own evidence. The quantization uses exllamav3 1.5.3: routed experts are 3-bit, attention/shared experts/dense layers are promoted, the MTP experts use 4 bits and the output head uses 6. The reported whole-model average is 3.04 bits per weight, across 39 shards totalling 292.7 GB. A “3.0 bpw” artifact name is therefore not a statement that every tensor has identical precision. The exact selected checkpoint revision is not retained in the recipe, so the artifact name and null revision remain separate fields. The source derives about 69.86 GiB weights per rank from its real configuration. That is a fit calculation, not a sampled peak. It excludes the complete inference working set, graph captures, cache buffers and operating-system demands. File GB, weight GiB and peak unified-memory usage should not be treated as interchangeable capacity numbers. Peak whole-system RAM and a time-series sampling method are absent. ## Keep the release tree distinct from its later fixes October 4 release measurements are labelled engine prefix `3eb35dd`. The document's later branch head is `ddfad35`, with an RDMA completion-matching correction and graph prewarm work on top. Only short executed prefixes are provided for this table; the document's full Git revision does not fill that gap. Both identities are retained so a reader can identify the measurement generation before trying a newer recipe. ![Measured release and later fixes differ. The release speed table remains attributed to 3eb35dd, not silently relabelled as the later branch head.](/assets/batch8-glmfull-identity.webp) *The release speed table remains attributed to 3eb35dd, not silently relabelled as the later branch head.* The RDMA incident involves matching queue-pair numbers across two devices, where a collision can produce `ibv_post_recv failed`. The later fix scopes matching to the device. Another change captures rounds and draft-step graphs during warm-up, removing work from the first concurrent burst. The source reports finite greedy/concurrent equality checks after those changes. It explicitly leaves the original release table unchanged; this guide does not relabel all old numbers as a fresh run of the new head. ## Two cache layouts make different promises Configuration A uses replicated Q5 cache, a 65,536-token limit per stream and four streams. It reports 82 aggregate code tokens/s, with 0.59–0.60-second first-token times, plus 71–73 aggregate chat tokens/s. These are source HTTP observations, not per-user rates. The prompts, cache arrangement and completion behavior matter to any comparison. Configuration B uses Q5 cache with decode context parallelism over four ranks and a shared 1,048,576-token pool. Four code streams report approximately 64 aggregate tokens/s; chat reports 56. Windows are extents in one pool. This trades a different cache arrangement for a larger configured context. It does not grant four independent million-token allocations. Without raw input and output counts for every concurrent cell, the report supports its own conditional observations rather than a normalized throughput claim. The source uses GLM's native MTP layer with three draft steps, and no DFlash. Vision is not implemented in this release. Those deployment choices stay attached to the speeds. A separate Flash model with another drafter, parser or visual path would not be a matched performance control. ## Audit the token-rate denominator before comparing engines The README describes its release numbers as HTTP measurements with prompt time included. The available [serial streaming probe](https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_bench.py) counts nonempty SSE chunks and derives a post-first-chunk rate. Chunks are not guaranteed to contain one tokenizer token, and that script does not independently validate them against usage counts. The exact concurrent harness and raw usage underlying the release table are not retained in this repository. It is not established that the concurrent release table was computed by this particular serial helper. These limitations should travel with the values, rather than becoming either an invented corrected speed or an accusation that every reported rate uses the helper's denominator. The separate [greedy-output script](https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_greedy.py) does read completion usage and wall time for its own requests. That does not retrospectively establish how every other table was calculated. A future reproducible comparison should retain synchronized group wall time, tokenizer or usage output totals, warm-up policy and both TTFT and decode definitions. ## The quality panel measures fidelity to BF16 The source's quantization panel uses wikitext-2 test text, 32 segments of 2,048 tokens, BF16 activations and no cache quantization. Forward KL from BF16 to quantized probabilities is 0.109, reverse KL is 0.123 and top-1 agreement is 90.2 percent. Perplexity is 2.911 for the quantized model versus 2.750 for BF16. These are specific probability/text-panel measurements, not percentages of correctly answered tasks. ![Quantization fidelity: one text panel. Panel perplexity is not MMLU, GSM8K or a general task score; no benchmark ranking is derived.](/assets/batch8-glmfull-perplexity.webp) *Panel perplexity is not MMLU, GSM8K or a general task score; no benchmark ranking is derived.* A served-engine prompt-path check reports KL 0.111 against stored BF16 logits on the same panel. Another longer text panel exercises sparse attention and has a different KL; a separate cache-quantization comparison answers yet another question. Combining weight quantization, cache quantization and served-engine differences into one “accuracy loss” would erase the controls. The selected JSON preserves each comparison's reference and scope. No MMLU, GSM8K or coding-task score is inferred from a small KL or a finite greedy equality check. ## The long-window evidence is useful and finite The report finds a needle at 1,039,064 actual input tokens. It also reports 402 prefill tokens/s at a source-labelled 32K input and approximately 76 seconds to first token near 31K. Those rounded input labels are not exact token counts. A planner interested in long documents should budget prefill latency separately from the better short-request decode rate. ![A configured window is not four full-window jobs. A per-rank weight estimate excludes cache, graphs, working memory and system services.](/assets/batch8-glmfull-capacity.webp) *A per-rank weight estimate excludes cache, graphs, working memory and system services.* The million-token result establishes a completed retrieval case in this setup. It does not establish multi-hop accuracy, repeated long-context stability or simultaneous full-window requests. The repository's needle helper and finite output-equality probes give useful inspection points, but missing repetitions, model revision, exact concurrent accounting and peak memory remain missing. This recipe is most useful when those boundaries guide the next experiment: hold the artifact and runtime fixed, record the fabric and cache layout, then capture latency, completion correctness and memory over the same workload interval. ## Original sources and conditions - bertholomus: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md - Document revision: 9a5c1100c68cd8ea832803c120ee63313049fc21 - SHA-256: 088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4 - bertholomus: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_bench.py - Document revision: 9a5c1100c68cd8ea832803c120ee63313049fc21 - SHA-256: 9652bf8a7359414ad451dc05ceee4f4bef6cfcf489e65334e9bec1e7895192b2 - bertholomus: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_needle.py - Document revision: 9a5c1100c68cd8ea832803c120ee63313049fc21 - SHA-256: cdf8ab15fc7ec88073a714696b561f2e7a292ac9a0490571935bbe2cc933a4ab - bertholomus: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_greedy.py - Document revision: 9a5c1100c68cd8ea832803c120ee63313049fc21 - SHA-256: 834c9f1e73af9431b585adf00aa3f14254d8713b638a4fa95c36b6a2074dbf37 Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "replicated Q5 four code streams", "metric": "aggregate_output_rate", "value": 82, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 4, "context_limit_tokens": 65536, "cache_state": "replicated", "metric_denominator": "source reports HTTP prompt time included; release concurrent raw usage/harness not retained", "token_accounting_verified": false, "available_serial_stream_probe_counts_nonempty_SSE_chunks": true, "release_was_computed_by_that_probe": null, "value_is_approximate": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 57, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L57", "source_record_id": "glm53-full-tp4-exl3" }, { "label": "DCP4 shared Q5 four code streams", "metric": "aggregate_output_rate", "value": 64, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 4, "context_limit_tokens": 1048576, "cache_state": "four streams share one DCP4 pool", "metric_denominator": "source reports HTTP prompt time included; release concurrent raw usage/harness not retained", "token_accounting_verified": false, "available_serial_stream_probe_counts_nonempty_SSE_chunks": true, "release_was_computed_by_that_probe": null, "value_is_approximate": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 68, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L68", "source_record_id": "glm53-full-tp4-exl3" }, { "label": "32K-labelled prefill", "metric": "prefill_rate", "value": 402, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 1, "input_length_source_label": "32K", "context_limit_tokens": 1048576, "cache_state": null, "metric_denominator": "source HTTP prefill measurement, exact prompt count unreported" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 71, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L71", "source_record_id": "glm53-full-tp4-exl3" }, { "label": "forward KL", "metric": "KL_reference_to_quant", "value": 0.109, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "forward KL" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 167, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L167", "source_record_id": "glm53-full-tp4-exl3" }, { "label": "reverse KL", "metric": "KL_quant_to_reference", "value": 0.123, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "reverse KL" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 168, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L168", "source_record_id": "glm53-full-tp4-exl3" }, { "label": "top-1 agreement", "metric": "top1_agreement", "value": 90.2, "unit": "percent", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "top-1 agreement" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 169, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L169", "source_record_id": "glm53-full-tp4-exl3" }, { "label": "quant perplexity", "metric": "perplexity", "value": 2.911, "unit": "perplexity", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "quant perplexity" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 170, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L170", "source_record_id": "glm53-full-tp4-exl3" }, { "label": "BF16 perplexity", "metric": "perplexity", "value": 2.75, "unit": "perplexity", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "BF16 perplexity" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 170, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L170", "source_record_id": "glm53-full-tp4-exl3" }, { "label": "served engine prompt-path KL", "metric": "KL_reference_to_engine", "value": 0.111, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "served path; panel-specific cache details not retained", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "served engine" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 173, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L173", "source_record_id": "glm53-full-tp4-exl3" } ], "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null }, "case_reports": [ { "classification": "source_reported_RDMA_completion_collision", "old_error": "ibv_post_recv failed", "fix_short": "cad0d2a", "measured_release_short": "3eb35dd", "later_head_short": "ddfad35", "full_execution_revision": null, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 35, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L35" } ], "quality_comparisons": [ { "panel": "32x2048 wikitext-2", "BF16_activations": true, "cache_quantization": false, "forward_KL": 0.109, "reverse_KL": 0.123, "top1_percent": 90.2, "quant_PPL": 2.911, "BF16_PPL": 2.75, "task_accuracy": null } ], "memory_observations": [ { "kind": "shape-derived weights per rank, not peak", "value": 69.86, "unit": "GiB", "is_measured_peak": false, "whole_system_peak": null, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 121, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L121" } ], "long_context_cases": [ { "actual_input_tokens": 1039064, "needle_found": true, "independent_repeats": null, "concurrent_full_window_requests_proven": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 71, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L71" } ] } ``` --- # GLM Flash Scores: Historical NVFP4 and the EXL3 Retraction ![GLM Flash: keep the scoring line attached. No NVFP4 score is assigned to the corrected EXL3 deployment. No cross-stack ranking.](/assets/batch8-glmflash-hero.webp) *No NVFP4 score is assigned to the corrected EXL3 deployment. No cross-stack ranking.* A deployment's scores need a version history just as much as its launch command does. Two original GLM-5.3-Flash reports illustrate why: a September NVFP4 task campaign and an October EXL3 correctness investigation. Both use three DGX Sparks, but they differ in quantization, plugin, engine and measurement purpose. The EXL3 author withdrew its old task scores after finding a missing router bias. Reading both reports is useful only if their boundaries survive the summary. Run AI Here has not rerun either deployment or assigned them a combined score. ## The historical NVFP4 campaign has its own identity The [September 3 benchmark ledger](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md) describes a continuous 09:08–18:08 UTC campaign using three GB10 nodes, TP3 plus expert parallelism, Marlin MoE, FP8 KV and DFlash2 k7. The image is `harem/glm53-lil:t10`, with a short engine identity. The configured context ceiling is **1,000,000 decimal tokens**, max sequences eight and batched tokens 2,048. Memory utilization for these benchmark runs is **0.85**, rather than the later production 0.88 setting. The [NVFP4 project README](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/README.md) says this line stopped maintenance on September 6 and moved to EXL3. The old measurements remain historical observations of that stack. Their checkpoint revision and executed image digest are not pinned, so neither is manufactured from the documentation commit. Most generation tests use temperature zero with low reasoning effort; MMLU is the explicit exception because it performs loglikelihood scoring without generation or sampling. ## Full MMLU is different from a small generation subset The saved [raw MMLU result](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json) gives accuracy **0.8594217347956131** and standard error **0.0028276391244761038**. This is the full 14,042-question, 57-subject, zero-shot run under lm-eval 0.4.9. The uncertainty field is a harness standard error, not a repeated-trial standard deviation or a confidence interval. Replacing that meaning with a generic plus/minus label would lose useful statistical information. The harness uses concurrency eight, a 36,000-second timeout and six retries. An initial attempt stalls around 81 percent because aiohttp's total timeout includes queued work. The repeated full run keeps the content and reports no engine errors. The failed harness attempt is not a measured zero-accuracy model run, and raising the request timeout is not an engine OOM fix. The final result and the reason for the rerun both remain queryable. ![The check that exposed the routing defect. Six texts, 21,066 scored positions; 6,601 assistant positions. Reference is not BF16.](/assets/batch8-glmflash-kl.webp) *Six texts, 21,066 scored positions; 6,601 assistant positions. Reference is not BF16.* The GSM8K result is instead 188 correct out of 200 selected questions, with eight-shot chain-of-thought prompts and low effort. It is 94.0 percent on that subset, not the full 1,319-question test and not a matched comparison to a zero-shot, thinking-off deployment. Different prompting and answer extraction can change what a score means even before hardware enters the comparison. ## Reliability and evaluator limits are part of the score The [raw tool-evaluation report](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json) uses tool-eval-bench 2.6.1.dev39+gd3352edf5, 88 hard scenarios and eight trials. The listed trial scores are 88, 88, 86, 87, 88, 88, 88 and 89. The publisher reports a rounded mean 87.8, SD 0.9 and 95 percent interval [87.1, 88.2]; the interval calculation is not independently verified here. The top-level final score is the last trial, not the mean. Pass^8 is 76.1 percent, a stricter repeated-success view than the average score. The same ledger's IFEval uses a custom evaluator: strict prompt accuracy is 427/541, or 78.9 percent, and instruction-level accuracy is 710/834, or 85.1 percent. A known suspicious title-format failure motivates checking the evaluator against an official implementation. One suspected false negative does not prove that every discrepancy goes in one direction, so this guide does not certify the score as a mathematical lower bound. Its evaluator remains unverified. The [needle report](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md) records 20 successful single-fact probes over four lengths and five depths, through 997,952 actual tokens. The initial 120-second timeout is raised to 3,600 seconds for the same-content rerun. That is finite retrieval evidence, not demonstrated long-context reasoning, concurrency at the full window or a whole-workload peak-memory trace. ![What the historical task scores measure. MMLU uses loglikelihood and no sampling. These are heterogeneous task scores, not a composite.](/assets/batch8-glmflash-scores.webp) *MMLU uses loglikelihood and no sampling. These are heterogeneous task scores, not a composite.* ## The EXL3 notice changes what can be quoted The [October 8 correctness notice](https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md) identifies an unloaded `e_score_correction_bias` in the old cuda-exl3 plugin. The router uses zeros instead of the model's balancing bias and selects wrong experts. The author withdraws the old EXL3 repository's task scores, including its prior MMLU, tool, GSM8K, IFEval and needle panels, until they are rerun. Old speed numbers also do not define corrected routing's speed. That notice concerns the EXL3 plugin line. It is not evidence that a separate Marlin NVFP4 implementation has the same defect. Conversely, the separate NVFP4 scores cannot validate the corrected EXL3 model. Keeping those two statements together avoids both discarding unrelated measurements without evidence and filling a corrected stack's missing task scores with a sibling stack's results. The [new TP3 main recipe](https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/tracks/tp3-main/README.md) pins target weights to `2a30229e67012798ba9f0cd832bb78abf4c363d5`, base vLLM to `21d93d0d8c0e9627900020382bfce4730e61cab7` and production cuda-exl3 to `448f1d6b8964b96e7d4e299494538bfc240149ef`. It adds a boot gate for the bias. The reported local image ID is retained as an ID, not mislabelled a registry digest. Published patch comments differ from the files in that image, so a byte-identical rebuild is not promised. TP2 has not been ported or tested on this corrected path. ## The rerun is probability health, not a new task leaderboard The detailed [KL investigation](https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md) compares the **same quantized checkpoint** to exllamav3 1.5.4, not to BF16. Six author-written texts have 21,072 tokens, 21,066 scored positions and 6,601 assistant positions. The partitioned reference-top-512-plus-remainder estimate is a lower-bound KL measurement; assistant positions form the headline because other regions behave differently. Pre-fix assistant KL is 0.1033; production a4i is 0.00828. The detailed table's perplexity 2.222 belongs to the **bias-fix a4g** arm, against reference 2.224. Production a4i perplexity is unreported in that table, so it remains null even where a short summary groups the numbers together. Neither value gives task accuracy. The prepared text set and full per-position logits are not public, which limits independent reproduction. ![Retractions and reruns belong in the result. Bias-fix perplexity 2.222 belongs to a4g. Production a4i perplexity remains null in the detailed table.](/assets/batch8-glmflash-reruns.webp) *Bias-fix perplexity 2.222 belongs to a4g. Production a4i perplexity remains null in the detailed table.* The [current quality-gate document](https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md) explicitly says task benchmarks have not been rerun on the corrected stack. The useful publication is therefore a correction with a missing-result ledger: historical NVFP4 tasks retain their own date and harness; withdrawn EXL3 tasks stay withdrawn; corrected EXL3 probability checks retain their reference; future task reruns remain pending. A search or AI summary should return those states alongside any numeric excerpt, rather than selecting the most flattering score from an older table. ## Original sources and conditions - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/tracks/tp3-main/README.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: 6cb79207b37211c1542b98dfe26408e7d0b741133be67286db7838c48976223f - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/README.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 5912e935333340dbd77ecd01cec8c7e207990bc6f446341f741bb3bd66d6cf4b - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/05-memory-ladder.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 29874450acd65f1e6614bd3fc04fe0815ef8b9036007867b8446e63ebb4092de - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 7a46e4d7b52956a0fb0662838081b526e7ceeb30e14e553648227bade4dc6df5 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: cceef7123bbb9e9d34c2c6a44bc00055b81f896cbf51977053766e770c30ae3d - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/README.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 15388126cd782399e618a52b5cbcd2c5af46f7de347f6903e4705d719f84b037 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: 9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: 9d0387be075bd6c214ab8e9b2bb70a46577b5052884c06c0db02d1869b295f28 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/docs/20-main-stack.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: b7ef6f4c82dd1bfa693452cb1fcfb57c8f3270455ca894b667e24048cc20786d - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/README.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 5912e935333340dbd77ecd01cec8c7e207990bc6f446341f741bb3bd66d6cf4b - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/05-memory-ladder.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 29874450acd65f1e6614bd3fc04fe0815ef8b9036007867b8446e63ebb4092de - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 7a46e4d7b52956a0fb0662838081b526e7ceeb30e14e553648227bade4dc6df5 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: cceef7123bbb9e9d34c2c6a44bc00055b81f896cbf51977053766e770c30ae3d - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/README.md - Document revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - SHA-256: 15388126cd782399e618a52b5cbcd2c5af46f7de347f6903e4705d719f84b037 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: 9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: 9d0387be075bd6c214ab8e9b2bb70a46577b5052884c06c0db02d1869b295f28 - NNNtrance: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/docs/20-main-stack.md - Document revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - SHA-256: b7ef6f4c82dd1bfa693452cb1fcfb57c8f3270455ca894b667e24048cc20786d Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "Tool-eval eight-trial mean", "metric": "tool_eval_score", "value": 87.8, "unit": "points", "statistic": "mean across eight trials; source SD 0.9, source CI [87.1,88.2]", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "tool-eval-bench 2.6.1.dev39+gd3352edf5", "scenarios": 88, "trials": 8, "seed": 42, "max_turns": 8, "concurrency": 1, "source_trial_SD": 0.9, "source_CI95": [ 87.1, 88.2 ], "CI_method_independently_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 47, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L47", "source_record_id": "glm53-flash-nvfp4-tp3-scores" }, { "label": "Tool-eval Pass^8", "metric": "all_trials_pass_rate", "value": 76.1, "unit": "percent", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "tool-eval-bench", "scenarios": 88, "trials": 8, "concurrency": 1 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 56, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L56", "source_record_id": "glm53-flash-nvfp4-tp3-scores" }, { "label": "GSM8K 200-question subset", "metric": "accuracy", "value": 94.0, "unit": "percent", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "GSM8K", "subset_size": 200, "full_test_size": 1319, "correct": 188, "shots": 8, "CoT": true, "concurrency": null }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 197, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L197", "source_record_id": "glm53-flash-nvfp4-tp3-scores" }, { "label": "IFEval prompt strict, custom evaluator", "metric": "accuracy", "value": 78.9, "unit": "percent", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "IFEval", "prompts": 541, "correct": 427, "concurrency": null, "evaluator": "custom; not independently cross-checked", "proven_lower_bound": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 155, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L155", "source_record_id": "glm53-flash-nvfp4-tp3-scores" }, { "label": "IFEval instruction level, custom evaluator", "metric": "accuracy", "value": 85.1, "unit": "percent", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "IFEval", "instructions": 834, "correct": 710, "concurrency": null, "evaluator": "custom; not independently cross-checked", "proven_lower_bound": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 156, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L156", "source_record_id": "glm53-flash-nvfp4-tp3-scores" }, { "label": "Needle four sizes x five depths", "metric": "retrieval_correct_count", "value": 20, "unit": "correct probes", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "single-fact NIAH", "correct": 20, "total": 20, "max_actual_input_tokens": 997952, "concurrency": null, "initial_timeout_seconds": 120, "rerun_timeout_seconds": 3600, "identical_content_seed_on_rerun": true, "multihop_validated": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 221, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L221", "source_record_id": "glm53-flash-nvfp4-tp3-scores" }, { "label": "Full MMLU exact accuracy", "metric": "accuracy", "value": 0.8594217347956131, "unit": "fraction", "statistic": "single full MMLU, harness standard error 0.0028276391244761038", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": null, "thinking": null, "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "MMLU", "subset_size": 14042, "subjects": 57, "shots": 0, "scoring": "loglikelihood, no generation or sampling", "harness": "lm-eval 0.4.9", "concurrency": 8, "timeout_seconds": 36000, "max_retries": 6, "accuracy_stderr": 0.0028276391244761038, "stderr_is_SD": false, "initial_harness_timeout_at_fraction": 0.81, "engine_errors_reported": 0, "rerun_same_content": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-mmlu", "source_document_sha256": "86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8", "source_line": 4, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json#L4", "source_record_id": "glm53-flash-nvfp4-tp3-scores" }, { "label": "Pre-fix assistant KL", "metric": "KL_reference_to_engine", "value": 0.1033, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4f", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-exl3-kl", "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1", "source_line": 40, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L40", "source_record_id": "glm53-flash-exl3-router-correction" }, { "label": "Production assistant KL", "metric": "KL_reference_to_engine", "value": 0.00828, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4i", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-exl3-kl", "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1", "source_line": 42, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L42", "source_record_id": "glm53-flash-exl3-router-correction" }, { "label": "Bias-fix assistant perplexity", "metric": "perplexity", "value": 2.222, "unit": "perplexity", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4g", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-exl3-kl", "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1", "source_line": 41, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L41", "source_record_id": "glm53-flash-exl3-router-correction" } ], "conditions": { "historical_NVFP4": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null }, "corrected_EXL3": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false } }, "case_reports": [ { "id": "NVFP4-harness-only-reruns", "classification": "source_reported_harness_timeouts", "MMLU_initial_progress_fraction": 0.81, "MMLU_initial_error": "aiohttp total timeout includes queue", "MMLU_rerun_timeout_seconds": 36000, "needle_initial_timeout_seconds": 120, "needle_rerun_timeout_seconds": 3600, "same_content_and_seed": true, "reported_engine_errors": 0, "initial_partial_not_full_score": true, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 35, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L35" }, { "id": "EXL3-router-bias-missing", "classification": "source_reported_correctness_bug_and_retraction", "plugin_pre_fix_short": "754421f or earlier", "missing_parameter": "e_score_correction_bias", "source_scope": "EXL3 plugin line only; no evidence that separate NVFP4 line shares defect", "old_task_scores_withdrawn": true, "current_task_score": null, "source_document_id": "glm-exl3-readme", "source_document_sha256": "a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0", "source_line": 18, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md#L18" } ], "quality_comparisons": [ { "id": "NVFP4-historical-task-panel", "record_id": "glm53-flash-nvfp4-tp3-scores", "task_scores_current_EXL3": false, "task_scores_withdrawn_by_EXL3_notice": false, "source_scope": "separate NVFP4 repository, September 3 measurements", "tool_trial_scores": [ 88, 88, 86, 87, 88, 88, 88, 89 ], "tool_mean": 87.8, "source_SD": 0.9, "source_CI95": [ 87.1, 88.2 ], "MMLU_accuracy": 0.8594217347956131, "MMLU_standard_error": 0.0028276391244761038, "IFEval_evaluator_crosschecked": false, "proven_IFEval_lower_bound": false }, { "id": "EXL3-retraction-and-rerun-gap", "record_id": "glm53-flash-exl3-router-correction", "withdrawn_on": "2026-10-08", "old_task_scores_withdrawn": true, "current_task_accuracy": null, "current_task_benchmarks_rerun": false, "router_bias_loaded_after_fix": true, "production_KL": 0.00828, "production_PPL": null, "bias_fix_PPL": 2.222, "reference_PPL": 2.224, "BF16_reference": false, "public_text_set_available": false, "full_per_position_logits_public": false, "TP2_fixed_or_validated": false } ], "memory_observations": [ { "kind": "historical operational free host memory, not peak", "value_range": [ 6.4, 10 ], "unit": "GB, source approximate", "historical_NVFP4_only": true, "peak_RAM": null, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 28, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L28" } ] } ``` --- # MiMo Pro on Eight Sparks: Scheduling, Rates and Regression Scores ![MiMo Pro: remove the queued eighth request. 43908/187.18 s vs 43900/143.16 s. Whole-wave wall includes prefill; this is not a 128K test.](/assets/batch8-mimopro-hero.webp) *43908/187.18 s vs 43900/143.16 s. Whole-wave wall includes prefill; this is not a 128K test.* The [eight-Spark MiMo-V2.6-Pro-RL deployment](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md) provides more than a fast decode headline. Its most interpretable comparison preserves a scheduler failure, cold-cache request bodies, completion totals, complete-wave wall time and answer checks across builds a13 and a14. That makes it useful for understanding concurrency behavior. It also shows why a workload called “long extraction” should not automatically be filed as a huge-input context test. Run AI Here has not run these model benchmarks or rented additional hardware. ## What the eight-node recipe requires The publisher identifies XiaomiMiMo/MiMo-V2.6-Pro-RL, described as a 1.02T-total, 42B-active MoE, at checkpoint revision `73875d00b30a89ef8cc353a0b60b0e9f9561952d`. Eight GB10 nodes use TP8 with EP1, two switched 200G RoCEv2 subnets and MTU 9,000. Each node has a full local weight copy; rank-local weight fit and total disk size are separate from peak runtime unified memory. The source's roughly 66–69 GB weights per rank is not a measured peak. The runtime is a patched SGLang with packed MXFP4 experts on Marlin W4A16, FA4 prefill/verify and DFlash8 with adaptive verification width. Optional FP8 output projections and draft components are part of the measured profile. These details mean “MXFP4 deployment” does not imply all arithmetic is 4-bit, and “eight Sparks” alone does not describe the communication topology. A two-node machine or an Ethernet setup without the stated fabric is another recipe. The [version lock](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/versions.lock.json) records base SGLang `06008c170dae333e22caceb51f36224071f9b5b3` and the model revision. A base commit does not fully identify a patched executable tree. The a14 local image ID is retained separately from the base-image digest; a portable distribution digest and a13 executed image ID are not reported. Those nulls are more useful than attaching a14's image to older measurements simply because it is the current serving build. ## The short table and the paired extraction use different rates The September 24 a13 table reports coding decode at 91.1 tokens/s for C1 and 42.3 per stream at C8. Its all-eight-category C8 aggregate is 190.8 tokens/s. The arithmetic mean across category rates and aggregate tokens divided by group wall time have different denominators. Multiplying a published mean by eight is not a reconstruction of the aggregate. Output length, acceptance behavior and task category also change which rate a user experiences. The source warns that individual short requests can move substantially between builds. That is a variability warning, not an estimated confidence interval or proof that every apparent gain is statistically established. The cold structured-extraction comparison is a more inspectable scheduling case because its original logs retain actual output counts and wave timings. ## The eighth request was queued by slot accounting Build a13 counts a request finishing its final prefill chunk against a request slot that it already holds. In the affected batch, seven streams run while the eighth waits until another finishes. Build a14 changes that scheduler accounting. The report's explanation is bounded to this path; it is not proof that all queueing in SGLang or all long-context failures have been fixed. | Cold extraction, three waves of eight | a13 | a14 | |---|---:|---:| | Correct requests | 24/24 | 24/24 | | Completion usage tokens | 43,908 | 43,900 | | Total wave wall time, seconds | 187.18 | 143.16 | | Aggregate tokens/s, prefill included | 234.57 | 306.66 | | Arithmetic mean per-stream tokens/s | 39.46 | 38.40 | These values come from the [a13 raw waves](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/long-extraction-c8.txt) and [a14 raw waves](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction-c8.txt). The output totals are close but not identical, and both arms meet the fixture's answer check. The improved aggregate accompanies a lower arithmetic per-stream mean. That is possible because the long queued tail changes the complete-wave denominator, while averaging individual stream rates weights the requests differently. ![Aggregate rate and per-stream mean differ. Do not multiply the arithmetic mean by eight or combine separate denominator definitions.](/assets/batch8-mimopro-denominators.webp) *Do not multiply the arithmetic mean by eight or combine separate denominator definitions.* The worst retained first-token observation falls from 46.39 seconds to 8.17 seconds. This explains the source's rounded 46-to-eight-second description without inventing p95 latency or a distribution of production traffic. Three waves are finite repeated fixtures, not a sustained unattended service test. ## Long extraction is principally a long-output task here The [C8 harness](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/long_extraction_c8.py) duplicates four CSV-to-JSON tasks per wave, uses temperature zero, seed 20260922, thinking off and a 4,096-token output cap. It supplies a unique cache salt, flushes prefix cache before each wave, consumes completion usage and requires both normal stop and an exact parsed-JSON oracle match. This is stronger evidence than HTTP success alone, while still covering only those fixtures. The [C1 raw log](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction.txt) records **1,498 input tokens** on every request, with about 1,829 output tokens. Twelve requests pass, with 21,952 completion tokens over 209.86 seconds and 104.6 end-to-end tokens/s. The [C1 method](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/long_extraction_c1.py) explicitly obtains the final tuple field from `usage.prompt_tokens`. C8 logs omit prompt counts, so those stay unknown even though they use the related fixtures. Neither table is a 128K-plus concurrent-input experiment. Separate a13 cold-prefill medians use source labels 5K, 20K, 81K and 163K, three repeats and a unique prefix. Selected rates are 1,880 tokens/s at nominal 5K and 1,499 at nominal 163K. Exact per-cell input counts are not supplied in the selected summary. These prefill screens should not be mixed with decode-only rates or used as a missing peak-memory trace. ## Keep the score's build, prompting and wrong answers The [a13 GSM8K result](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/gsm8k200.json) has 196 correct of the first 200 fixed-order test questions: 98.0 percent on that subset. Failed IDs are 93, 119, 162 and 187. The [scoring method](https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/eval/gsm8k_eval.py) uses zero-shot brief-working prompts, greedy generation with thinking off, a 1,024-token output cap and a custom numeric extractor with a last-number fallback. Its default concurrency is eight, but the saved result does not establish the executed command's concurrency. ![A regression subset, with its failures. Needle results at 127,803 and 244,168 tokens belong to older a5/a6 builds, not a13/a14.](/assets/batch8-mimopro-score.webp) *Needle results at 127,803 and 244,168 tokens belong to older a5/a6 builds, not a13/a14.* This is a regression check between kernel/backend choices. It is not a full GSM8K leaderboard score, nor comparable to an eight-shot thinking-enabled subset without a matched protocol. There is no a14 GSM8K rerun in the selected evidence. The scheduler fix's 24 correct extraction answers do not fill that missing task rerun. ## Older needle checks do not automatically follow the current image Three-fact needle results at 127,803 input tokens belong to a5/a6, while 244,168 belongs to a5. Mixed-workload success is also documented on older builds. Current a14 finite smoke checks include text, reasoning split, tool use and a simple image task; audio and video are not validated. These are useful capabilities and gaps, not permission to move an older long-context result onto every subsequent build. ![Eight-node deployment has a real fabric requirement. Image ID, base image digest and the actual patched engine revision are separate provenance fields.](/assets/batch8-mimopro-setup.webp) *Image ID, base image digest and the actual patched engine revision are separate provenance fields.* For deployment planning, the missing next evidence is clear: a14 long-input concurrent tests with actual per-request token counts, synchronized correctness, GPU/unified and host-memory telemetry, and a rerun of the task panel. Retaining the initial scheduling defect and its exact retest makes that next test concrete. A static evidence page can then show what changed, what passed and what remains unmeasured without building an elaborate serving backend or inventing a universal model score. ## Original sources and conditions - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: 97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/versions.lock.json - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: e67a69bc019e7dc5e1a9c5184190901ca147b5f585ac4d1605ca23fde2b1b20c - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/gsm8k200.json - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: 4a281c460a7fa3a07b41cf3856faeba767c229a6d43b874ee7ff5d6729865cff - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/long-extraction-c8.txt - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: 443ce4520c8472347ac496024483ff46ab1b9a6a3851803d64bf382b66a26f37 - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction-c8.txt - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: 36b64993a9c32e4b08be0ee600aa743e3fe4c62c917a6ae3ca05f9560c42233a - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction.txt - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: aa85e6233aaf60ab708d6580efada76287b2d560b4cc65f76f72f1cd3f5ab0d3 - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/eval/gsm8k_eval.py - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: 207a2f4d38a1870a876bd62f8b9070baaf2a07faccbf25ac9ffe194c64b53353 - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/mimobench.py - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: d0425c39d1e5ac0f768ea8a451e39a3ad552e8808c2a5c269061faea3bb0b918 - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/long_extraction_c1.py - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: 3012886b036a75e049296c0d9a02aff20898003c806821fd39e1b8405eb6aabd - rhys101: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/long_extraction_c8.py - Document revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - SHA-256: 3c27bb057d58d7afef9fd137bb7a67bb9341a05c85e0f29947e12362b37c685a Checked 2026-10-18. AI-assisted writing, source checked by Run AI Here. No independent model tests. Conditions and selected source facts are in the adjacent JSON export. ```json { "facts": [], "reported_results": [ { "label": "coding C1", "metric": "decode_rate", "value": 91.1, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "metric_denominator": "source per-stream decode" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 28, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L28", "source_record_id": "mimopro-tp8-sglang-scheduler" }, { "label": "coding C8 per stream", "metric": "decode_rate", "value": 42.3, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "metric_denominator": "source per-stream decode" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 28, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L28", "source_record_id": "mimopro-tp8-sglang-scheduler" }, { "label": "eight categories C8 aggregate", "metric": "aggregate_output_rate", "value": 190.8, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "metric_denominator": "all eight category output / total wall; not mean per-stream multiplied by 8" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 35, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L35", "source_record_id": "mimopro-tp8-sglang-scheduler" }, { "label": "a13 cold CSV extraction C8", "metric": "aggregate_output_rate", "value": 234.57, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": "three waves, 8 requests per wave", "cache_state": "unique cache salt; prefix cache flushed before each wave", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "input_length_source_label": "four CSV fixtures, duplicated per wave; C8 input counts not retained", "output_tokens_cap": 4096, "actual_output_tokens_group": 43908, "group_wall_seconds": 187.18, "correct": 24, "total": 24, "seed": 20260922, "metric_denominator": "all completion usage tokens / total synchronized wave wall including prefill" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-a13-long8", "source_document_sha256": "443ce4520c8472347ac496024483ff46ab1b9a6a3851803d64bf382b66a26f37", "source_line": 4, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/long-extraction-c8.txt#L4", "source_record_id": "mimopro-tp8-sglang-scheduler" }, { "label": "a14 cold CSV extraction C8", "metric": "aggregate_output_rate", "value": 306.66, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": "sha256:db156b3c950a0ec9c4a7d629f7574f65c1023150bcf2d0755c45d4f2c3cab661", "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a14", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": "three waves, 8 requests per wave", "cache_state": "unique cache salt; prefix cache flushed before each wave", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "input_length_source_label": "four CSV fixtures, duplicated per wave; C8 input counts not retained", "output_tokens_cap": 4096, "actual_output_tokens_group": 43900, "group_wall_seconds": 143.16, "correct": 24, "total": 24, "seed": 20260922, "metric_denominator": "all completion usage tokens / total synchronized wave wall including prefill" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-a14-long8", "source_document_sha256": "36b64993a9c32e4b08be0ee600aa743e3fe4c62c917a6ae3ca05f9560c42233a", "source_line": 4, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction-c8.txt#L4", "source_record_id": "mimopro-tp8-sglang-scheduler" }, { "label": "a14 cold CSV extraction C1", "metric": "aggregate_output_rate", "value": 104.6, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a14", "temperature": 0, "thinking": false, "input_tokens": 1498, "output_tokens": null, "repetitions": "four tasks x three repeats", "cache_state": "unique salt and cache flush before every request", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "output_tokens_cap": 4096, "actual_output_tokens_group": 21952, "group_wall_seconds": 209.86, "correct": 12, "total": 12, "seed": 20260922, "metric_denominator": "completion usage / summed client wall including prefill" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-a14-long1", "source_document_sha256": "aa85e6233aaf60ab708d6580efada76287b2d560b4cc65f76f72f1cd3f5ab0d3", "source_line": 13, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction.txt#L13", "source_record_id": "mimopro-tp8-sglang-scheduler" }, { "label": "a13 GSM8K first 200", "metric": "accuracy", "value": 98.0, "unit": "percent", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": null, "harness_default_concurrency": 8, "actual_harness_concurrency_reported": false, "benchmark": "first 200 GSM8K test questions in fixed order", "correct": 196, "total": 200, "shots": 0, "output_tokens_cap": 1024, "answer_extraction": "Answer: numeric, else last numeric; abs tolerance <1e-6", "wrong_ids": [ 93, 119, 162, 187 ], "a14_GSM8K_rerun": false, "leaderboard_equivalent": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 83, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L83", "source_record_id": "mimopro-tp8-sglang-scheduler" }, { "label": "a13 5K cold prefill", "metric": "prefill_rate", "value": 1880, "unit": "tokens/s", "statistic": "median of three source runs; no SD", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": 3, "cache_state": "unique cold prefix", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "input_length_source_label": "5K", "statistic": "median", "metric_denominator": "TTFT based" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 57, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L57", "source_record_id": "mimopro-tp8-sglang-scheduler" }, { "label": "a13 163K cold prefill", "metric": "prefill_rate", "value": 1499, "unit": "tokens/s", "statistic": "median of three source runs; no SD", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": 3, "cache_state": "unique cold prefix", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "input_length_source_label": "163K", "statistic": "median", "metric_denominator": "TTFT based" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 58, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L58", "source_record_id": "mimopro-tp8-sglang-scheduler" } ], "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null }, "case_reports": [ { "id": "a13-eighth-stream-queued", "classification": "source_reported_scheduler_slot_accounting_bug", "initial_build": "a13", "fixed_build": "a14", "initial_C8_aggregate": 234.57, "fixed_C8_aggregate": 306.66, "initial_tokens": 43908, "fixed_tokens": 43900, "initial_wall_seconds": 187.18, "fixed_wall_seconds": 143.16, "initial_correct": 24, "fixed_correct": 24, "total": 24, "initial_mean_per_stream": 39.46, "fixed_mean_per_stream": 38.4, "initial_max_TTFT_seconds": 46.39, "fixed_max_TTFT_seconds": 8.17, "C1_fixture_input_tokens": 1498, "C8_input_counts_retained": false, "universal_long_context_fix": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 49, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L49" } ], "quality_comparisons": [ { "benchmark": "GSM8K first 200, build a13", "correct": 196, "total": 200, "wrong_ids": [ 93, 119, 162, 187 ], "shots": 0, "thinking": false, "a14_rerun": false, "leaderboard_score": false, "full_test": false } ], "memory_observations": [ { "kind": "weight storage / rank fit description, not peak RAM", "weight_files_GB": 535, "weights_per_rank_GB": [ 66, 69 ], "peak_runtime_RAM": null, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 118, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L118" } ], "long_context_cases": [ { "actual_input_tokens": 127803, "builds": [ "a5", "a6" ], "three_facts": true, "current_a13_a14_rerun": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 84, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L84" }, { "actual_input_tokens": 244168, "builds": [ "a5" ], "three_facts": true, "current_a13_a14_rerun": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 84, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L84" } ] } ``` ---