# Run AI Here — community source collection Source-reported. Selected attributed figures retain individual conditions and citations. No independent model tests. # DeepSeek-V4.1-Flash-1017: official model and deployment card The 17 October checkpoint: official weights, protocol details and minimal local inference. Start here to identify 1017 before reading a deployment recipe. - Type: Official model card - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: deepseek-ai/DeepSeek-V4.1-Flash-1017 - Original author: deepseek-ai - Original project: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 - Pinned document: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md - Source revision: 1bae0e1088993b9b82de08f59de5be5a34e42170 - Source commit time (not experiment date): 2026-10-17T05:46:55.000Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: Reference inference - Hardware topics: - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The publisher supplies an encoder rather than a Jinja chat template. - Minimal inference demonstrates converted per-rank weights; it is a reference implementation. - Capability tables use named harnesses; they are not workstation speed measurements. ## Limits - Older V4.1 community speed reports do not establish 1017 performance. - An advertised 1M context is not a tested capacity for an arbitrary machine. ## License declaration {"declared_spdx": "mit", "declared_name": null, "scope": "Publisher model-card declaration; inspect the linked LICENSE and artifact-specific terms."} ## Editorial answer The 17 October checkpoint: official weights, protocol details and minimal local inference. Start here to identify 1017 before reading a deployment recipe. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What changed in the 1017 checkpoint? DeepSeek published this weight refresh on 17 October 2026. The model name includes 1017, so it identifies a different artifact from the September launch checkpoint used by the older workstation reports. Start from the pinned weight revision on this page and preserve the encoder, tokenizer and conversion identity with it. A runtime recipe that now defaults to 1017 does not retroactively update an earlier speed experiment. Sources: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L43 ## How should prompts and tool calls be encoded? The release supplies a Python encoding reference instead of a Jinja chat template. Its maintained deepseek-recipe toolkit translates API messages into the model protocol and parses responses. That toolkit does not perform inference, run tools or provide HTTP transport. A generic chat-template substitution can therefore change behavior even when the weights load successfully. Sources: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L69 ## What do the capability figures establish? The publisher reports HLE without tools as 42.6 on the full benchmark and 44.0 on its text-only subset. These populations differ. DeepSWE v1.1 uses Mini-SWE-V2, while Terminal-Bench 4.0 uses Claude Code. The figures describe the publisher’s evaluation protocol; no local GPU configuration or tokens-per-second measurement is established by that table. Sources: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L65 ## Which inference settings should travel with a reproduction? The card recommends temperature 1.0, top_p 1.0, a 1M-token context and an output budget of at least 256K tokens. These are publisher recommendations, not evidence that a particular workstation can allocate that window. The separate minimal-inference example is a reference implementation requiring conversion into per-rank weights; it is not a drop-in production server. Sources: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L77 ## Attributed configuration facts ```json [ { "name": "Recommended sampling", "value": "temperature 1.0; top_p 1.0", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L81", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" }, { "name": "Published context / recommended output budget", "value": "1M context; max_tokens at least 256K", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L84", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" }, { "name": "Prompt interface", "value": "Python encoder; no Jinja-format chat template", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L69", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" } ] ``` ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "HLE without tools, full set", "metric": "HLE", "value": 42.6, "unit": "reported score", "statistic": "reported value", "uncertainty": null, "conditions": { "model": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "hardware": null, "executed_runtime_revision": null, "input_tokens": null, "output_tokens": null, "repetitions": null, "measurement_scope": "publisher capability evaluation, not local speed", "benchmark": "HLE without tools", "population": "full set", "tools": false }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L51", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" }, { "label": "HLE without tools, text-only subset", "metric": "HLE", "value": 44.0, "unit": "reported score", "statistic": "reported value", "uncertainty": null, "conditions": { "model": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "hardware": null, "executed_runtime_revision": null, "input_tokens": null, "output_tokens": null, "repetitions": null, "measurement_scope": "publisher capability evaluation, not local speed", "benchmark": "HLE without tools", "population": "text-only subset", "tools": false }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L51", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" }, { "label": "DeepSWE v1.1", "metric": "DeepSWE", "value": 77.0, "unit": "reported score", "statistic": "reported value", "uncertainty": null, "conditions": { "model": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "hardware": null, "executed_runtime_revision": null, "input_tokens": null, "output_tokens": null, "repetitions": null, "measurement_scope": "publisher capability evaluation, not local speed", "benchmark": "DeepSWE v1.1", "harness": "Mini-SWE-V2" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L53", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" }, { "label": "Terminal-Bench 4.0", "metric": "Terminal_Bench", "value": 42.3, "unit": "reported score", "statistic": "reported value", "uncertainty": null, "conditions": { "model": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "hardware": null, "executed_runtime_revision": null, "input_tokens": null, "output_tokens": null, "repetitions": null, "measurement_scope": "publisher capability evaluation, not local speed", "benchmark": "Terminal-Bench 4.0", "harness": "Claude Code" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md#L52", "source_document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a" } ] ``` --- # Qwen3.8-Flash-Next: official model and deployment card Publisher model card with serving links and distinct thinking-mode settings. Use its exact artifact identity alongside the selected engine recipe. - Type: Official model card - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen/Qwen3.8-Flash-Next - Original author: Qwen - Original project: https://huggingface.co/Qwen/Qwen3.8-Flash-Next - Pinned document: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md - Source revision: de4b8e4d43b917e7706784d8bb445c9af86a3540 - Source commit time (not experiment date): 2026-08-27T05:03:36.000Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM, SGLang, TokenSpeed - Hardware topics: - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "de4b8e4d43b917e7706784d8bb445c9af86a3540", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The card separates thinking and non-thinking sampling. - Serving references point to model-specific vLLM, SGLang and TokenSpeed documentation. ## Limits - Runtime support and usable context depend on the selected build and precision. ## License declaration {"declared_spdx": "other", "declared_name": "qwen-community-1.0", "scope": "Publisher model-card declaration; inspect the linked LICENSE and artifact-specific terms."} ## Editorial answer Publisher model card with serving links and distinct thinking-mode settings. Use its exact artifact identity alongside the selected engine recipe. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## How is the model’s size counted? The card describes a 125B language-model component with 6B active parameters, plus 51B of n-gram embeddings and a 4B multi-token prediction component. The vLLM recipe uses a 176B summary label. Those counts cover different components; neither label should be substituted for the size of a downloaded artifact. The card lists 48 layers and a repeated hybrid of Gated DeltaNet and sparse attention. Sources: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L46 ## Is the open checkpoint natively a million-token model? The published native window is 262,144 tokens. Reaching 1,000,000 tokens requires the documented extension configuration; the hosted Qwen3.8-Flash product and the open Flash-Next checkpoint also have different production features. Record the actual RoPE settings and total input-plus-output budget before evaluating long tasks. Sources: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L71 ## How do thinking and non-thinking settings differ? Thinking mode uses temperature 1.0, top_p 0.95 and presence_penalty 0.0; non-thinking uses 0.7, 0.80 and 1.5 respectively. Both recommendations retain top_k 20, min_p 0.0 and repetition_penalty 1.0. The card also preserves historical thinking blocks by default. Changing that conversation policy changes both the prompt and cache behavior. Sources: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L342 ## How should its benchmark table be read? The coding evaluations specify harnesses, sampling and context. In particular, DeepSWE reports the better result from Claude Code and mini-SWE-agent rather than one universally shared harness. For vision tables, pass@3 and the mean over three attempts are different statistics. Keep the footnote for the selected task instead of treating every published cell as directly comparable. Sources: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L227 ## Attributed configuration facts ```json [ { "name": "LM component / active / n-gram / MTP", "value": "125B / 6B / 51B / 4B", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L46", "source_document_sha256": "35ca37ccc366f1ba478dab33841a2c0c18ce53fd62f291ca05341f7728b225b2" }, { "name": "Layers", "value": 48, "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L50", "source_document_sha256": "35ca37ccc366f1ba478dab33841a2c0c18ce53fd62f291ca05341f7728b225b2" }, { "name": "Native / extended context", "value": "262144 / up to 1000000 tokens", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L71", "source_document_sha256": "35ca37ccc366f1ba478dab33841a2c0c18ce53fd62f291ca05341f7728b225b2" }, { "name": "Thinking sampling", "value": "temperature 1.0; top_p 0.95; top_k 20", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L342", "source_document_sha256": "35ca37ccc366f1ba478dab33841a2c0c18ce53fd62f291ca05341f7728b225b2" }, { "name": "Non-thinking sampling", "value": "temperature 0.7; top_p 0.80; presence_penalty 1.5", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/de4b8e4d43b917e7706784d8bb445c9af86a3540/README.md#L343", "source_document_sha256": "35ca37ccc366f1ba478dab33841a2c0c18ce53fd62f291ca05341f7728b225b2" } ] ``` --- # GLM-5.3-Flash: official model and deployment card Official Flash checkpoint and local-serving references. Its reasoning effort and chat-template controls matter when attempting evaluation reproduction. - Type: Official model card - Evidence: source_reported; site reproduced: false - Model family: GLM - Model: zai-org/GLM-5.3-Flash - Original author: zai-org - Original project: https://huggingface.co/zai-org/GLM-5.3-Flash - Pinned document: https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md - Source revision: 46a050bec13002ad29ee795c16e61cf4fa3c4a9c - Source commit time (not experiment date): 2026-10-15T06:32:52.000Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM, SGLang, Transformers, KTransformers, TokenSpeed - Hardware topics: - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "46a050bec13002ad29ee795c16e61cf4fa3c4a9c", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The card specifies low/high/max reasoning effort and a default of max. - The clear_thinking chat setting is a separate control. ## Limits - Flash and the full GLM-5.3 checkpoint have separate artifacts and terms. ## License declaration {"declared_spdx": "mit", "declared_name": null, "scope": "Publisher model-card declaration; inspect the linked LICENSE and artifact-specific terms."} ## Editorial answer Official Flash checkpoint and local-serving references. Its reasoning effort and chat-template controls matter when attempting evaluation reproduction. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What distinguishes Flash from the full GLM model? The Flash card describes a newly trained, natively multimodal model with 320B total and 18B active parameters, using sparse and linear attention with mHC. The deployment recipe rounds the total differently at 321B. This page retains the publisher’s wording and the recipe’s number in their own records; neither establishes memory use under a serving workload. Sources: https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md#L25 ## Which reasoning controls affect reproduction? reasoning_effort accepts low, high and max. Omitting it, or supplying another value, selects max; the publisher asks benchmark reproductions to retain that setting. clear_thinking is independent and defaults to false. The card recommends explicitly enabling clear_thinking for chat. Keep both settings in an evaluation record rather than inferring one from the other. Sources: https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md#L45 ## Why are benchmark context budgets different? HLE with tools uses a 300,000-token managed context and a 163,840-token generation ceiling, with temperature 1.0 and top_p 0.95. DeepSWE instead specifies 400K context, temperature 0.95, top_p 1.0 and a six-hour timeout. These are task-specific harness settings, not one global chat preset or a local speed measurement. Sources: https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md#L50 ## Where should a deployment begin? The card links model-specific vLLM, SGLang, TokenSpeed, Transformers and other integration guides. Choose a versioned engine route, then match the actual artifact and hardware profile. A link in the publisher card establishes an intended integration path; this collection has not installed those stacks or verified their performance on a reader’s device. Sources: https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md#L32 ## Attributed configuration facts ```json [ { "name": "Total / active parameters as counted by card", "value": "320B / 18B", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md#L25", "source_document_sha256": "e6c5d41400e28f5a89418a0d15b09b9c113314a7553d40d7d4a8a72fcdb87d02" }, { "name": "Reasoning effort", "value": "low / high / max; default max", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md#L45", "source_document_sha256": "e6c5d41400e28f5a89418a0d15b09b9c113314a7553d40d7d4a8a72fcdb87d02" }, { "name": "Chat thinking cleanup", "value": "clear_thinking false by default; true recommended for chat", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md#L46", "source_document_sha256": "e6c5d41400e28f5a89418a0d15b09b9c113314a7553d40d7d4a8a72fcdb87d02" }, { "name": "HLE-with-tools context / output cap", "value": "300000 / 163840 tokens", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/zai-org/GLM-5.3-Flash/blob/46a050bec13002ad29ee795c16e61cf4fa3c4a9c/README.md#L50", "source_document_sha256": "e6c5d41400e28f5a89418a0d15b09b9c113314a7553d40d7d4a8a72fcdb87d02" } ] ``` --- # GLM-5.3: official model and deployment card The full GLM-5.3 model card, with serving routes and evaluation notes. Keep it separate from Flash and from quantized community conversions. - Type: Official model card - Evidence: source_reported; site reproduced: false - Model family: GLM - Model: zai-org/GLM-5.3 - Original author: zai-org - Original project: https://huggingface.co/zai-org/GLM-5.3 - Pinned document: https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md - Source revision: ba32f8c19c8738e2c57c71798c601d477a7bf844 - Source commit time (not experiment date): 2026-10-14T12:22:48.000Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM, SGLang, Transformers, KTransformers, TokenSpeed - Hardware topics: - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "ba32f8c19c8738e2c57c71798c601d477a7bf844", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Reasoning effort and conversation handling are documented with the checkpoint. - The card links a separate Ascend deployment route. ## Limits - Published capability evaluations do not demonstrate latency on the reader’s hardware. ## License declaration {"declared_spdx": "other", "declared_name": "glm-5.3", "scope": "Publisher model-card declaration; inspect the linked LICENSE and artifact-specific terms."} ## Editorial answer The full GLM-5.3 model card, with serving routes and evaluation notes. Keep it separate from Flash and from quantized community conversions. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact does this page identify? This is the full GLM-5.3 card, separate from Flash and from community quantizations. Its local-serving section includes a distinct Ascend route. Use the exact checkpoint and the selected conversion as separate identities. The companion recipe’s 743B total and 39B active labels belong to that recipe rather than a measured memory receipt. Sources: https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L41 ## What is the correct reasoning configuration? The publisher’s controls match the documented low, high and max reasoning-effort levels, defaulting to max. clear_thinking remains a separate chat-template option with a false default. Retaining the default effort matters for benchmark reproduction; enabling clear_thinking for a chat workflow should be recorded as its own configuration choice. Sources: https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L55 ## How much of the published score can be reproduced from a headline? The card reports 66.9 on DeepSWE v1.1 and 88.2 on Terminal-Bench 2.1. Their footnotes use different sampling, context and timeout settings. Terminal-Bench 3.0 is another protocol: three rollouts per task, isolated task containers, up to 600 turns and a ten-hour timeout. A score from one task is not a prediction of workstation throughput or another task’s accuracy. Sources: https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L26 ## What should be retained before comparing results? Keep the exact task revision, harness, tool policy, reasoning effort, maximum output and treatment of failures. Some published tasks change installation checks or use a fallback score when a run fails. Those decisions are part of the measurement. Read the footnote for the specific benchmark before comparing this card with a report from another author. Sources: https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L58 ## Attributed configuration facts ```json [ { "name": "Reasoning effort", "value": "low / high / max; default max", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L55", "source_document_sha256": "ed1c0a4563c437a32f8953d637a1db9bd831de1b24bd3062a5ace9e04340b1cf" }, { "name": "HLE-with-tools context / output cap", "value": "300000 / 163840 tokens", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L60", "source_document_sha256": "ed1c0a4563c437a32f8953d637a1db9bd831de1b24bd3062a5ace9e04340b1cf" }, { "name": "DeepSWE context / timeout", "value": "400K tokens / 6 hours", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L62", "source_document_sha256": "ed1c0a4563c437a32f8953d637a1db9bd831de1b24bd3062a5ace9e04340b1cf" }, { "name": "Terminal-Bench 3.0 repeats / turns / timeout", "value": "3 rollouts / 600 agent turns / 10 hours", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L64", "source_document_sha256": "ed1c0a4563c437a32f8953d637a1db9bd831de1b24bd3062a5ace9e04340b1cf" } ] ``` ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "DeepSWE v1.1", "metric": "DeepSWE", "value": 66.9, "unit": "reported score", "statistic": "reported value", "uncertainty": null, "conditions": { "model": "zai-org/GLM-5.3", "model_revision": "ba32f8c19c8738e2c57c71798c601d477a7bf844", "hardware": null, "executed_runtime_revision": null, "input_tokens": null, "output_tokens": null, "repetitions": null, "measurement_scope": "publisher capability evaluation, not local speed", "benchmark": "DeepSWE v1.1", "harness": "mini-swe-agent", "temperature": 0.95, "top_p": 1.0, "context": "400K", "timeout_hours": 6, "reasoning_effort": "max" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L26", "source_document_sha256": "ed1c0a4563c437a32f8953d637a1db9bd831de1b24bd3062a5ace9e04340b1cf" }, { "label": "Terminal-Bench 2.1", "metric": "Terminal_Bench", "value": 88.2, "unit": "reported score", "statistic": "reported value", "uncertainty": null, "conditions": { "model": "zai-org/GLM-5.3", "model_revision": "ba32f8c19c8738e2c57c71798c601d477a7bf844", "hardware": null, "executed_runtime_revision": null, "input_tokens": null, "output_tokens": null, "repetitions": null, "measurement_scope": "publisher capability evaluation, not local speed", "benchmark": "Terminal-Bench 2.1", "harness": "Claude Code 2.1.207", "temperature": 1.0, "top_p": 1.0, "max_new_tokens": 65536, "timeout_hours": 6, "reasoning_effort": "max" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://huggingface.co/zai-org/GLM-5.3/blob/ba32f8c19c8738e2c57c71798c601d477a7bf844/README.md#L24", "source_document_sha256": "ed1c0a4563c437a32f8953d637a1db9bd831de1b24bd3062a5ace9e04340b1cf" } ] ``` --- # Kimi-K3: official model and deployment card Publisher checkpoint and deployment references for Kimi K3, including its preserved-thinking conversation requirements. - Type: Official model card - Evidence: source_reported; site reproduced: false - Model family: Kimi - Model: moonshotai/Kimi-K3 - Original author: moonshotai - Original project: https://huggingface.co/moonshotai/Kimi-K3 - Pinned document: https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/README.md - Source revision: f831ab66814297da540d832a5235f8e904f29d06 - Source commit time (not experiment date): 2026-09-02T02:22:41.000Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM, SGLang, TokenSpeed - Hardware topics: - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "f831ab66814297da540d832a5235f8e904f29d06", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Thinking is always enabled; the card exposes reasoning_effort levels. - Multi-turn usage preserves complete assistant reasoning and tool-call messages. ## Limits - A hosted API alias is not an immutable local checkpoint. - Review the named Kimi license before redistribution or commercial use. ## License declaration {"declared_spdx": "other", "declared_name": "kimi-k3", "scope": "Publisher model-card declaration; inspect the linked LICENSE and artifact-specific terms."} ## Editorial answer Publisher checkpoint and deployment references for Kimi K3, including its preserved-thinking conversation requirements. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What is the published architecture and context? Moonshot describes Kimi K3 as a 2.8T-parameter open-weight model with native vision and a one-million-token context, built around Kimi Delta Attention and Attention Residuals. These are architecture and capability declarations. A trillion-parameter label does not tell a reader which quantized file will load, how much KV memory remains or how many users it will serve. Sources: https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/README.md#L40 ## How should a conversation preserve reasoning? Thinking is always active in the documented usage. Keep complete assistant reasoning and tool-call messages when sending subsequent turns, and set the desired reasoning effort explicitly. Dropping the reasoning history is a prompt transformation; it is not equivalent to reproducing the publisher’s agent configuration. The minimal API example’s short output limit should not be mistaken for the benchmark budget. Sources: https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/README.md#L625 ## Which sampling and harness conditions matter? The evaluation notes use max reasoning effort and temperature 1.0. Single-step tasks use top_p 0.95, while agent tasks use 1.0. Tool-free and tool-augmented scores appear together in several cells and must be read in that order. DeepSWE also distinguishes the Kimi Code harness from the separately cited mini-SWE-agent result. Sources: https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/README.md#L582 ## What does native MXFP4 mean for deployment? The card has a dedicated native MXFP4 section; select that artifact and a compatible kernel path rather than treating every four-bit conversion as equivalent. The recipe still labels its footprint as an estimate. Preserve scale format, activation precision, engine image and parallelism, and review the custom Kimi license for the intended application. Sources: https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/README.md#L611 ## Attributed configuration facts ```json [ { "name": "Total parameters / context", "value": "2.8T / 1 million tokens", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/README.md#L40", "source_document_sha256": "57de265b5842dfa465c6e73b368b0e15a89b8793b5450528dad577da202cc6fe" }, { "name": "Evaluation reasoning / temperature", "value": "max / 1.0", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/README.md#L582", "source_document_sha256": "57de265b5842dfa465c6e73b368b0e15a89b8793b5450528dad577da202cc6fe" }, { "name": "Single-step / agentic top_p", "value": "0.95 / 1.0", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/README.md#L582", "source_document_sha256": "57de265b5842dfa465c6e73b368b0e15a89b8793b5450528dad577da202cc6fe" } ] ``` --- # MiniMax-M3: official model and deployment card Original M3 weights and inference guidance, linking several local engines. This entry identifies the upstream model rather than a Spark-specific conversion. - Type: Official model card - Evidence: source_reported; site reproduced: false - Model family: MiniMax - Model: MiniMaxAI/MiniMax-M3 - Original author: MiniMaxAI - Original project: https://huggingface.co/MiniMaxAI/MiniMax-M3 - Pinned document: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/f0e1c1e04d40177e4673a22097036854f536e9c0/README.md - Source revision: f0e1c1e04d40177e4673a22097036854f536e9c0 - Source commit time (not experiment date): 2026-07-23T04:25:20.000Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM, SGLang, Transformers, KTransformers - Hardware topics: - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "f0e1c1e04d40177e4673a22097036854f536e9c0", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The publisher links model-specific engine documentation. - Sampling recommendations are separate from a community benchmark’s actual controls. ## Limits - Model terms and a converted artifact’s terms must both be checked. ## License declaration {"declared_spdx": "other", "declared_name": "minimax-community", "scope": "Publisher model-card declaration; inspect the linked LICENSE and artifact-specific terms."} ## Editorial answer Original M3 weights and inference guidance, linking several local engines. This entry identifies the upstream model rather than a Spark-specific conversion. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Why does the model-size number differ between sources? The official card gives approximately 428B total and 23B activated parameters, whereas the vLLM recipe lists 427B and 26B. The difference is unresolved in these references. We preserve both source declarations instead of averaging them or silently choosing one. The card advertises a native multimodal model with a 1M context window. Sources: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/f0e1c1e04d40177e4673a22097036854f536e9c0/README.md#L34 ## What does the sparse-attention speed claim mean? MiniMax describes relative prefill and decode improvements over M2 at a one-million-token context. The card does not provide a matching workstation measurement for the reader’s hardware in that sentence. Treat it as a publisher comparison under its own conditions, and use the separately dated three-Spark operator report for an actual deployment account. Sources: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/f0e1c1e04d40177e4673a22097036854f536e9c0/README.md#L38 ## Which sampling settings are recommended? The card recommends temperature 1.0 and top_p 0.95. Those recommendations differ from a greedy regression test at temperature zero. The three-Spark report contains both settings and reports different outcomes, so a reader should not combine those samples into a single accuracy or speed figure. Sources: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/f0e1c1e04d40177e4673a22097036854f536e9c0/README.md#L90 ## How should a local serving route be chosen? The publisher’s local-deployment section links engine integrations. The companion recipe defaults to a large BF16 artifact, while the three-Spark deployment uses an NVIDIA NVFP4 conversion, a DSpark drafter and custom tensor-parallel patches. Those are different artifacts and environments. Capture the conversion and runtime revisions before using either as a reproduction plan. Sources: https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/f0e1c1e04d40177e4673a22097036854f536e9c0/README.md#L66 ## Attributed configuration facts ```json [ { "name": "Card total / active parameters", "value": "approximately 428B / 23B", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/f0e1c1e04d40177e4673a22097036854f536e9c0/README.md#L34", "source_document_sha256": "6148856dbf8be26dcf3f9b913be0252c38454e852108fdab01212fe965f3e327" }, { "name": "Published context", "value": "1M tokens", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/f0e1c1e04d40177e4673a22097036854f536e9c0/README.md#L34", "source_document_sha256": "6148856dbf8be26dcf3f9b913be0252c38454e852108fdab01212fe965f3e327" }, { "name": "Recommended sampling", "value": "temperature 1.0; top_p 0.95", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/MiniMaxAI/MiniMax-M3/blob/f0e1c1e04d40177e4673a22097036854f536e9c0/README.md#L90", "source_document_sha256": "6148856dbf8be26dcf3f9b913be0252c38454e852108fdab01212fe965f3e327" } ] ``` --- # Step-3.7-Flash: official model and deployment card Official Step 3.7 Flash card with serving examples and high-memory workstation options, including distinct engine paths. - Type: Official model card - Evidence: source_reported; site reproduced: false - Model family: Step - Model: stepfun-ai/Step-3.7-Flash - Original author: stepfun-ai - Original project: https://huggingface.co/stepfun-ai/Step-3.7-Flash - Pinned document: https://huggingface.co/stepfun-ai/Step-3.7-Flash/blob/5f6244077ac62e04eec3f320501ff8c2b293373a/README.md - Source revision: 5f6244077ac62e04eec3f320501ff8c2b293373a - Source commit time (not experiment date): 2026-06-03T04:11:57.000Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM, SGLang, Transformers, llama.cpp - Hardware topics: apple, amd, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "5f6244077ac62e04eec3f320501ff8c2b293373a", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Local deployment sections distinguish vLLM, SGLang and Transformers. - The card discusses high-memory Apple, AMD and NVIDIA workstation paths. ## Limits - Hardware mentions are publisher guidance, not a Run AI Here compatibility test. ## License declaration {"declared_spdx": "apache-2.0", "declared_name": null, "scope": "Publisher model-card declaration; inspect the linked LICENSE and artifact-specific terms."} ## Editorial answer Official Step 3.7 Flash card with serving examples and high-memory workstation options, including distinct engine paths. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What is included in Step 3.7 Flash? The publisher describes about 198B parameters in total, combining a 196B language backbone and a 1.8B vision encoder, with roughly 11B active parameters per token. The advertised context is 256K tokens and reasoning has low, medium and high levels. These architecture labels do not establish a local memory budget. Sources: https://huggingface.co/stepfun-ai/Step-3.7-Flash/blob/5f6244077ac62e04eec3f320501ff8c2b293373a/README.md#L17 ## Is the advertised throughput a desktop benchmark? The introduction advertises up to 400 tokens per second, but does not identify a matching local workstation configuration in that sentence. We do not reuse it as a measured Spark, Apple or consumer-GPU result. Its capability scores also name distinct tasks and tool settings, such as SimpleVQA with search and V* with Python. Sources: https://huggingface.co/stepfun-ai/Step-3.7-Flash/blob/5f6244077ac62e04eec3f320501ff8c2b293373a/README.md#L17 ## Which local engine paths are documented? The card separates vLLM, SGLang, Transformers and llama.cpp paths. Transformers is presented as a debugging and verification route. The vLLM recipe has explicit precision and multi-GPU choices; the existence of a llama.cpp section does not mean a generic GGUF or an arbitrary runtime revision supports the model. Sources: https://huggingface.co/stepfun-ai/Step-3.7-Flash/blob/5f6244077ac62e04eec3f320501ff8c2b293373a/README.md#L124 ## What must be fixed before testing? Choose the exact weight format, reasoning level, parser and engine version, then retain prompt/output lengths and cache policy. A vision request also needs its image preprocessing and encoder placement. Keep a short load check separate from a timed or quality evaluation, and attribute all figures here to the publisher until an independent run exists. Sources: https://huggingface.co/stepfun-ai/Step-3.7-Flash/blob/5f6244077ac62e04eec3f320501ff8c2b293373a/README.md#L128 ## Attributed configuration facts ```json [ { "name": "Total / language / vision / active parameters", "value": "198B / 196B / 1.8B / approximately 11B", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/stepfun-ai/Step-3.7-Flash/blob/5f6244077ac62e04eec3f320501ff8c2b293373a/README.md#L17", "source_document_sha256": "b0d04c993d40a28fff3ecac99045de502e75b0c0635e233f2aa8614783972519" }, { "name": "Published context / reasoning levels", "value": "256K / low, medium, high", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/stepfun-ai/Step-3.7-Flash/blob/5f6244077ac62e04eec3f320501ff8c2b293373a/README.md#L17", "source_document_sha256": "b0d04c993d40a28fff3ecac99045de502e75b0c0635e233f2aa8614783972519" } ] ``` --- # MiMo-V2.6-Flash-RL: official model and deployment card The RL checkpoint’s original card, with engine examples, parallelism and parser requirements. Keep this artifact separate from the MOPD variant. - Type: Official model card - Evidence: source_reported; site reproduced: false - Model family: MiMo - Model: XiaomiMiMo/MiMo-V2.6-Flash-RL - Original author: XiaomiMiMo - Original project: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL - Pinned document: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md - Source revision: dcd3fd83fb181622fbc606fbb1709901982a62c2 - Source commit time (not experiment date): 2026-10-09T06:30:29.000Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM, SGLang - Hardware topics: - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "dcd3fd83fb181622fbc606fbb1709901982a62c2", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The SGLang and vLLM examples use different parallelism settings. - The model-specific parsers and recommended sampling remain part of the configuration. ## Limits - The related recipe defaults to MOPD; select the RL variant when reproducing this checkpoint. ## License declaration {"declared_spdx": "mit", "declared_name": null, "scope": "Publisher model-card declaration; inspect the linked LICENSE and artifact-specific terms."} ## Editorial answer The RL checkpoint’s original card, with engine examples, parallelism and parser requirements. Keep this artifact separate from the MOPD variant. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which MiMo checkpoint is this? This is the RL checkpoint, not MOPD. Its card describes 309B total and 15B activated parameters, with a 1M-token context. The companion vLLM recipe now defaults to the MOPD artifact, so copying that default would change the weights. Always record the complete repository name and revision when following the recipe. Sources: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L80 ## Why do the two engine examples use different topology? The SGLang example uses tensor parallelism eight and data parallelism two, with DP attention and an EAGLE speculative path. The vLLM example uses tensor parallelism four and its own memory and parser settings. They are alternative launch examples; neither is a controlled comparison of two engines on identical hardware. Sources: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L167 ## Which model-specific controls belong in a record? Both serving paths select the mimo reasoning and tool-call parsers. The card recommends temperature 1.0 and top_p 0.95. The architecture table separates sliding-window and global-attention layers, and describes a multi-layer speculative decoder. A conventional full-attention KV estimate cannot simply be pasted over this hybrid design. Sources: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L207 ## How should the model-card benchmarks be interpreted? The results table labels a MiMo-V2.6 Flash column alongside other products; it does not establish speed on the topology of either local launch example. Keep the benchmark column identity, task protocol and actual checkpoint evidence separate. This record uses architecture and launch facts and does not turn those product-level quality cells into RL-specific workstation measurements. Sources: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L98 ## Attributed configuration facts ```json [ { "name": "Total / active parameters", "value": "309B / 15B", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L80", "source_document_sha256": "132596ea76492320f5283756015e3f696f289b87fcf38e4b71602a8c901b0557" }, { "name": "Published context", "value": "1M tokens", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L81", "source_document_sha256": "132596ea76492320f5283756015e3f696f289b87fcf38e4b71602a8c901b0557" }, { "name": "SGLang TP / DP example", "value": "8 / 2", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L173", "source_document_sha256": "132596ea76492320f5283756015e3f696f289b87fcf38e4b71602a8c901b0557" }, { "name": "vLLM TP example", "value": 4, "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L197", "source_document_sha256": "132596ea76492320f5283756015e3f696f289b87fcf38e4b71602a8c901b0557" }, { "name": "Recommended sampling", "value": "temperature 1.0; top_p 0.95", "classification": "publisher_declaration", "evidence_url": "https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL/blob/dcd3fd83fb181622fbc606fbb1709901982a62c2/README.md#L207", "source_document_sha256": "132596ea76492320f5283756015e3f696f289b87fcf38e4b71602a8c901b0557" } ] ``` --- # DeepSeek-V4.1-Flash: vLLM recipe The current recipe selects the 1017 checkpoint by default. Check precision, engine version and hardware-specific flags before serving. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: deepseek-ai/DeepSeek-V4.1-Flash - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.31.0", "nightly_required": false, "model_context_limit_tokens": 1048576, "compatible_strategies": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "hardware_claims": [ { "profile": "h100", "maintainer_status": "verified" }, { "profile": "h200", "maintainer_status": "verified" }, { "profile": "b200", "maintainer_status": "verified" }, { "profile": "gb200", "maintainer_status": "verified" }, { "profile": "gb300", "maintainer_status": "verified" }, { "profile": "b300", "maintainer_status": "verified" }, { "profile": "mi300x", "maintainer_status": "verified" }, { "profile": "mi325x", "maintainer_status": "verified" }, { "profile": "mi350x", "maintainer_status": "verified" }, { "profile": "mi355x", "maintainer_status": "verified" }, { "profile": "dgx_station_gb300", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-17", "effective_default_minimum_runtime_version": "0.31.0" }, "deployment_options": [ { "variant": "default", "model_id": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 614, "minimum_runtime_version": "0.31.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "deepseek-ai/DeepSeek-V4.1-Flash", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 614, "minimum_runtime_version": "0.31.0", "extra_args": [], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.31.0. - Default artifact: deepseek-ai/DeepSeek-V4.1-Flash-1017. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - The FP8 variant still points to the base V4.1 artifact; default and FP8 are different identities. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned DeepSeek-V4.1-Flash recipe defaults to deepseek-ai/DeepSeek-V4.1-Flash-1017 (fp8). Its effective default-variant minimum is vLLM 0.31.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies deepseek-ai/DeepSeek-V4.1-Flash as its model family entry and selects deepseek-ai/DeepSeek-V4.1-Flash-1017 by default. The default precision label is fp8; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114 ## What needs special attention for this model? The default is the 1017 refresh, while the separate FP8 option names the original Flash artifact. The default’s fp8 label covers a mixed checkpoint, including MXFP4 experts and MXFP8 dense weights. On the documented H100 route, Engram offload and a 4,096 batched-token cap are memory controls; the latter is not the maximum context length. AMD has a different nightly image and disables adaptive verification in its speculative configuration. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L31 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L191 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L125 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L423 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L50", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Total parameters as counted by recipe", "value": "552B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L51", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Active parameters as counted by recipe", "value": "8-16B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L52", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Native context declaration (tokens)", "value": 1048576, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L53", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Model-wide minimum vLLM", "value": "0.31.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L42", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Effective default-variant minimum vLLM", "value": "0.31.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L42", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Default artifact", "value": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Default precision label", "value": "fp8", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 614, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L125", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--tokenizer-mode", "deepseek_v41" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding", "text_only", "encoder_parallel" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" }, { "name": "Features selected by default", "value": [ "spec_decoding", "text_only" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4.1-Flash.yaml#L114", "source_document_sha256": "8fa89be8c171136cc34460cd1d8d64134573c79ee64245d942a057ee4a4d449e" } ] ``` --- # Qwen3.8-Flash-Next: vLLM recipe Model-specific serving recipe with NVFP4, FP8 and BF16 paths, hardware overrides and optional acceleration. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen/Qwen3.8-Flash-Next - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.30.0", "nightly_required": true, "model_context_limit_tokens": 262144, "compatible_strategies": [ "single_node_tp", "single_node_tep", "single_node_dep" ], "hardware_claims": [ { "profile": "h200", "maintainer_status": "verified" }, { "profile": "h100", "maintainer_status": "verified" }, { "profile": "gb200", "maintainer_status": "verified" }, { "profile": "gb300", "maintainer_status": "verified" }, { "profile": "mi355x", "maintainer_status": "verified" }, { "profile": "rtx_pro_6000_4x", "maintainer_status": "verified" }, { "profile": "dgx_station_gb300", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-13", "effective_default_minimum_runtime_version": "0.30.0" }, "deployment_options": [ { "variant": "default", "model_id": "Inferact/Qwen3.8-Flash-Next-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 130, "minimum_runtime_version": "0.30.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "Qwen/Qwen3.8-Flash-Next-FP8", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 250, "minimum_runtime_version": "0.30.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "bf16", "model_id": "Qwen/Qwen3.8-Flash-Next", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 423, "minimum_runtime_version": "0.30.0", "extra_args": [], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.30.0. - Default artifact: Inferact/Qwen3.8-Flash-Next-NVFP4. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - The default points to an Inferact NVFP4 conversion, not the publisher’s BF16 repository. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned Qwen3.8-Flash-Next recipe defaults to Inferact/Qwen3.8-Flash-Next-NVFP4 (nvfp4). Its effective default-variant minimum is vLLM 0.30.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies Qwen/Qwen3.8-Flash-Next as its model family entry and selects Inferact/Qwen3.8-Flash-Next-NVFP4 by default. The default precision label is nvfp4; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89 ## What needs special attention for this model? The NVFP4 default is an Inferact conversion, not the original publisher’s unconverted artifact. The card’s parameter breakdown and this recipe’s summary count use different component boundaries. Native context is 262,144; the long-context feature is an extension. Preserve the prefix-cache state, hybrid-cache settings and the selected NEXTN or other drafting mode before timing a run. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L23 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L162 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L93 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L193 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L36", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Total parameters as counted by recipe", "value": "176B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L37", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Active parameters as counted by recipe", "value": "6B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L38", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Native context declaration (tokens)", "value": 262144, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L39", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Model-wide minimum vLLM", "value": "0.30.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L25", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Effective default-variant minimum vLLM", "value": "0.30.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L25", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Default artifact", "value": "Inferact/Qwen3.8-Flash-Next-NVFP4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Default precision label", "value": "nvfp4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 130, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L93", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--max-num-seqs", "256", "--gpu-memory-utilization", "0.90", "--enable-prefix-caching" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "single_node_dep" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding", "text_only", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" }, { "name": "Features selected by default", "value": [ "text_only", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-Flash-Next.yaml#L89", "source_document_sha256": "609bdc85409afce063766c0bd31ce9510f072f2bc1200944bdc0bb097c81ee00" } ] ``` --- # Qwen3.8-27B: vLLM recipe Dense Qwen3.8 deployment reference with separate quantization artifacts and workstation or accelerator profiles. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen/Qwen3.8-27B - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, ascend, intel, nvidia, spark - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.17.0", "nightly_required": false, "model_context_limit_tokens": 262144, "compatible_strategies": [ "single_node_tp" ], "hardware_claims": [ { "profile": "gb300", "maintainer_status": "verified" }, { "profile": "b200", "maintainer_status": "verified" }, { "profile": "rtx_5090", "maintainer_status": "verified" }, { "profile": "rtx_5090_2x", "maintainer_status": "verified" }, { "profile": "rtx_pro_6000", "maintainer_status": "verified" }, { "profile": "dgx_spark_gb10", "maintainer_status": "verified" }, { "profile": "ascend_950pr", "maintainer_status": "verified" }, { "profile": "arc_pro_b60", "maintainer_status": "verified" }, { "profile": "arc_pro_b70", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-15", "effective_default_minimum_runtime_version": "0.17.0" }, "deployment_options": [ { "variant": "default", "model_id": "Qwen/Qwen3.8-27B", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 67, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "Qwen/Qwen3.8-27B-FP8", "precision": "fp8", "supported_hardware": [ "h100", "h200", "b200", "b300", "gb200", "gb300", "mi300x", "mi325x", "mi355x", "rtx_5090", "rtx_5090_2x", "ascend_950pr" ], "model_revision": null, "declared_planning_minimum_gb": 38, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "Inferact/Qwen3.8-27B-NVFP4", "precision": "nvfp4", "supported_hardware": [ "b200", "b300", "gb200", "gb300", "rtx_5090", "rtx_5090_2x" ], "model_revision": null, "declared_planning_minimum_gb": 32, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4_nvidia", "model_id": "nvidia/Qwen3.8-27B-NVFP4", "precision": "nvfp4", "supported_hardware": [ "rtx_pro_6000", "dgx_spark_gb10" ], "model_revision": null, "declared_planning_minimum_gb": 32, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "int4", "model_id": "RedHatAI/Qwen3.8-27B-INT4", "precision": "int4", "supported_hardware": [ "h100", "h200", "b200", "b300", "gb200", "gb300", "rtx_5090", "rtx_5090_2x" ], "model_revision": null, "declared_planning_minimum_gb": 24, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "mxfp4", "model_id": "RedHatAI/Qwen3.8-27B-MXFP4", "precision": "mxfp4", "supported_hardware": [ "h100", "h200", "b200", "b300", "gb200", "gb300", "rtx_5090", "rtx_5090_2x" ], "model_revision": null, "declared_planning_minimum_gb": 24, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "ascend_w8a8", "model_id": "Eco-Tech/Qwen3.8-27B-w8a8", "precision": "int8", "supported_hardware": [ "ascend_950pr" ], "model_revision": null, "declared_planning_minimum_gb": 39, "minimum_runtime_version": "0.17.0", "extra_args": [ "--quantization", "ascend", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}", "--additional-config", "{\"enable_cpu_binding\":true}", "--max-num-batched-tokens", "16384" ], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.17.0. - Default artifact: Qwen/Qwen3.8-27B. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Per-variant hardware allowlists differ; do not inherit support across conversions. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned Qwen3.8-27B recipe defaults to Qwen/Qwen3.8-27B (bf16). Its effective default-variant minimum is vLLM 0.17.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies Qwen/Qwen3.8-27B as its model family entry and selects Qwen/Qwen3.8-27B by default. The default precision label is bf16; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L106 ## What needs special attention for this model? The default is BF16, and the listed 67 GB memory minimum is a recipe planning declaration. It is not a measured maximum-context peak. Converted FP8 and NVFP4 alternatives identify different repositories. Intel overrides and supported-hardware allowlists belong to the selected variant; do not apply NVIDIA settings merely because the model name is the same. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L40 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L326 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L173 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L211 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "dense", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L60", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Total parameters as counted by recipe", "value": "27B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L61", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Active parameters as counted by recipe", "value": "27B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L62", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Native context declaration (tokens)", "value": 262144, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L65", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Model-wide minimum vLLM", "value": "0.17.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L53", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Effective default-variant minimum vLLM", "value": "0.17.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L53", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Default artifact", "value": "Qwen/Qwen3.8-27B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L106", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Default precision label", "value": "bf16", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L106", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 67, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L173", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L106", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Shared base arguments (incomplete launch)", "value": [], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L106", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L106", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding", "text_only", "encoder_parallel", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L106", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" }, { "name": "Features selected by default", "value": [ "spec_decoding", "text_only", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-27B.yaml#L106", "source_document_sha256": "3876aa94eefed23ad770fd4112700190f5481d2037e39dffbe894b61a72eb98b" } ] ``` --- # Qwen3.8-2.4T-A95B: vLLM recipe Large MoE recipe with explicit multi-GPU strategies and distinct native versus extended context claims. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen/Qwen3.8-2.4T-A95B - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "nightly", "nightly_required": true, "model_context_limit_tokens": 262144, "compatible_strategies": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tep", "multi_node_dep", "multi_node_tp_pp", "multi_node_tp_dp", "pd_cluster" ], "hardware_claims": [ { "profile": "b300", "maintainer_status": "verified" }, { "profile": "gb300", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-09-14", "effective_default_minimum_runtime_version": "nightly" }, "deployment_options": [ { "variant": "default", "model_id": "Qwen/Qwen3.8-2.4T-A95B", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 5871, "minimum_runtime_version": "nightly", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "Qwen/Qwen3.8-2.4T-A95B-FP8", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 2996, "minimum_runtime_version": "nightly", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "mxfp4", "model_id": "Inferact/Qwen3.8-2.4T-A95B-MXFP4", "precision": "mxfp4", "supported_hardware": [ "mi300x", "mi325x", "mi355x" ], "model_revision": null, "declared_planning_minimum_gb": 1917, "minimum_runtime_version": "nightly", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "Inferact/Qwen3.8-2.4T-A95B-NVFP4", "precision": "nvfp4", "supported_hardware": [ "b200", "b300", "gb200", "gb300" ], "model_revision": null, "declared_planning_minimum_gb": 1737, "minimum_runtime_version": "nightly", "extra_args": [ "--linear-backend", "flashinfer_cutedsl" ], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4_reap25", "model_id": "RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25", "precision": "nvfp4", "supported_hardware": [ "b200", "b300", "gb200", "gb300" ], "model_revision": null, "declared_planning_minimum_gb": 1373, "minimum_runtime_version": "nightly", "extra_args": [], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: nightly. - Default artifact: Qwen/Qwen3.8-2.4T-A95B. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Nightly is required by this pinned recipe. - A four-GPU NVL4 tray is not an eight-GPU server; retain the actual topology. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned Qwen3.8-2.4T-A95B recipe defaults to Qwen/Qwen3.8-2.4T-A95B (bf16). Its effective default-variant minimum is vLLM nightly. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies Qwen/Qwen3.8-2.4T-A95B as its model family entry and selects Qwen/Qwen3.8-2.4T-A95B by default. The default precision label is bf16; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L102 ## What needs special attention for this model? This recipe requires nightly vLLM and a very large distributed deployment. Its native window is 262,144 tokens; the separate 1M extension has its own validation boundary. The default describes 24 GPUs as three eight-GPU B300 nodes or six four-GPU GB300 NVL4 trays. A tray is not an eight-GPU node, and total cluster capacity is not one automatically pooled memory space. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L31 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L159 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L106 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L237 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L50", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Total parameters as counted by recipe", "value": "2.4T", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L51", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Active parameters as counted by recipe", "value": "95B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L52", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Native context declaration (tokens)", "value": 262144, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L57", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Model-wide minimum vLLM", "value": "nightly", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L44", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Effective default-variant minimum vLLM", "value": "nightly", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L44", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Default artifact", "value": "Qwen/Qwen3.8-2.4T-A95B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L102", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Default precision label", "value": "bf16", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L102", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 5871, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L106", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L102", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Shared base arguments (incomplete launch)", "value": [], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L102", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tep", "multi_node_dep", "multi_node_tp_pp", "multi_node_tp_dp", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L102", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L102", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" }, { "name": "Features selected by default", "value": [ "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.8-2.4T-A95B.yaml#L102", "source_document_sha256": "dbaaa7ecb550abcd5a38bc81480b6c94799064883d215d42e67f929bfe393fb2" } ] ``` --- # GLM-5.3-Flash: vLLM recipe Serving recipe for Flash with FP8, NVFP4 and BF16 choices and model-specific engine settings. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: GLM - Model: zai-org/GLM-5.3-Flash - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, ascend, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.29.0", "nightly_required": true, "model_context_limit_tokens": 1048576, "compatible_strategies": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tp_pp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "hardware_claims": [ { "profile": "h100", "maintainer_status": "verified" }, { "profile": "b200", "maintainer_status": "verified" }, { "profile": "gb200", "maintainer_status": "verified" }, { "profile": "mi355x", "maintainer_status": "verified" }, { "profile": "ascend_950pr_8x", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-08", "effective_default_minimum_runtime_version": "0.29.0" }, "deployment_options": [ { "variant": "default", "model_id": "zai-org/GLM-5.3-Flash", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 386, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "RedHatAI/GLM-5.3-Flash-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 229, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "bf16", "model_id": "zai-org/GLM-5.3-Flash-BF16", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 772, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.29.0. - Default artifact: zai-org/GLM-5.3-Flash. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Source hardware verification labels belong to the maintainers, not this index. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned GLM-5.3-Flash recipe defaults to zai-org/GLM-5.3-Flash (fp8). Its effective default-variant minimum is vLLM 0.29.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies zai-org/GLM-5.3-Flash as its model family entry and selects zai-org/GLM-5.3-Flash by default. The default precision label is fp8; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L74 ## What needs special attention for this model? The recipe lists 321B total parameters while the publisher card says 320B; keep the attribution rather than manufacturing an exact reconciled total. The native FP8 default, NVFP4 conversion and BF16 option are separate artifacts. Hardware-specific sparse-attention and speculative-decoding settings matter, particularly for GB10 or Ascend routes. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L20 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L106 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L77 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L204 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L31", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Total parameters as counted by recipe", "value": "321B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L32", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Active parameters as counted by recipe", "value": "18B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L33", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Native context declaration (tokens)", "value": 1048576, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L34", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Model-wide minimum vLLM", "value": "0.29.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L22", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Effective default-variant minimum vLLM", "value": "0.29.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L22", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Default artifact", "value": "zai-org/GLM-5.3-Flash", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L74", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Default precision label", "value": "fp8", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L74", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 386, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L77", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L74", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Shared base arguments (incomplete launch)", "value": [], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L74", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tp_pp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L74", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L74", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" }, { "name": "Features selected by default", "value": [ "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3-Flash.yaml#L74", "source_document_sha256": "ffcd619ddd168d83e5517bc9fc00ca4b00e740f05cda8e6c0fc98149e672f92a" } ] ``` --- # GLM-5.3: vLLM recipe Full GLM-5.3 recipe, including quantization alternatives and accelerator-specific serving controls. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: GLM - Model: zai-org/GLM-5.3 - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, ascend, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.29.0", "nightly_required": false, "model_context_limit_tokens": 1048576, "compatible_strategies": [ "single_node_tp", "single_node_tep", "multi_node_tp", "multi_node_tp_pp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "hardware_claims": [ { "profile": "b300", "maintainer_status": "verified" }, { "profile": "h200", "maintainer_status": "verified" }, { "profile": "mi325x", "maintainer_status": "verified" }, { "profile": "mi350x", "maintainer_status": "verified" }, { "profile": "ascend_950pr_8x", "maintainer_status": "verified" }, { "profile": "dgx_station_gb300", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-15", "effective_default_minimum_runtime_version": "0.29.0" }, "deployment_options": [ { "variant": "default", "model_id": "zai-org/GLM-5.3", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 893, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "Inferact/GLM-5.3-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 558, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "w4a8", "model_id": "gpustack/GLM-5.3-W4A8", "precision": "int4", "supported_hardware": [ "h100", "h200" ], "model_revision": null, "declared_planning_minimum_gb": 447, "minimum_runtime_version": "0.29.0", "extra_args": [ "--enable-expert-parallel", "--trust-remote-code" ], "tensor_parallel_by_hardware": null }, { "variant": "bf16", "model_id": "zai-org/GLM-5.3-BF16", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 1786, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.29.0. - Default artifact: zai-org/GLM-5.3. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - The full and Flash models are distinct, even where the engine is shared. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned GLM-5.3 recipe defaults to zai-org/GLM-5.3 (fp8). Its effective default-variant minimum is vLLM 0.29.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies zai-org/GLM-5.3 as its model family entry and selects zai-org/GLM-5.3 by default. The default precision label is fp8; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L84 ## What needs special attention for this model? This is the full model rather than Flash. The base FP8 KV setting and the target precision are different quantities. Check the hardware override, tool parser and reasoning effort with the chosen artifact. The recipe’s 893 GB planning minimum cannot predict the usable context, concurrency or peak allocation of an eight-GPU deployment. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L31 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L181 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L105 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L255 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L41", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Total parameters as counted by recipe", "value": "743B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L42", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Active parameters as counted by recipe", "value": "39B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L43", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Native context declaration (tokens)", "value": 1048576, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L44", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Model-wide minimum vLLM", "value": "0.29.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L37", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Effective default-variant minimum vLLM", "value": "0.29.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L37", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Default artifact", "value": "zai-org/GLM-5.3", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L84", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Default precision label", "value": "fp8", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L84", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 893, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L105", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L84", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--kv-cache-dtype", "fp8_e4m3" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L84", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "multi_node_tp", "multi_node_tp_pp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L84", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L84", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" }, { "name": "Features selected by default", "value": [ "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/zai-org/GLM-5.3.yaml#L84", "source_document_sha256": "0ed75e46d1c66e8ac5adad351385b3a76bd01526895c597a5ec4f7570cc9fed6" } ] ``` --- # Kimi-K3: vLLM recipe Kimi K3 deployment recipe with separate MXFP4, NVFP4 and other artifact paths and parallelism strategies. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Kimi - Model: moonshotai/Kimi-K3 - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, ascend, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.29.0", "nightly_required": false, "model_context_limit_tokens": 1048576, "compatible_strategies": [ "single_node_tp", "multi_node_tp", "multi_node_tep", "multi_node_tp_pp", "multi_node_dep", "pd_cluster" ], "hardware_claims": [ { "profile": "h200", "maintainer_status": "verified" }, { "profile": "b200", "maintainer_status": "verified" }, { "profile": "b300", "maintainer_status": "verified" }, { "profile": "gb200", "maintainer_status": "verified" }, { "profile": "gb300", "maintainer_status": "verified" }, { "profile": "mi355x", "maintainer_status": "verified" }, { "profile": "ascend_910c", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-16", "effective_default_minimum_runtime_version": "0.29.0" }, "deployment_options": [ { "variant": "default", "model_id": "moonshotai/Kimi-K3", "precision": "mxfp4", "supported_hardware": [ "h100", "h200", "b200", "b300", "gb200", "gb300", "mi300x", "mi325x", "mi355x" ], "model_revision": null, "declared_planning_minimum_gb": 1680, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "RedHatAI/Kimi-K3-NVFP4", "precision": "nvfp4", "supported_hardware": [ "b200", "b300", "gb200", "gb300" ], "model_revision": null, "declared_planning_minimum_gb": 1650, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "ascend_w4a8", "model_id": "Eco-Tech/Kimi-K3-w4a8", "precision": "int4", "supported_hardware": [ "ascend_910c" ], "model_revision": null, "declared_planning_minimum_gb": 1711, "minimum_runtime_version": "0.29.0", "extra_args": [ "--quantization", "ascend", "--enable-prefix-caching", "--enable-expert-parallel", "--max-num-seqs", "16", "--max-model-len", "131072", "--max-num-batched-tokens", "24576", "--gpu-memory-utilization", "0.9", "--compilation-config", "{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}", "--mm-processor-cache-gb", "0", "--additional-config", "{\"enable_cpu_binding\":true, \"enable_flashcomm1\":true}", "--mm-encoder-tp-mode", "data", "--limit-mm-per-prompt", "{\"vision_chunk\": 2}", "--allowed-local-media-path", "/" ], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "RedHatAI/Kimi-K3-FP8-BLOCK", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 3360, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.29.0. - Default artifact: moonshotai/Kimi-K3. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Do not merge measurements made with different quantizations or parallelism. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned Kimi-K3 recipe defaults to moonshotai/Kimi-K3 (mxfp4). Its effective default-variant minimum is vLLM 0.29.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies moonshotai/Kimi-K3 as its model family entry and selects moonshotai/Kimi-K3 by default. The default precision label is mxfp4; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L132 ## What needs special attention for this model? The default MXFP4 footprint is explicitly described as estimated in the pinned recipe. Do not promote that estimate to a measured deployment just because the current model repository is available. Quantization-aware-trained MXFP4 with MXFP8 activations is a particular kernel path. Keep decode-context parallelism, tensor parallelism and draft options separate. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L26 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L201 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L137 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L443 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L39", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Total parameters as counted by recipe", "value": "2.8T", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L40", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Active parameters as counted by recipe", "value": "16 experts/token + shared (of 896 routed)", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L41", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Native context declaration (tokens)", "value": 1048576, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L42", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Model-wide minimum vLLM", "value": "0.29.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L32", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Effective default-variant minimum vLLM", "value": "0.29.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L32", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Default artifact", "value": "moonshotai/Kimi-K3", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L132", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Default precision label", "value": "mxfp4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L132", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 1680, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L137", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Default variant hardware allowlist", "value": [ "h100", "h200", "b200", "b300", "gb200", "gb300", "mi300x", "mi325x", "mi355x" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L132", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--trust-remote-code", "--gpu-memory-utilization", "0.95" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L132", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "multi_node_tp", "multi_node_tep", "multi_node_tp_pp", "multi_node_dep", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L132", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding", "text_only", "decode_context_parallelism" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L132", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" }, { "name": "Features selected by default", "value": [ "spec_decoding", "text_only", "decode_context_parallelism" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/moonshotai/Kimi-K3.yaml#L132", "source_document_sha256": "edd8e731474a0ae2d12612cca5b55b66f398eee13ee15fd87d49b803f177ebf6" } ] ``` --- # MiniMax-M3: vLLM recipe M3 serving recipe covering full and converted weights, hardware overrides and optional acceleration. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: MiniMax - Model: MiniMaxAI/MiniMax-M3 - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.24.0", "nightly_required": true, "model_context_limit_tokens": 1048576, "compatible_strategies": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tp_pp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "hardware_claims": [ { "profile": "h200", "maintainer_status": "verified" }, { "profile": "b200", "maintainer_status": "verified" }, { "profile": "b300", "maintainer_status": "verified" }, { "profile": "mi300x", "maintainer_status": "verified" }, { "profile": "mi355x", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-06", "effective_default_minimum_runtime_version": "0.24.0" }, "deployment_options": [ { "variant": "default", "model_id": "MiniMaxAI/MiniMax-M3", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 1025, "minimum_runtime_version": "0.24.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "mxfp8", "model_id": "MiniMaxAI/MiniMax-M3-MXFP8", "precision": "mxfp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 513, "minimum_runtime_version": "0.24.0", "extra_args": [ "--attention_config.indexer_kv_dtype", "fp8" ], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "nvidia/MiniMax-M3-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 257, "minimum_runtime_version": "0.24.0", "extra_args": [ "--trust-remote-code", "--kv-cache-dtype", "fp8", "--attention_config.backend", "FLASHINFER", "--attention_config.use_trtllm_attention", "true", "--attention_config.indexer_kv_dtype", "fp8", "--attention_config.minimax_m3_msa_decode_backend", "cutlass" ], "tensor_parallel_by_hardware": null }, { "variant": "mxfp4", "model_id": "amd/MiniMax-M3-MXFP4", "precision": "mxfp4", "supported_hardware": [ "mi355x" ], "model_revision": null, "declared_planning_minimum_gb": 257, "minimum_runtime_version": "0.24.0", "extra_args": [], "tensor_parallel_by_hardware": 8 } ] } ``` ## Reading notes - Recipe minimum engine version: 0.24.0. - Default artifact: MiniMaxAI/MiniMax-M3. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - A generic recipe is not proof that a patched three-Spark report used identical settings. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned MiniMax-M3 recipe defaults to MiniMaxAI/MiniMax-M3 (bf16). Its effective default-variant minimum is vLLM 0.24.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies MiniMaxAI/MiniMax-M3 as its model family entry and selects MiniMaxAI/MiniMax-M3 by default. The default precision label is bf16; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L127 ## What needs special attention for this model? The default is BF16 and is much larger than the three-Spark NVFP4 operator setup. The recipe’s 427B total / 26B active labels differ from the official card’s approximately 428B / 23B. Neither source is silently corrected here. The block-size setting, attention backend and DSpark choice travel with the selected precision and topology. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L21 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L252 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L134 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L290 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L34", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Total parameters as counted by recipe", "value": "427B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L38", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Active parameters as counted by recipe", "value": "26B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L39", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Native context declaration (tokens)", "value": 1048576, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L42", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Model-wide minimum vLLM", "value": "0.24.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L27", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Effective default-variant minimum vLLM", "value": "0.24.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L27", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Default artifact", "value": "MiniMaxAI/MiniMax-M3", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L127", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Default precision label", "value": "bf16", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L127", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 1025, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L134", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L127", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--block-size", "128" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L127", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tp_pp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L127", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding", "text_only", "thinking_always_on", "encoder_parallel" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L127", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" }, { "name": "Features selected by default", "value": [ "spec_decoding", "text_only", "thinking_always_on", "encoder_parallel" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/MiniMaxAI/MiniMax-M3.yaml#L127", "source_document_sha256": "1ffa05682199d5b354fa25116dfc19c8bbcdc782612b5a9eff774076b953c2fd" } ] ``` --- # Step-3.7-Flash: vLLM recipe Step 3.7 Flash engine recipe with separate BF16, FP8 and NVFP4 artifacts. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Step - Model: stepfun-ai/Step-3.7-Flash - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.23.0", "nightly_required": false, "model_context_limit_tokens": 262144, "compatible_strategies": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "hardware_claims": [ { "profile": "h200", "maintainer_status": "verified" }, { "profile": "b200", "maintainer_status": "verified" }, { "profile": "dgx_station_gb300", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-05-30", "effective_default_minimum_runtime_version": "0.23.0" }, "deployment_options": [ { "variant": "default", "model_id": "stepfun-ai/Step-3.7-Flash", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 475, "minimum_runtime_version": "0.23.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "stepfun-ai/Step-3.7-Flash-FP8", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 238, "minimum_runtime_version": "0.23.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "stepfun-ai/Step-3.7-Flash-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 119, "minimum_runtime_version": "0.23.0", "extra_args": [ "--quantization", "modelopt", "--kv-cache-dtype", "fp8", "--gpu-memory-utilization", "0.9", "--async-scheduling" ], "tensor_parallel_by_hardware": 4 } ] } ``` ## Reading notes - Recipe minimum engine version: 0.23.0. - Default artifact: stepfun-ai/Step-3.7-Flash. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Consult the selected artifact and source hardware profile together. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned Step-3.7-Flash recipe defaults to stepfun-ai/Step-3.7-Flash (bf16). Its effective default-variant minimum is vLLM 0.23.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies stepfun-ai/Step-3.7-Flash as its model family entry and selects stepfun-ai/Step-3.7-Flash by default. The default precision label is bf16; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L57 ## What needs special attention for this model? The BF16 route includes expert parallelism and disables cascade attention. Lower-precision variants are different artifacts. The default’s documented multi-GPU recommendation is not evidence of a single-Spark load. Keep reasoning level, parser and expert/tensor parallelism with a reproduction; all alter the workload or execution path. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L19 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L82 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L60 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L95 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L26", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Total parameters as counted by recipe", "value": "198B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L27", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Active parameters as counted by recipe", "value": "11B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L28", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Native context declaration (tokens)", "value": 262144, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L29", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Model-wide minimum vLLM", "value": "0.23.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L21", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Effective default-variant minimum vLLM", "value": "0.23.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L21", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Default artifact", "value": "stepfun-ai/Step-3.7-Flash", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L57", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Default precision label", "value": "bf16", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L57", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 475, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L60", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L57", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--trust-remote-code", "--enable-expert-parallel", "--disable-cascade-attn" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L57", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L57", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L57", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" }, { "name": "Features selected by default", "value": [ "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/stepfun-ai/Step-3.7-Flash.yaml#L57", "source_document_sha256": "77f9f16a2b23acb5d60a4c94361573d96380cd59ec55ddf3b84ab2b991956c21" } ] ``` --- # MiMo-V2.6-Flash-RL: vLLM recipe MiMo serving reference whose default artifact is MOPD and whose RL variant preserves the separate RL checkpoint. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: MiMo - Model: XiaomiMiMo/MiMo-V2.6-Flash-RL - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.31.0", "nightly_required": false, "model_context_limit_tokens": 1048576, "compatible_strategies": [ "single_node_tp", "single_node_tep", "multi_node_tp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "hardware_claims": [ { "profile": "h200", "maintainer_status": "verified" }, { "profile": "gb300", "maintainer_status": "verified" }, { "profile": "mi300x", "maintainer_status": "verified" }, { "profile": "mi325x", "maintainer_status": "verified" }, { "profile": "mi355x", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-07", "effective_default_minimum_runtime_version": "0.31.0" }, "deployment_options": [ { "variant": "default", "model_id": "XiaomiMiMo/MiMo-V2.6-Flash-MOPD", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 208, "minimum_runtime_version": "0.31.0", "extra_args": [ "--tensor-parallel-size", "4", "--gpu-memory-utilization", "0.95" ], "tensor_parallel_by_hardware": { "mi300x": 2 } }, { "variant": "rl", "model_id": "XiaomiMiMo/MiMo-V2.6-Flash-RL", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 208, "minimum_runtime_version": "0.31.0", "extra_args": [ "--tensor-parallel-size", "4", "--gpu-memory-utilization", "0.95" ], "tensor_parallel_by_hardware": { "mi300x": 2 } } ] } ``` ## Reading notes - Recipe minimum engine version: 0.31.0. - Default artifact: XiaomiMiMo/MiMo-V2.6-Flash-MOPD. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - The recipe filename alone does not identify its default served weights. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned MiMo-V2.6-Flash-RL recipe defaults to XiaomiMiMo/MiMo-V2.6-Flash-MOPD (fp8). Its effective default-variant minimum is vLLM 0.31.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies XiaomiMiMo/MiMo-V2.6-Flash-RL as its model family entry and selects XiaomiMiMo/MiMo-V2.6-Flash-MOPD by default. The default precision label is fp8; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L63 ## What needs special attention for this model? Despite the recipe title, its default points to MiMo-V2.6-Flash-MOPD. Select the explicit RL option when reproducing the RL card. The default description distinguishes MXFP4-stored experts from FP8 compute; a single precision label cannot describe the entire execution path. The SGLang example in the publisher card uses another topology. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L22 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L113 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L71 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L131 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L30", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Total parameters as counted by recipe", "value": "309B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L31", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Active parameters as counted by recipe", "value": "15B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L32", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Native context declaration (tokens)", "value": 1048576, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L33", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Model-wide minimum vLLM", "value": "0.31.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L24", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Effective default-variant minimum vLLM", "value": "0.31.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L24", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Default artifact", "value": "XiaomiMiMo/MiMo-V2.6-Flash-MOPD", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L63", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Default precision label", "value": "fp8", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L63", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 208, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L71", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L63", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--trust-remote-code", "--max-model-len", "auto", "--generation-config", "vllm" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L63", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "multi_node_tp", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L63", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L63", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" }, { "name": "Features selected by default", "value": [ "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/XiaomiMiMo/MiMo-V2.6-Flash-RL.yaml#L63", "source_document_sha256": "1cd36a2a0c053ed42b916965cb620f12326f16d0a464bf8703191335ca4752f4" } ] ``` --- # NVIDIA Nemotron 3.5 Lightning: vLLM recipe Nemotron Lightning recipe with distinct NVFP4 and BF16 choices and engine constraints. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Nemotron - Model: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: nvidia, spark - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.27.1", "nightly_required": false, "model_context_limit_tokens": 262144, "compatible_strategies": [ "single_node_tp", "single_node_tep" ], "hardware_claims": [ { "profile": "h100", "maintainer_status": "verified" }, { "profile": "dgx_spark_gb10", "maintainer_status": "verified" }, { "profile": "rtx_pro_6000", "maintainer_status": "verified" }, { "profile": "dgx_station_gb300", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-08", "effective_default_minimum_runtime_version": "0.27.1" }, "deployment_options": [ { "variant": "default", "model_id": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 18, "minimum_runtime_version": "0.27.1", "extra_args": [ "--kv-cache-dtype", "fp8", "--moe-backend", "marlin" ], "tensor_parallel_by_hardware": null }, { "variant": "bf16", "model_id": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 72, "minimum_runtime_version": "0.27.1", "extra_args": [ "--moe-backend", "flashinfer_cutlass" ], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.27.1. - Default artifact: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - The default is NVFP4 even though the source filename contains BF16. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 recipe defaults to nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 (nvfp4). Its effective default-variant minimum is vLLM 0.27.1. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 as its model family entry and selects nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 by default. The default precision label is nvfp4; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L69 ## What needs special attention for this model? The title contains BF16, but the default is the NVIDIA NVFP4 conversion with FP8 KV cache. The documented draft models target that NVFP4 path. The Mamba backend and cache mode are part of the hybrid model’s execution configuration; a generic Transformer KV formula is not a verified memory model for it. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L19 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L188 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L117 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L196 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L23", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Total parameters as counted by recipe", "value": "30B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L24", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Active parameters as counted by recipe", "value": "3B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L25", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Native context declaration (tokens)", "value": 262144, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L26", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Model-wide minimum vLLM", "value": "0.27.1", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L21", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Effective default-variant minimum vLLM", "value": "0.27.1", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L21", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Default artifact", "value": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L69", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Default precision label", "value": "nvfp4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L69", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 18, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L117", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L69", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--mamba-backend", "flashinfer", "--mamba-cache-mode", "align", "--enable-prefix-caching", "--max-num-batched-tokens", "16384" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L69", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L69", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Optional feature names", "value": [ "reasoning", "tool_calling", "mamba_fast_ssm_cache", "spec_decoding", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L69", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" }, { "name": "Features selected by default", "value": [ "spec_decoding", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.yaml#L69", "source_document_sha256": "244058cc70b3d0d131dbf8bbd5dcea627fc81c9c487dacf9c64db870a91ea1ff" } ] ``` --- # NVIDIA Nemotron 3.5 Super VL: vLLM recipe Recent multimodal Nemotron recipe with per-artifact hardware restrictions and selected GB200 layouts. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Nemotron - Model: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: nvidia, spark - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.31.0", "nightly_required": false, "model_context_limit_tokens": 1048576, "compatible_strategies": [ "single_node_tp" ], "hardware_claims": [ { "profile": "h100", "maintainer_status": "verified" }, { "profile": "h200", "maintainer_status": "verified" }, { "profile": "gb200", "maintainer_status": "verified" }, { "profile": "dgx_spark_gb10", "maintainer_status": "verified" }, { "profile": "rtx_pro_6000", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-15", "effective_default_minimum_runtime_version": "0.31.0" }, "deployment_options": [ { "variant": "default", "model_id": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4", "precision": "nvfp4", "supported_hardware": [ "h100", "h200", "gb200", "dgx_spark_gb10", "rtx_pro_6000" ], "model_revision": null, "declared_planning_minimum_gb": 91, "minimum_runtime_version": "0.31.0", "extra_args": [], "tensor_parallel_by_hardware": { "h100": 2, "h200": 1, "gb200": 1, "dgx_spark_gb10": 1, "rtx_pro_6000": 1 } }, { "variant": "fp8", "model_id": "RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 155, "minimum_runtime_version": "0.31.0", "extra_args": [ "--gpu-memory-utilization", "0.89", "--prefix-match-unit", "64", "--block-size", "64", "--compilation-config", "{\"max_cudagraph_capture_size\":192}", "--enable-expert-parallel", "--max-num-seqs", "128", "--max-model-len", "114688", "--max-num-batched-tokens", "16384", "--enable-prefix-caching", "--async-scheduling", "--mamba-backend", "flashinfer", "--mamba-ssm-cache-dtype", "float16", "--enable-mamba-cache-stochastic-rounding", "--mamba-cache-philox-rounds", "5", "--chat-template-content-format", "string", "--mamba-cache-mode", "align", "--enable-mamba-fine-grained-prefix-cache" ], "tensor_parallel_by_hardware": { "hopper": 2, "blackwell": 1 } }, { "variant": "bf16", "model_id": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16", "precision": "bf16", "supported_hardware": [ "h100", "gb200" ], "model_revision": null, "declared_planning_minimum_gb": 299, "minimum_runtime_version": "0.31.0", "extra_args": [], "tensor_parallel_by_hardware": { "h100": 4, "gb200": 2 } }, { "variant": "nvfp4_gb200_4x", "model_id": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4", "precision": "nvfp4", "supported_hardware": [ "gb200" ], "model_revision": null, "declared_planning_minimum_gb": 91, "minimum_runtime_version": "0.31.0", "extra_args": [ "--enable-expert-parallel", "--max-num-seqs", "512", "--max-model-len", "28672", "--no-enable-prefix-caching", "--async-scheduling", "--mamba-backend", "flashinfer", "--mamba-ssm-cache-dtype", "float16", "--enable-mamba-cache-stochastic-rounding", "--mamba-cache-philox-rounds", "5", "--chat-template-content-format", "string", "--mamba-cache-mode", "align" ], "tensor_parallel_by_hardware": 4 }, { "variant": "bf16_gb200_4x", "model_id": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16", "precision": "bf16", "supported_hardware": [ "gb200" ], "model_revision": null, "declared_planning_minimum_gb": 299, "minimum_runtime_version": "0.31.0", "extra_args": [ "--enable-expert-parallel", "--max-num-seqs", "512", "--max-model-len", "28672", "--no-enable-prefix-caching", "--async-scheduling", "--mamba-backend", "flashinfer", "--mamba-ssm-cache-dtype", "float32", "--chat-template-content-format", "string", "--mamba-cache-mode", "align" ], "tensor_parallel_by_hardware": 4 } ] } ``` ## Reading notes - Recipe minimum engine version: 0.31.0. - Default artifact: nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Some variants have narrow hardware allowlists; a topic tag is not universal support. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 recipe defaults to nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 (nvfp4). Its effective default-variant minimum is vLLM 0.31.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 as its model family entry and selects nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 by default. The default precision label is nvfp4; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L94 ## What needs special attention for this model? This multimodal NVFP4 recipe declares different tensor-parallel counts for different devices: H100 uses two while the listed H200, GB200, Spark and RTX PRO 6000 profiles use one. That declaration is specific to the pinned variant. The 91 GB planning figure is not a measured peak with images or a 1M context, and local-media options need an explicit input policy. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L21 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L247 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L120 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L254 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L25", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Total parameters as counted by recipe", "value": "120B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L26", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Active parameters as counted by recipe", "value": "12B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L27", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Native context declaration (tokens)", "value": 1048576, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L28", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Model-wide minimum vLLM", "value": "0.31.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L23", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Effective default-variant minimum vLLM", "value": "0.31.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L23", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Default artifact", "value": "nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L94", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Default precision label", "value": "nvfp4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L94", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 91, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L120", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Default variant hardware allowlist", "value": [ "h100", "h200", "gb200", "dgx_spark_gb10", "rtx_pro_6000" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L94", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--trust-remote-code" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L94", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L94", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Optional feature names", "value": [ "spec_decoding", "tool_calling", "reasoning", "local_media" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L94", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" }, { "name": "Features selected by default", "value": [ "local_media" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4.yaml#L94", "source_document_sha256": "6152bc4d92191a6d26dc554401fe54d9f5ba314c5fc457435def47cbe9534dd7" } ] ``` --- # Qwen3.6-35B-A3B: vLLM recipe A smaller MoE deployment recipe with publisher and third-party FP8/NVFP4 alternatives. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen/Qwen3.6-35B-A3B - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, intel, nvidia, spark - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.17.0", "nightly_required": false, "model_context_limit_tokens": 262144, "compatible_strategies": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tp_pp", "multi_node_dep", "multi_node_tep" ], "hardware_claims": [ { "profile": "dgx_spark_gb10", "maintainer_status": "verified" }, { "profile": "h100", "maintainer_status": "verified" }, { "profile": "h200", "maintainer_status": "verified" }, { "profile": "mi300x", "maintainer_status": "verified" }, { "profile": "mi325x", "maintainer_status": "verified" }, { "profile": "mi355x", "maintainer_status": "verified" }, { "profile": "rtx_pro_6000", "maintainer_status": "verified" }, { "profile": "dgx_station_gb300", "maintainer_status": "verified" }, { "profile": "rtx_5090", "maintainer_status": "verified" }, { "profile": "arc_pro_b60", "maintainer_status": "verified" }, { "profile": "arc_pro_b70", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-09-14", "effective_default_minimum_runtime_version": "0.17.0" }, "deployment_options": [ { "variant": "default", "model_id": "Qwen/Qwen3.6-35B-A3B", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 84, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "Qwen/Qwen3.6-35B-A3B-FP8", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 42, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "nvidia/Qwen3.6-35B-A3B-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 21, "minimum_runtime_version": "0.28.0", "extra_args": [], "tensor_parallel_by_hardware": 1 }, { "variant": "redhat_nvfp4", "model_id": "RedHatAI/Qwen3.6-35B-A3B-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 21, "minimum_runtime_version": "0.17.0", "extra_args": [ "--kv-cache-dtype", "fp8" ], "tensor_parallel_by_hardware": null }, { "variant": "redhat_fp8", "model_id": "RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 42, "minimum_runtime_version": "0.17.0", "extra_args": [ "--kv-cache-dtype", "fp8" ], "tensor_parallel_by_hardware": null }, { "variant": "redhat_fp8_plain", "model_id": "RedHatAI/Qwen3.6-35B-A3B-FP8", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 42, "minimum_runtime_version": "0.17.0", "extra_args": [ "--kv-cache-dtype", "fp8" ], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.17.0. - Default artifact: Qwen/Qwen3.6-35B-A3B. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Conversions from different publishers remain distinct artifacts. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned Qwen3.6-35B-A3B recipe defaults to Qwen/Qwen3.6-35B-A3B (bf16). Its effective default-variant minimum is vLLM 0.17.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies Qwen/Qwen3.6-35B-A3B as its model family entry and selects Qwen/Qwen3.6-35B-A3B by default. The default precision label is bf16; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L74 ## What needs special attention for this model? The 35B total / 3B active MoE labels describe parameter activation, not guaranteed quality or memory proportional to 3B. BF16 and converted FP8/NVFP4 artifacts are separate. The recipe includes different AMD and Intel overrides, with distinct text-only, encoder placement and long-context controls. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L28 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L258 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L106 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L278 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L32", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Total parameters as counted by recipe", "value": "35B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L33", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Active parameters as counted by recipe", "value": "3B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L34", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Native context declaration (tokens)", "value": 262144, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L35", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Model-wide minimum vLLM", "value": "0.17.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L30", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Effective default-variant minimum vLLM", "value": "0.17.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L30", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Default artifact", "value": "Qwen/Qwen3.6-35B-A3B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L74", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Default precision label", "value": "bf16", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L74", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 84, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L106", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L74", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--trust-remote-code" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L74", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tp", "multi_node_tp_pp", "multi_node_dep", "multi_node_tep" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L74", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding", "text_only", "encoder_parallel", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L74", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" }, { "name": "Features selected by default", "value": [ "spec_decoding", "text_only", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.6-35B-A3B.yaml#L74", "source_document_sha256": "b09823ad261d3e52081322869ceef0ce3ff5b2500b59cd4666f76d0845b2f71d" } ] ``` --- # Qwen3.5-4B: vLLM recipe Compact Qwen3.5 recipe with explicit Intel CPU/GPU artifact allowlists in the pinned version. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen/Qwen3.5-4B - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: cpu, intel - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.17.0", "nightly_required": false, "model_context_limit_tokens": 262144, "compatible_strategies": [ "single_node_tp" ], "hardware_claims": [ { "profile": "xeon6", "maintainer_status": "verified" }, { "profile": "arc_pro_b60", "maintainer_status": "verified" }, { "profile": "arc_pro_b70", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-09-14", "effective_default_minimum_runtime_version": "0.17.0" }, "deployment_options": [ { "variant": "default", "model_id": "Qwen/Qwen3.5-4B", "precision": "bf16", "supported_hardware": [ "xeon6", "arc_pro_b60", "arc_pro_b70" ], "model_revision": null, "declared_planning_minimum_gb": 10, "minimum_runtime_version": "0.17.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "RedHatAI/Qwen3.5-4B-FP8-dynamic", "precision": "fp8", "supported_hardware": [ "arc_pro_b60", "arc_pro_b70" ], "model_revision": null, "declared_planning_minimum_gb": 5, "minimum_runtime_version": "0.17.0", "extra_args": [ "--kv-cache-dtype", "fp8" ], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.17.0. - Default artifact: Qwen/Qwen3.5-4B. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Do not present generic recipe availability as every-GPU certification. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned Qwen3.5-4B recipe defaults to Qwen/Qwen3.5-4B (bf16). Its effective default-variant minimum is vLLM 0.17.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies Qwen/Qwen3.5-4B as its model family entry and selects Qwen/Qwen3.5-4B by default. The default precision label is bf16; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L69 ## What needs special attention for this model? The pinned default’s hardware allowlist specifically names Xeon 6 and Intel Arc Pro B60/B70. Its 10 GB memory planning figure does not certify another 10 GB card. A small parameter label does not remove model-architecture, remote-code or backend support requirements. Select the exact supported variant and platform. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L21 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L86 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L72 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L96 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "dense", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L25", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Total parameters as counted by recipe", "value": "4B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L26", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Active parameters as counted by recipe", "value": "4B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L27", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Native context declaration (tokens)", "value": 262144, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L28", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Model-wide minimum vLLM", "value": "0.17.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L23", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Effective default-variant minimum vLLM", "value": "0.17.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L23", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Default artifact", "value": "Qwen/Qwen3.5-4B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L69", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Default precision label", "value": "bf16", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L69", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 10, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L72", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Default variant hardware allowlist", "value": [ "xeon6", "arc_pro_b60", "arc_pro_b70" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L69", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--trust-remote-code" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L69", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L69", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding", "text_only", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L69", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" }, { "name": "Features selected by default", "value": [ "spec_decoding", "text_only", "long_context" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen3.5-4B.yaml#L69", "source_document_sha256": "4f6be4f961c07425d4473c83c7b0916a53fc1c483aa8279a7cd19800d1696169" } ] ``` --- # GPT-OSS 20B: vLLM recipe MXFP4 serving recipe with accelerator allowlists and model-specific serving controls. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: GPT-OSS - Model: openai/gpt-oss-20b - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, cpu, intel, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.10.1", "nightly_required": false, "model_context_limit_tokens": 131072, "compatible_strategies": [ "single_node_tp" ], "hardware_claims": [ { "profile": "h100", "maintainer_status": "verified" }, { "profile": "h200", "maintainer_status": "verified" }, { "profile": "b200", "maintainer_status": "verified" }, { "profile": "mi300x", "maintainer_status": "verified" }, { "profile": "mi325x", "maintainer_status": "verified" }, { "profile": "mi355x", "maintainer_status": "verified" }, { "profile": "arc_pro_b60", "maintainer_status": "verified" }, { "profile": "arc_pro_b70", "maintainer_status": "verified" }, { "profile": "xeon6", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-09-12", "effective_default_minimum_runtime_version": "0.10.1" }, "deployment_options": [ { "variant": "default", "model_id": "openai/gpt-oss-20b", "precision": "mxfp4", "supported_hardware": [ "h100", "h200", "b200", "mi300x", "mi325x", "mi355x", "arc_pro_b60", "arc_pro_b70", "xeon6" ], "model_revision": null, "declared_planning_minimum_gb": 16, "minimum_runtime_version": "0.10.1", "extra_args": [], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.10.1. - Default artifact: openai/gpt-oss-20b. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Use the exact engine and artifact instructions; this entry is not a fresh model release. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned gpt-oss-20b recipe defaults to openai/gpt-oss-20b (mxfp4). Its effective default-variant minimum is vLLM 0.10.1. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies openai/gpt-oss-20b as its model family entry and selects openai/gpt-oss-20b by default. The default precision label is mxfp4; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L52 ## What needs special attention for this model? The model name says 20b while the recipe summarizes approximately 21B total and 3.6B active parameters. Its MXFP4 expert representation and explicit accelerator allowlist must stay with the artifact. The declared 16 GB planning minimum does not establish the maximum context or user count of a consumer GPU. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L25 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L59 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L55 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L88 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L28", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Total parameters as counted by recipe", "value": "21B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L29", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Active parameters as counted by recipe", "value": "3.6B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L30", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Native context declaration (tokens)", "value": 131072, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L31", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Model-wide minimum vLLM", "value": "0.10.1", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L27", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Effective default-variant minimum vLLM", "value": "0.10.1", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L27", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Default artifact", "value": "openai/gpt-oss-20b", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L52", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Default precision label", "value": "mxfp4", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L52", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 16, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L55", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Default variant hardware allowlist", "value": [ "h100", "h200", "b200", "mi300x", "mi325x", "mi355x", "arc_pro_b60", "arc_pro_b70", "xeon6" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L52", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Shared base arguments (incomplete launch)", "value": [], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L52", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L52", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Optional feature names", "value": [ "tool_calling", "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L52", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" }, { "name": "Features selected by default", "value": [ "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/openai/gpt-oss-20b.yaml#L52", "source_document_sha256": "0cb6342ac0c6225518d4a62677b75aa952e03cca2ffe44df1b836f284c373a46" } ] ``` --- # Gemma 4 12B IT: vLLM recipe Gemma 4 serving recipe with BF16 and multiple converted precision paths. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Gemma - Model: google/gemma-4-12B-it - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: intel, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.23.0", "nightly_required": false, "model_context_limit_tokens": 131072, "compatible_strategies": [ "single_node_tp", "multi_node_tp" ], "hardware_claims": [ { "profile": "h100", "maintainer_status": "verified" }, { "profile": "arc_pro_b60", "maintainer_status": "verified" }, { "profile": "arc_pro_b70", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-09-09", "effective_default_minimum_runtime_version": "0.23.0" }, "deployment_options": [ { "variant": "default", "model_id": "google/gemma-4-12B-it", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 29, "minimum_runtime_version": "0.23.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "RedHatAI/gemma-4-12B-it-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 8, "minimum_runtime_version": "0.23.0", "extra_args": [ "--kv-cache-dtype", "fp8" ], "tensor_parallel_by_hardware": null }, { "variant": "w4a16", "model_id": "google/gemma-4-12B-it-qat-w4a16-ct", "precision": "int4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 9, "minimum_runtime_version": "0.23.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "RedHatAI/gemma-4-12B-it-FP8-Dynamic", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 15, "minimum_runtime_version": "0.23.0", "extra_args": [ "--kv-cache-dtype", "fp8" ], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.23.0. - Default artifact: google/gemma-4-12B-it. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Model distribution terms are independent of the recipe repository’s license. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned gemma-4-12B-it recipe defaults to google/gemma-4-12B-it (bf16). Its effective default-variant minimum is vLLM 0.23.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies google/gemma-4-12B-it as its model family entry and selects google/gemma-4-12B-it by default. The default precision label is bf16; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L64 ## What needs special attention for this model? The default is the instruction-tuned BF16 artifact. Its 29 GB planning minimum and a single 40 GB-or-larger GPU description are separate from actual workload memory. Alternative conversions change the artifact; Intel overrides are platform-specific. Preserve the model’s access and license conditions independently of the Apache-licensed recipe. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L22 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L92 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L67 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L117 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "dense", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L29", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Total parameters as counted by recipe", "value": "12B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L30", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Active parameters as counted by recipe", "value": "12B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L31", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Native context declaration (tokens)", "value": 131072, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L32", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Model-wide minimum vLLM", "value": "0.23.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L24", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Effective default-variant minimum vLLM", "value": "0.23.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L24", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Default artifact", "value": "google/gemma-4-12B-it", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L64", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Default precision label", "value": "bf16", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L64", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 29, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L67", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L64", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Shared base arguments (incomplete launch)", "value": [], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L64", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "multi_node_tp" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L64", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L64", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" }, { "name": "Features selected by default", "value": [ "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Google/gemma-4-12B-it.yaml#L64", "source_document_sha256": "5b07225e3b3c7eed4930aca685100707c913bcd648685940dca5b927406e6b54" } ] ``` --- # DeepSeek-V4-Pro: vLLM recipe Large DeepSeek Pro recipe with a dated default checkpoint and separate draft/quantized variants. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: deepseek-ai/DeepSeek-V4-Pro - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: amd, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.20.0", "nightly_required": false, "model_context_limit_tokens": 1048576, "compatible_strategies": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "hardware_claims": [ { "profile": "h200", "maintainer_status": "verified" }, { "profile": "b200", "maintainer_status": "verified" }, { "profile": "gb200", "maintainer_status": "verified" }, { "profile": "b300", "maintainer_status": "verified" }, { "profile": "gb300", "maintainer_status": "verified" }, { "profile": "mi300x", "maintainer_status": "unsupported" }, { "profile": "mi325x", "maintainer_status": "unsupported" }, { "profile": "mi355x", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-10-13", "effective_default_minimum_runtime_version": "0.25.0" }, "deployment_options": [ { "variant": "default", "model_id": "deepseek-ai/DeepSeek-V4-Pro-0813", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 960, "minimum_runtime_version": "0.25.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "deepseek-ai/DeepSeek-V4-Pro", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 960, "minimum_runtime_version": "0.20.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "nvfp4", "model_id": "nvidia/DeepSeek-V4-Pro-NVFP4", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 960, "minimum_runtime_version": "0.20.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "dspark", "model_id": "deepseek-ai/DeepSeek-V4-Pro-DSpark", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 960, "minimum_runtime_version": "0.25.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "redhat_nvfp4", "model_id": "RedHatAI/DeepSeek-V4-Pro-NVFP4-FP8", "precision": "nvfp4", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 1000, "minimum_runtime_version": "0.20.0", "extra_args": [], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.20.0. - Default artifact: deepseek-ai/DeepSeek-V4-Pro-0813. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - The default is the 0813 checkpoint; Pro and Flash are different models. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned DeepSeek-V4-Pro recipe defaults to deepseek-ai/DeepSeek-V4-Pro-0813 (fp8). Its effective default-variant minimum is vLLM 0.25.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies deepseek-ai/DeepSeek-V4-Pro as its model family entry and selects deepseek-ai/DeepSeek-V4-Pro-0813 by default. The default precision label is fp8; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L67 ## What needs special attention for this model? The model-wide minimum is 0.20.0, but the default 0813 variant overrides it to 0.25.0. Use the effective variant requirement. The default also enables a DSpark draft mode and names a dated Pro checkpoint, not Flash or 1017. Draft weights, FP8 KV and long-context memory all need separate identities. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L23 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L163 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L97 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L279 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "moe", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L29", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Total parameters as counted by recipe", "value": "1600B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L30", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Active parameters as counted by recipe", "value": "49B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L31", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Native context declaration (tokens)", "value": 1048576, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L32", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Model-wide minimum vLLM", "value": "0.20.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L25", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Effective default-variant minimum vLLM", "value": "0.25.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L25", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Default artifact", "value": "deepseek-ai/DeepSeek-V4-Pro-0813", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L67", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Default precision label", "value": "fp8", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L67", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 960, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L97", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L67", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Shared base arguments (incomplete launch)", "value": [ "--trust-remote-code", "--kv-cache-dtype", "fp8", "--block-size", "256" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L67", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp", "single_node_tep", "single_node_dep", "multi_node_tep", "multi_node_dep", "pd_cluster" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L67", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Optional feature names", "value": [ "tool_calling", "reasoning", "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L67", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" }, { "name": "Features selected by default", "value": [ "spec_decoding" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/deepseek-ai/DeepSeek-V4-Pro.yaml#L67", "source_document_sha256": "c6e21995adfcc7f44259c24d6cbb2f321e6f3df7970da21e86e557aa3d9cadce" } ] ``` --- # Qwen-Image-2.1: vLLM-Omni recipe Image-generation recipe with vLLM-Omni requirements and separate precision choices. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen/Qwen-Image-2.1 - Original author: vllm-project - Original project: https://github.com/vllm-project/recipes - Pinned document: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml - Source revision: d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94 - Source commit time (not experiment date): 2026-10-17T09:22:45Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM-Omni - Hardware topics: nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [], "recipe_parameters": { "minimum_runtime_version": "0.29.0", "nightly_required": false, "model_context_limit_tokens": null, "compatible_strategies": [ "single_node_tp" ], "hardware_claims": [ { "profile": "gb300", "maintainer_status": "verified" } ], "recipe_updated_on": "2026-09-19", "effective_default_minimum_runtime_version": "0.29.0" }, "deployment_options": [ { "variant": "default", "model_id": "Qwen/Qwen-Image-2.1", "precision": "bf16", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 40, "minimum_runtime_version": "0.29.0", "extra_args": [], "tensor_parallel_by_hardware": null }, { "variant": "fp8", "model_id": "Qwen/Qwen-Image-2.1", "precision": "fp8", "supported_hardware": null, "model_revision": null, "declared_planning_minimum_gb": 40, "minimum_runtime_version": "0.29.0", "extra_args": [ "--diffusion-quantization-config", "{\"transformer\":{\"method\":\"fp8\",\"ignored_layers\":[\"img_mlp\"]}}" ], "tensor_parallel_by_hardware": null } ] } ``` ## Reading notes - Recipe minimum engine version: 0.29.0. - Default artifact: Qwen/Qwen-Image-2.1. - Variant and hardware settings are source declarations; select the matching original profile. ## Limits - Image throughput is not comparable with text-token throughput. - Input length, output policy, concurrency and cache are workload-specific; no single benchmark is inferred from the recipe. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Recipe repository only; each model and conversion has independent distribution terms."} ## Editorial answer The pinned Qwen-Image-2.1 recipe defaults to Qwen/Qwen-Image-2.1 (bf16). Its effective default-variant minimum is vLLM 0.29.0. Select the exact variant, hardware override and topology before treating it as a launch plan. No deployment was run by this site. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which artifact and engine does this recipe select? The recipe identifies Qwen/Qwen-Image-2.1 as its model family entry and selects Qwen/Qwen-Image-2.1 by default. The default precision label is bf16; model-wide and variant-specific runtime requirements are kept separately below. The document revision fixes what the recipe said, while the weight revision and executed runtime revision remain separate fields. A model-family title alone cannot establish either artifact identity. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L104 ## What needs special attention for this model? This is image generation through vLLM-Omni, not a text decode benchmark. The YAML’s context-length zero is a not-applicable sentinel, not a zero-token model. The BF16 description reports a 34.0 GB peak on one GB300 at 1024×1024, 40 steps, CFG off and VAE tiling; its 40 GB planning minimum is a separate derived budget. Do not express image performance in language-model tokens per second. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L21 ## How should the settings be applied? Choose one artifact variant and one hardware profile from the original recipe, then resolve its shared, variant, hardware, strategy and optional-feature settings together. The base arguments listed here are only one layer and are deliberately not presented as a complete runnable command. Check any nightly-image requirement and the model’s parsers before loading weights. A context declaration is a model limit, while batched-token and sequence settings control a serving workload. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L118 ## What does the memory number mean? The GB figure is the recipe’s declared planning minimum for the selected variant. It is not a site measurement, a universal peak allocation or a promise of maximum context. KV cache, auxiliary/draft models, image encoders, temporary workspace and active requests still require memory. Keep GB and GiB distinct, and use the actual artifact size and runtime allocation for a machine-specific budget. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L108 ## What is needed to reproduce a deployment? Save the model and conversion revisions, weight hashes, tokenizer/template, container or runtime digest, device count/topology and complete resolved launch. First check one short public request. For a subsequent measurement, fix input and output counts, concurrency, cache state, warmup, sampling, reasoning effort, repetitions and failures. The declarations on this page are source configuration facts; no throughput or quality outcome follows from them. Sources: https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L125 ## Attributed configuration facts ```json [ { "name": "Architecture", "value": "dense", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L40", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Total parameters as counted by recipe", "value": "16B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L41", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Active parameters as counted by recipe", "value": "16B", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L42", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Native context declaration (tokens)", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L43", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Model-wide minimum vLLM", "value": "0.29.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L27", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Effective default-variant minimum vLLM", "value": "0.29.0", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L27", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Default artifact", "value": "Qwen/Qwen-Image-2.1", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L104", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Default precision label", "value": "bf16", "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L104", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Default planning minimum (GB; recipe declaration)", "value": 40, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L108", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Default variant hardware allowlist", "value": null, "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L104", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Shared base arguments (incomplete launch)", "value": [], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L104", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Declared compatible strategies", "value": [ "single_node_tp" ], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L104", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Optional feature names", "value": [], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L104", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" }, { "name": "Features selected by default", "value": [], "classification": "recipe_declaration", "evidence_url": "https://github.com/vllm-project/recipes/blob/d5fc1f55cee70f4e1944f2dd4f7569cb8276ad94/models/Qwen/Qwen-Image-2.1.yaml#L104", "source_document_sha256": "b7423875f68d7111765c3206e0fab9ec9c590b98f9d041c8f956339479fb23cb" } ] ``` --- # llama.cpp: GGUF deployment paths Start with the upstream build and server documentation for a GGUF deployment. Choose the backend for your own platform and keep the model artifact separate from the runtime revision. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: General - Model: None - Original author: ggml-org - Original project: https://github.com/ggml-org/llama.cpp - Pinned document: https://github.com/ggml-org/llama.cpp/tree/f74e2fa6a31cc33ab4ee4ae516641cfe62df3924 - Source revision: f74e2fa6a31cc33ab4ee4ae516641cfe62df3924 - Source commit time (not experiment date): 2026-10-17T02:03:41Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: llama.cpp - Hardware topics: cpu, apple, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - CPU, Metal and CUDA are separate build paths. - A GGUF file still needs an architecture supported by the selected build. - Record the actual model hash, context and offloaded layers when trying a recipe. ## Limits - Model revision and artifact: selected by the reader. - A repository revision is a documentation pin, not a reproduced deployment. ## License declaration {"declared_spdx": "MIT", "scope": "Repository declaration only; models, containers, datasets and per-file terms remain separate."} ## Editorial answer Start with the upstream build and server documentation for a GGUF deployment. Choose the backend for your own platform and keep the model artifact separate from the runtime revision. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What does the upstream project provide? llama.cpp supplies C/C++ inference with multiple backend paths, a CLI and an API server. The pinned README separates package installation, prebuilt binaries, Docker and source builds. A source checkout and a downloaded GGUF are different artifacts; pin both when preparing a reproducible setup. The current README also presents the newer llama cli and llama serve launcher forms. Sources: https://github.com/ggml-org/llama.cpp/blob/f74e2fa6a31cc33ab4ee4ae516641cfe62df3924/README.md#L20 ## How should a backend be selected? Apple silicon uses the native Metal route; NVIDIA CUDA, AMD HIP and other backends have their own build requirements. CPU/GPU hybrid execution can offload part of a model. That support list is a maintainer declaration across the project, not evidence that every model architecture or quantization runs on every backend. Sources: https://github.com/ggml-org/llama.cpp/blob/f74e2fa6a31cc33ab4ee4ae516641cfe62df3924/README.md#L76 ## What should be recorded during a first run? Record the runtime commit, compiler and CMake configuration, model and conversion revision, complete GGUF hash, selected backend, layer placement, context and KV types. Read the startup log to verify actual placement. A built CUDA binary and a successful CPU request do not establish that layers were placed on the GPU. Sources: https://github.com/ggml-org/llama.cpp/blob/f74e2fa6a31cc33ab4ee4ae516641cfe62df3924/README.md#L72 ## How is a deployment check different from a benchmark? A short prompt confirms that the selected artifact and executable can complete a request. A speed test additionally needs fixed input and output lengths, concurrency, cache state, sampling, repeated trials and failure counts. Use the linked installation guide for a minimal check and the community reports for dated measurement examples; the project README itself is not a speed result. Sources: https://github.com/ggml-org/llama.cpp/blob/f74e2fa6a31cc33ab4ee4ae516641cfe62df3924/README.md#L60 --- # Ollama: native apps and a local API A maintainer starting point for native installation and local model serving. Use its linked documentation to check platform support, installed model identity and local/cloud behavior. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: General - Model: None - Original author: ollama - Original project: https://github.com/ollama/ollama - Pinned document: https://github.com/ollama/ollama/tree/d1ed3fd13ac73f60cff00bbead8af24a68c4200d - Source revision: d1ed3fd13ac73f60cff00bbead8af24a68c4200d - Source commit time (not experiment date): 2026-10-17T01:45:46Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: Ollama - Hardware topics: cpu, apple, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Select a platform-specific installation path. - Capture the installed model digest rather than relying on a name alone. - Check the client workflow separately from the local server. ## Limits - Model, digest and actual hardware fit are not established by this reference. ## License declaration {"declared_spdx": "MIT", "scope": "Repository declaration only; models, containers, datasets and per-file terms remain separate."} ## Editorial answer A maintainer starting point for native installation and local model serving. Use its linked documentation to check platform support, installed model identity and local/cloud behavior. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What is the smallest useful local check? Install a recorded platform release, start the intended local app or server, and query its model list on the default loopback API port 11434. An empty model list is a valid response before weights are installed. Choose an already available model and keep its returned digest with the runtime version before sending one small public prompt. Sources: https://github.com/ollama/ollama/blob/d1ed3fd13ac73f60cff00bbead8af24a68c4200d/README.md#L90 ## Why does a model name need more provenance? A tag can move while its friendly name stays the same. Capture the installed model digest, model origin and any imported conversion. The upstream README includes local model execution as well as integrations with agent applications; an integration name does not prove that the entire client workflow stays offline. Sources: https://github.com/ollama/ollama/blob/d1ed3fd13ac73f60cff00bbead8af24a68c4200d/README.md#L78 ## Which installation and client paths are separate? The README provides distinct macOS, Windows, Linux and Docker routes, plus Python and JavaScript libraries. Select the documented path for the host rather than mixing shell commands across operating systems. Retain the previous installer and configuration so a version change can be reversed without deleting the model store. Sources: https://github.com/ollama/ollama/blob/d1ed3fd13ac73f60cff00bbead8af24a68c4200d/README.md#L29 ## How should local-only behavior be verified? A loopback endpoint establishes where that HTTP request goes. Cloud features, a selected cloud model or remote functions in the client are separate boundaries. Use the local-only guide to disable the documented server cloud feature and test the exact already-downloaded workflow offline. This reference has not inspected the reader’s machine or run its models. Sources: https://github.com/ollama/ollama/blob/d1ed3fd13ac73f60cff00bbead8af24a68c4200d/README.md#L70 --- # vLLM: model-serving starting point Follow the serving project and its documentation when planning a compatible model-serving environment. Backend, architecture, package and model support are version-dependent. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: General - Model: None - Original author: vllm-project - Original project: https://github.com/vllm-project/vllm - Pinned document: https://github.com/vllm-project/vllm/tree/ce49c47cbb241b59a32ef110079a5261dbd08e49 - Source revision: ce49c47cbb241b59a32ef110079a5261dbd08e49 - Source commit time (not experiment date): 2026-10-17T02:26:57Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Pin the runtime and review its support matrix. - Keep context, concurrency, cache and output policy with every test. - Use request-level measurements for service latency. ## Limits - The NVIDIA/Linux tag is this entry’s focus, not an exhaustive support matrix. - No Windows-native or arbitrary ARM container compatibility is inferred. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Repository declaration only; models, containers, datasets and per-file terms remain separate."} ## Editorial answer Follow the serving project and its documentation when planning a compatible model-serving environment. Backend, architecture, package and model support are version-dependent. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What does vLLM contribute to a serving stack? The upstream project provides high-throughput model serving, memory-management and scheduling mechanisms, with an OpenAI-compatible server among its entry points. Model architecture, quantization kernel, accelerator backend and package version remain a joint compatibility question. Use a model-specific recipe alongside this project overview. Sources: https://github.com/vllm-project/vllm/blob/ce49c47cbb241b59a32ef110079a5261dbd08e49/README.md ## How should the model and runtime be pinned? Record the installed wheel or container digest and executed source revision, then pin the original model and any converted weights independently. A README revision identifies the documentation reviewed here; it does not identify a runtime launched by this site. The collection’s recipe version minima are compatibility declarations, not reproduction receipts. Sources: https://github.com/vllm-project/vllm/blob/ce49c47cbb241b59a32ef110079a5261dbd08e49/README.md ## Which controls change service performance? Context length, active sequences, batched tokens, KV precision, prefix caching, expert/tensor parallelism and speculative decoding can change latency and memory use. Keep the complete launch and client request with each result. Decode tokens per second, aggregate throughput and end-to-end served rate use different denominators. Sources: https://github.com/vllm-project/vllm/blob/ce49c47cbb241b59a32ef110079a5261dbd08e49/README.md ## What should a first performance report contain? Begin with a load and correctness check, then preserve the prompt corpus, token counts, output stopping policy, users, cache state, warmup, repeats, failures and latency distribution. Record device count and topology rather than summing nominal memory into a presumed shared pool. This page does not launch a server or claim measurements of its own. Sources: https://github.com/vllm-project/vllm/blob/ce49c47cbb241b59a32ef110079a5261dbd08e49/README.md --- # SGLang: serving and deployment references An upstream reference for model-serving deployments. Begin with the project documentation, then verify the exact backend, model and container combination you intend to use. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: General - Model: None - Original author: sgl-project - Original project: https://github.com/sgl-project/sglang - Pinned document: https://github.com/sgl-project/sglang/tree/2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e - Source revision: 2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e - Source commit time (not experiment date): 2026-10-17T02:12:41Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: SGLang - Hardware topics: nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Use a version-specific installation guide. - Model architecture and speculative decoding support need their own checks. - Save the complete serving configuration with a performance report. ## Limits - This source tag is a discovery aid; it does not certify a deployment on your device. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Repository declaration only; models, containers, datasets and per-file terms remain separate."} ## Editorial answer An upstream reference for model-serving deployments. Begin with the project documentation, then verify the exact backend, model and container combination you intend to use. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Where does SGLang fit? SGLang is a model-serving framework with its own scheduler, kernel and deployment integrations. Its upstream overview is a starting point; a model-specific cookbook supplies the detailed launch route. A matching model name is insufficient when the backend, quantization or container architecture differs. Sources: https://github.com/sgl-project/sglang/blob/2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e/README.md ## Why keep cache and speculative decoding explicit? A reused prefix can replace measured prefill with a lookup. A draft model consumes memory and may have a model-specific acceptance pattern. Record whether caching is on, which prompts warm it, and the exact draft artifact and decoding policy. The two-Spark report in this collection shows why these controls materially change reported served rate. Sources: https://github.com/sgl-project/sglang/blob/2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e/README.md ## What changes for distributed serving? Capture the tensor/data/expert parallelism, interconnect, NCCL transport, per-node image and kernel cache state. A fast standalone link test does not prove that a serving container inherited the same network configuration. The DGXArley transport reference and two-Spark audit describe those distinctions without certifying another cluster. Sources: https://github.com/sgl-project/sglang/blob/2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e/README.md ## How should a reproducible trial be scoped? First establish architecture and artifact support with a small request. Then record a fixed workload, input/output lengths, concurrency, sampling, warmup, cache, timing boundaries and failures. Keep failed configurations with their conditions. A later successful setting is a new observation rather than proof that the original attempt was valid. Sources: https://github.com/sgl-project/sglang/blob/2d3dd3168d4a8e5b69ebd4c9b41a28fa93e6c16e/README.md --- # MiniMax M3 on three DGX Sparks An original operator repository describing a three-node TP=3 deployment and its benchmark checkpoint. It also retains a four-node baseline, so the two sections must be read as different conditions. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: MiniMax - Model: MiniMax M3 NVFP4 + DSpark - Original author: ChrisTheCoolHut - Original project: https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark - Pinned document: https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/tree/1008cf57355f49d5412f45880289516f29761f0b - Source revision: 1008cf57355f49d5412f45880289516f29761f0b - Source commit time (not experiment date): 2026-08-05T23:05:16Z - Checked: 2026-10-17 - Experiment date: 2026-08-05 - Runtime topics: vLLM - Hardware topics: spark - Reported setup: {"system": "NVIDIA DGX Spark / GB10", "node_count": 3, "parallelism": "custom native vLLM TP=3", "kv_dtype": "FP8", "context_limit_tokens": 262144, "experiment_date": "2026-08-05"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The three-node path uses custom virtual-TP sharding patches. - The repository distinguishes target weights, a DSpark drafter and FP8 KV cache. - Read the dated three-node checkpoint and the actual recipe file together. ## Limits - Three-node and four-node sections must not be merged. - The checkpoint compares max_num_seqs=3 and 8; choose the matching experiment rather than treating them as one configuration. - A declared container digest alone does not prove the TP=3 build matches the documented patch set. ## License declaration {"declared_spdx": "MIT", "scope": "Repository declaration only; models, containers, datasets and per-file terms remain separate."} ## Editorial answer The operator pads attention-head layouts to make native tensor parallelism three possible, using a custom target and draft loader. The run uses three GB10 nodes linked in a switchless RoCE triangle, FP8 KV cache and a 262,144-token context ceiling. This is a patched serving stack rather than an ordinary unmodified vLLM launch. The report’s 80.41 GiB per-rank weights and 122-second load are source observations, not a capacity guarantee for another build. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## How does the three-node path work? The operator pads attention-head layouts to make native tensor parallelism three possible, using a custom target and draft loader. The run uses three GB10 nodes linked in a switchless RoCE triangle, FP8 KV cache and a 262,144-token context ceiling. This is a patched serving stack rather than an ordinary unmodified vLLM launch. The report’s 80.41 GiB per-rank weights and 122-second load are source observations, not a capacity guarantee for another build. Sources: https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L9 ## What changes when the sequence cap is raised? The greedy 80-item GSM8K sweep at eight parallel requests reports 64.63 aggregate tokens/s with max_num_seqs three and 97.28 with eight, while both report 98.8% accuracy. That comparison changes admission behavior. It should not be mixed with the short 128-token llama-benchy runs or interpreted as a 97-token/s single-user decode rate. Sources: https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L102 ## Why keep the sampled run separate? At the publisher’s temperature 1.0 and top_p 0.95, the parallel-eight quality run reports 97.5% accuracy and 86.46 aggregate tokens/s. Greedy and sampled runs answer different questions. The 80-item corpus is a small task-specific check; even a perfect result on that subset does not prove general equivalence of a patched sharding implementation. Sources: https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L131 ## What is still unknown? The record does not establish a full original weight hash or executed runtime commit. The document also compares older four-node data, which belongs to a separate configuration. Preserve the exact custom patch set, container, topology, cache policy and workload before attempting a reproduction. This site has not run the stack. Sources: https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L80 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Short generation, depth 0, C1", "metric": "aggregate_output", "value": 23.55, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "NVIDIA DGX Spark / GB10", "node_count": 3, "parallelism": "custom native vLLM TP=3", "kv_dtype": "FP8", "context_limit_tokens": 262144, "experiment_date": "2026-08-05", "model": "nvidia/MiniMax-M3-NVFP4", "draft": "nvidia/MiniMax-M3-DSpark", "speculative_tokens": 8, "runtime": "custom vLLM TP3", "prefix_cache": true, "harness": "llama-benchy 0.4.0", "repetitions": 3, "max_num_seqs": 3, "output_tokens": 128, "retained_depth_tokens": 0, "concurrency": 1 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L27", "source_document_sha256": "fd06b6a7b2d74ead35c184ae7b31a99b7e43de3516a1dc5d56c1c218853682fc" }, { "label": "Generation, depth 100000, C1", "metric": "aggregate_output", "value": 26.83, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "NVIDIA DGX Spark / GB10", "node_count": 3, "parallelism": "custom native vLLM TP=3", "kv_dtype": "FP8", "context_limit_tokens": 262144, "experiment_date": "2026-08-05", "model": "nvidia/MiniMax-M3-NVFP4", "draft": "nvidia/MiniMax-M3-DSpark", "speculative_tokens": 8, "runtime": "custom vLLM TP3", "prefix_cache": true, "harness": "llama-benchy 0.4.0", "repetitions": 3, "max_num_seqs": 3, "output_tokens": 128, "retained_depth_tokens": 100000, "concurrency": 1 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L50", "source_document_sha256": "fd06b6a7b2d74ead35c184ae7b31a99b7e43de3516a1dc5d56c1c218853682fc" }, { "label": "GSM8K greedy, parallel 8, sequence cap 3", "metric": "aggregate_output", "value": 64.63, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "NVIDIA DGX Spark / GB10", "node_count": 3, "parallelism": "custom native vLLM TP=3", "kv_dtype": "FP8", "context_limit_tokens": 262144, "experiment_date": "2026-08-05", "model": "nvidia/MiniMax-M3-NVFP4", "draft": "nvidia/MiniMax-M3-DSpark", "speculative_tokens": 8, "runtime": "custom vLLM TP3", "prefix_cache": true, "harness": "llama-benchy 0.4.0", "repetitions": null, "max_num_seqs": 3, "corpus_items": 80, "temperature": 0, "concurrency": 8 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L117", "source_document_sha256": "fd06b6a7b2d74ead35c184ae7b31a99b7e43de3516a1dc5d56c1c218853682fc" }, { "label": "GSM8K greedy, parallel 8, sequence cap 8", "metric": "aggregate_output", "value": 97.28, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "NVIDIA DGX Spark / GB10", "node_count": 3, "parallelism": "custom native vLLM TP=3", "kv_dtype": "FP8", "context_limit_tokens": 262144, "experiment_date": "2026-08-05", "model": "nvidia/MiniMax-M3-NVFP4", "draft": "nvidia/MiniMax-M3-DSpark", "speculative_tokens": 8, "runtime": "custom vLLM TP3", "prefix_cache": true, "harness": "llama-benchy 0.4.0", "repetitions": null, "max_num_seqs": 8, "corpus_items": 80, "temperature": 0, "concurrency": 8 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L125", "source_document_sha256": "fd06b6a7b2d74ead35c184ae7b31a99b7e43de3516a1dc5d56c1c218853682fc" }, { "label": "GSM8K sampled, parallel 8", "metric": "accuracy", "value": 97.5, "unit": "percent", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "NVIDIA DGX Spark / GB10", "node_count": 3, "parallelism": "custom native vLLM TP=3", "kv_dtype": "FP8", "context_limit_tokens": 262144, "experiment_date": "2026-08-05", "model": "nvidia/MiniMax-M3-NVFP4", "draft": "nvidia/MiniMax-M3-DSpark", "speculative_tokens": 8, "runtime": "custom vLLM TP3", "prefix_cache": true, "harness": "llama-benchy 0.4.0", "repetitions": null, "max_num_seqs": 8, "corpus_items": 80, "temperature": 1, "top_p": 0.95, "concurrency": 8 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ChrisTheCoolHut/MiniMax-M3-NVFP4-DSpark-vLLM-3x-DGX-Spark/blob/1008cf57355f49d5412f45880289516f29761f0b/benchmarks/2026-08-05-tp3-dspark.md#L140", "source_document_sha256": "fd06b6a7b2d74ead35c184ae7b31a99b7e43de3516a1dc5d56c1c218853682fc" } ] ``` --- # DGXArley: cluster deployment and transport reports An operator-maintained K3s/Ansible repository for a four-Spark cluster. It contains deployment notes and reported transport/inference comparisons. Select the exact model profile and experiment before reusing a number. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Multiple - Model: Multiple model profiles; see the selected report - Original author: vroomfondel - Original project: https://github.com/vroomfondel/dgxarley - Pinned document: https://github.com/vroomfondel/dgxarley/tree/5f1f3762797eac0ea1ad43e403fa979e3e2c3934 - Source revision: 5f1f3762797eac0ea1ad43e403fa979e3e2c3934 - Source commit time (not experiment date): 2026-10-16T07:08:17Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: SGLang - Hardware topics: spark - Reported setup: {"system": "NVIDIA DGX Spark / K3s cluster", "node_count": 4, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null }, "test_profiles": [], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The repository contains multiple model profiles and parallelism choices. - Transport, topology and concurrency belong to each reported experiment. - Per-file licensing may differ from the repository’s main license declaration. ## Limits - A default deployment profile is not necessarily the profile used in a benchmark paragraph. - Prompt lengths, output policy, cache and model revision need verification in the underlying report. - Do not compare these results directly with the three-node MiniMax configuration. ## License declaration {"declared_spdx": "LGPL-3.0", "scope": "Repository declaration only; models, containers, datasets and per-file terms remain separate."} ## Editorial answer This document describes NCCL all-reduce profiling for a Spark serving cluster. It recommends inspecting small messages as well as the large-message bandwidth plateau. The author discusses roughly 185–190 versus 92–95 Gbit/s under different port-spanning conditions, but those are transport observations without an identified workload row here. They are not language-model decode speeds. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What is the report measuring? This document describes NCCL all-reduce profiling for a Spark serving cluster. It recommends inspecting small messages as well as the large-message bandwidth plateau. The author discusses roughly 185–190 versus 92–95 Gbit/s under different port-spanning conditions, but those are transport observations without an identified workload row here. They are not language-model decode speeds. Sources: https://github.com/vroomfondel/dgxarley/blob/5f1f3762797eac0ea1ad43e403fa979e3e2c3934/NCCL_BANDWIDTH_PROFILING.md#L12 ## Why is a healthy headline link rate insufficient? A standalone benchmark can inherit different interface, channel or RoCE settings from the serving container. Large messages can also hide the latency-bound region that tensor-parallel decode uses. Record the actual serving path and inspect its small-message behavior before attributing an inference slowdown to model kernels. Sources: https://github.com/vroomfondel/dgxarley/blob/5f1f3762797eac0ea1ad43e403fa979e3e2c3934/NCCL_BANDWIDTH_PROFILING.md#L25 ## How is the sweep defined? The tool’s default range starts at 1 MB and ends at 1 GB, with a factor-two size step, five warmup and twenty timed iterations per size. The document asks operators to lower the floor to 8 KB for decode-oriented profiling. The ratio below 0.4 is an author-defined diagnostic flag, not an industry pass/fail threshold or a prediction of tokens per second. Sources: https://github.com/vroomfondel/dgxarley/blob/5f1f3762797eac0ea1ad43e403fa979e3e2c3934/NCCL_BANDWIDTH_PROFILING.md#L49 ## What belongs in a usable transport receipt? Keep node count, per-node network device, MTU path, transport, process/container network settings, message size and both algorithm and bus bandwidth. Match those settings to the inference process before linking network results to generation performance. Maintenance and live-workload conflicts require the operator’s own planning; this site does not inspect or alter any cluster. Sources: https://github.com/vroomfondel/dgxarley/blob/5f1f3762797eac0ea1ad43e403fa979e3e2c3934/NCCL_BANDWIDTH_PROFILING.md#L130 ## Attributed configuration facts ```json [ { "name": "Decode-oriented message region", "value": "8 KB–1 MB", "classification": "publisher_declaration", "evidence_url": "https://github.com/vroomfondel/dgxarley/blob/5f1f3762797eac0ea1ad43e403fa979e3e2c3934/NCCL_BANDWIDTH_PROFILING.md#L33", "source_document_sha256": "293d119b6374d8073f040fe6dd277fe42f50f5f9baaadfdfa0d2120e65f12568" }, { "name": "Default warmup / timed iterations", "value": "5 / 20", "classification": "publisher_declaration", "evidence_url": "https://github.com/vroomfondel/dgxarley/blob/5f1f3762797eac0ea1ad43e403fa979e3e2c3934/NCCL_BANDWIDTH_PROFILING.md#L52", "source_document_sha256": "293d119b6374d8073f040fe6dd277fe42f50f5f9baaadfdfa0d2120e65f12568" }, { "name": "Author’s small-region warning threshold", "value": "ratio below 0.4× overall peak", "classification": "publisher_declaration", "evidence_url": "https://github.com/vroomfondel/dgxarley/blob/5f1f3762797eac0ea1ad43e403fa979e3e2c3934/NCCL_BANDWIDTH_PROFILING.md#L59", "source_document_sha256": "293d119b6374d8073f040fe6dd277fe42f50f5f9baaadfdfa0d2120e65f12568" } ] ``` --- # DeepSeek V4.1: eight-5090 placement campaign Paired serving-optimization report on eight RTX 5090 D cards. Interactive, wide-batch and held-out text sections have different workloads. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: DeepSeek-V4.1-Flash (base checkpoint) - Original author: devin-lai - Original project: https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel - Pinned document: https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md - Source revision: 4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8 - Source commit time (not experiment date): 2026-09-16T07:56:48Z - Checked: 2026-10-17 - Experiment date: 2026-09-16 - Runtime topics: vLLM - Hardware topics: nvidia - Reported setup: {"system": "8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only", "node_count": 1, "parallelism": "TP8 + expert parallelism", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-16"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": "MXFP4 experts / FP8 dense / BF16 expert activations", "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Paired interactive placement", "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": [ 1, 8 ], "cache_state": null, "quantization": "MXFP4 experts / FP8 dense / BF16 expert activations", "speculative_decoding": null, "temperature": null, "top_p": null, "hardware": "8 x RTX 5090 D; 503 GiB RAM", "repetitions": 3 } }, { "label": "Wide-batch synthetic workload", "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": 1024, "output_tokens": 128, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null, "repetitions": 1 } }, { "label": "Held-out text workload", "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": 256, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null, "repetitions": 1 } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The comparison baseline is already patched vLLM, not stock upstream. - MXFP4 experts, FP8 dense weights and BF16 expert activations are retained. ## Limits - Interactive three-trial comparisons and single-trial wide batches have different evidence strength. - These are base-checkpoint results, not 1017 results. ## License declaration {"declared_spdx": "MIT", "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."} ## Editorial answer The paired experiment runs on eight 32 GB RTX 5090 D GPUs, two Xeon Gold 6530 sockets and 503 GiB host memory over PCIe. Both arms already use custom vLLM patches and CUDA graphs. The placed arm moves the same offload budget onto rarely selected expert pages and enables exact staging; it is not a stock-versus-optimized upstream comparison. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What is being compared? The paired experiment runs on eight 32 GB RTX 5090 D GPUs, two Xeon Gold 6530 sockets and 503 GiB host memory over PCIe. Both arms already use custom vLLM patches and CUDA graphs. The placed arm moves the same offload budget onto rarely selected expert pages and enables exact staging; it is not a stock-versus-optimized upstream comparison. Sources: https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L20 ## What improves, and what regresses? Across three trials, interactive C1 output rises from 75.37 to 97.23 tokens/s, while C8 rises from 166.55 to 247.77. The same placement reduces the random-token C1 result from 67.46 to 63.00. The negative result is central: calibration-dependent expert placement is useful when traffic resembles its routing profile and can be harmful when it does not. Sources: https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L42 ## Why is the largest number not interactive speed? The 368.55-token/s held-out-text result is aggregate output at 128 concurrent requests, with 1,000-token inputs and 256-token outputs in a single trial. The corresponding latency and queueing behavior differ from C1. The synthetic 337.45-token/s result at 256 users is another workload and another memory budget. Do not collapse them into one headline for a single person. Sources: https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L147 ## How strong is the evidence? The paired interactive comparison has three trials, while placement-only screens and wide-batch campaigns have one. The corpus and prompt shape differ across sections. The report measures the launch checkpoint, not the 1017 refresh, and the collection does not fill in missing prompt lengths or weight hashes from a different source. Sources: https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L28 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Interactive C1, patched baseline", "metric": "output_throughput", "value": 75.37, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only", "node_count": 1, "parallelism": "TP8 + expert parallelism", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-16", "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "quantization": "MXFP4 experts / FP8 dense / BF16 expert activations", "runtime": "already patched graph-enabled vLLM", "calibration": "shipped mixed real-text profile", "concurrency": 1, "repetitions": 3, "placement": "layer offload" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L35", "source_document_sha256": "b2d519cd355ea9c77692eab01e960ca98234e38fcf552589dacd5edb21475078" }, { "label": "Interactive C1, expert placement + staging", "metric": "output_throughput", "value": 97.23, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only", "node_count": 1, "parallelism": "TP8 + expert parallelism", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-16", "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "quantization": "MXFP4 experts / FP8 dense / BF16 expert activations", "runtime": "already patched graph-enabled vLLM", "calibration": "shipped mixed real-text profile", "concurrency": 1, "repetitions": 3, "placement": "cold expert pages + exact staging" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L35", "source_document_sha256": "b2d519cd355ea9c77692eab01e960ca98234e38fcf552589dacd5edb21475078" }, { "label": "Interactive C8, expert placement + staging", "metric": "output_throughput", "value": 247.77, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only", "node_count": 1, "parallelism": "TP8 + expert parallelism", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-16", "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "quantization": "MXFP4 experts / FP8 dense / BF16 expert activations", "runtime": "already patched graph-enabled vLLM", "calibration": "shipped mixed real-text profile", "concurrency": 8, "repetitions": 3, "placement": "cold expert pages + exact staging" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L36", "source_document_sha256": "b2d519cd355ea9c77692eab01e960ca98234e38fcf552589dacd5edb21475078" }, { "label": "Random-token C1, baseline", "metric": "output_throughput", "value": 67.46, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only", "node_count": 1, "parallelism": "TP8 + expert parallelism", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-16", "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "quantization": "MXFP4 experts / FP8 dense / BF16 expert activations", "runtime": "already patched graph-enabled vLLM", "calibration": "shipped mixed real-text profile", "concurrency": 1, "repetitions": 3, "workload": "uniform random token IDs" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L42", "source_document_sha256": "b2d519cd355ea9c77692eab01e960ca98234e38fcf552589dacd5edb21475078" }, { "label": "Random-token C1, placed", "metric": "output_throughput", "value": 63.0, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only", "node_count": 1, "parallelism": "TP8 + expert parallelism", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-16", "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "quantization": "MXFP4 experts / FP8 dense / BF16 expert activations", "runtime": "already patched graph-enabled vLLM", "calibration": "shipped mixed real-text profile", "concurrency": 1, "repetitions": 3, "workload": "uniform random token IDs", "placement": "cold expert pages + staging" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L42", "source_document_sha256": "b2d519cd355ea9c77692eab01e960ca98234e38fcf552589dacd5edb21475078" }, { "label": "Held-out text C128, 128-sequence preset", "metric": "aggregate_output", "value": 368.55, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only", "node_count": 1, "parallelism": "TP8 + expert parallelism", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-16", "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "quantization": "MXFP4 experts / FP8 dense / BF16 expert activations", "runtime": "already patched graph-enabled vLLM", "calibration": "shipped mixed real-text profile", "concurrency": 128, "input_tokens": 1000, "output_tokens": 256, "repetitions": 1, "max_num_seqs": 128 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel/blob/4a4e88ec2cdcf2fc7c802e1f6b7c943d897770c8/benchmarks/results/2026-09-16-expert-placement-and-wide-batches.md#L158", "source_document_sha256": "b2d519cd355ea9c77692eab01e960ca98234e38fcf552589dacd5edb21475078" } ] ``` --- # DeepSeek V4.1: four-Spark boot-10 baseline A clearly identified TP4 baseline using disk-backed Engram and DSpark. Read the boot-10 section separately from later EXL3 and SGLang lanes. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: DeepSeek-V4.1-Flash release checkpoint - Original author: tonyd2wild - Original project: https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark - Pinned document: https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md - Source revision: d45538f67366c62da668c35ec1afd4fcd0e4c8b4 - Source commit time (not experiment date): 2026-09-19T13:16:08Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "4 x NVIDIA DGX Spark", "node_count": 4, "parallelism": "TP4", "kv_dtype": null, "context_limit_tokens": 300000, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Boot 10 streaming sweep", "evidence_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#benchmark-boot-10-the-serving-config", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": "approximately 30–120", "output_tokens": "budget 150–256; natural stopping", "concurrency": null, "cache_state": "after a warmup; see per-row prefill policy", "quantization": "MXFP4 experts / MXFP8 dense", "speculative_decoding": "DSpark k=5", "temperature": 0, "top_p": null, "thinking": false } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The selected baseline uses four Sparks, TP4 and DSpark k=5. - Its short-prompt streaming sweep retains thinking and sampling controls. ## Limits - Later EXL3, TP3 and 1M-context boots are separate experiments. - No 1017 checkpoint performance is established. ## License declaration {"declared_spdx": "MIT", "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."} ## Editorial answer Boot 10 uses four Sparks at TP4 with node-local disk-backed Engram rows, DSpark k5, tools and vision enabled, 300K configured context and GPU memory utilization 0.80. Later EXL3, TP3 and longer-context boots in the same repository are separate experiments. This page keeps the boot-10 measurements anchored to their own section. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which boot does this article cover? Boot 10 uses four Sparks at TP4 with node-local disk-backed Engram rows, DSpark k5, tools and vision enabled, 300K configured context and GPU memory utilization 0.80. Later EXL3, TP3 and longer-context boots in the same repository are separate experiments. This page keeps the boot-10 measurements anchored to their own section. Sources: https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L113 ## Why are aggregate and decode rates different? The author counts tokens from the server usage field rather than stream chunks. Aggregate output divides all stream tokens by batch wall time, including the wait for the first token. Per-stream decode counts tokens after the first over time after that first token. At C1 the category means are consequently 37.95 aggregate and 43.12 decode tokens/s, with 0.441-second mean TTFT. Sources: https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L121 ## How repeatable is the published sweep? The fixed category prompt set is warmed, sampled at temperature zero with thinking off, and released in concurrent batches. Each cell has one batch. The report warns of a GB10 slow-clock state that can move an individual cell substantially. Its counting ceiling is excluded from the eight-category mean, and should stay excluded when quoting that mean. Sources: https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L124 ## What can the data be used for? Use the report to understand this TP4 configuration and its content-dependent latency. It does not establish 1017 performance, a broad workload average or a controlled comparison with later engine lanes. The prefill table uses cold unique prompts and one-token replies, a different profile from the short streaming sweep. Sources: https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L179 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Eight-category mean, C1", "metric": "aggregate_output", "value": 37.95, "unit": "tokens/s", "statistic": "mean across eight categories; counting excluded", "uncertainty": null, "conditions": { "system": "4 x NVIDIA DGX Spark", "node_count": 4, "parallelism": "TP4", "kv_dtype": null, "context_limit_tokens": 300000, "experiment_date": null, "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "speculative_decoding": "DSpark k5", "temperature": 0, "thinking": false, "warmup": "performed", "repetitions": "one batch per cell", "input_tokens": "about 30–120; summary prompt longer", "output_token_budget": "150–256; natural stopping", "weight_precision": "MXFP4 experts / MXFP8 dense", "concurrency": 1 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L130", "source_document_sha256": "8c72dbb42a49a0f9b5b5f8a70055a467823993012bdb47cad748b5b0e04e36a7" }, { "label": "Eight-category mean, C1", "metric": "per_stream_decode", "value": 43.12, "unit": "tokens/s", "statistic": "mean across eight categories; counting excluded", "uncertainty": null, "conditions": { "system": "4 x NVIDIA DGX Spark", "node_count": 4, "parallelism": "TP4", "kv_dtype": null, "context_limit_tokens": 300000, "experiment_date": null, "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "speculative_decoding": "DSpark k5", "temperature": 0, "thinking": false, "warmup": "performed", "repetitions": "one batch per cell", "input_tokens": "about 30–120; summary prompt longer", "output_token_budget": "150–256; natural stopping", "weight_precision": "MXFP4 experts / MXFP8 dense", "concurrency": 1, "denominator": "after first token" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L130", "source_document_sha256": "8c72dbb42a49a0f9b5b5f8a70055a467823993012bdb47cad748b5b0e04e36a7" }, { "label": "Eight-category mean, C1", "metric": "ttft", "value": 0.441, "unit": "seconds", "statistic": "mean across eight categories", "uncertainty": null, "conditions": { "system": "4 x NVIDIA DGX Spark", "node_count": 4, "parallelism": "TP4", "kv_dtype": null, "context_limit_tokens": 300000, "experiment_date": null, "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "speculative_decoding": "DSpark k5", "temperature": 0, "thinking": false, "warmup": "performed", "repetitions": "one batch per cell", "input_tokens": "about 30–120; summary prompt longer", "output_token_budget": "150–256; natural stopping", "weight_precision": "MXFP4 experts / MXFP8 dense", "concurrency": 1 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L130", "source_document_sha256": "8c72dbb42a49a0f9b5b5f8a70055a467823993012bdb47cad748b5b0e04e36a7" }, { "label": "Eight-category mean, C6", "metric": "aggregate_output", "value": 131.86, "unit": "tokens/s", "statistic": "mean across eight categories; counting excluded", "uncertainty": null, "conditions": { "system": "4 x NVIDIA DGX Spark", "node_count": 4, "parallelism": "TP4", "kv_dtype": null, "context_limit_tokens": 300000, "experiment_date": null, "model": "DeepSeek V4.1 Flash launch checkpoint, not 1017", "speculative_decoding": "DSpark k5", "temperature": 0, "thinking": false, "warmup": "performed", "repetitions": "one batch per cell", "input_tokens": "about 30–120; summary prompt longer", "output_token_budget": "150–256; natural stopping", "weight_precision": "MXFP4 experts / MXFP8 dense", "concurrency": 6 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tonyd2wild/DeepSeek-V4.1-Flash-vLLM-DGX-Spark/blob/d45538f67366c62da668c35ec1afd4fcd0e4c8b4/README.md#L135", "source_document_sha256": "8c72dbb42a49a0f9b5b5f8a70055a467823993012bdb47cad748b5b0e04e36a7" } ] ``` --- # DeepSeek V4.1: hybrid RTX/Spark v14 decode Versioned decode campaign comparing one- and two-RTX coordinators with four Spark expert workers, using interleaved sessions. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: DeepSeek-V4.1-Flash @ dba1be0 - Original author: tpurtell - Original project: https://github.com/tpurtell/ds41rt - Pinned document: https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md - Source revision: 3067d0684a0b572d88a17d38ca3f7d10599619b4 - Source commit time (not experiment date): 2026-09-25T11:32:19Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: DS41RT - Hardware topics: spark, nvidia - Reported setup: {"system": "1 or 2 RTX PRO 6000 Blackwell coordinators plus 4 DGX Spark workers", "node_count": null, "parallelism": "4 Spark expert workers, TP4", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "dba1be0a40aa45a94ad051997016db3960a90277", "weight_hash": null, "runtime_execution_revision": "af9d96923f70af1cd385960e28450e5ec0653ff8", "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Interleaved v13/v14 decode sessions", "evidence_url": "https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md", "conditions": { "model_revision": "dba1be0a40aa45a94ad051997016db3960a90277", "weight_hash": null, "runtime_execution_revision": "af9d96923f70af1cd385960e28450e5ec0653ff8", "input_tokens": null, "output_tokens": null, "concurrency": "C1–C16 by workload", "cache_state": "discarded warmup battery and sweep; each arm relaunched", "quantization": null, "speculative_decoding": "target-only and local DSpark arms are separate", "temperature": "greedy or 0.7 in separate cases", "top_p": "0.9 for sampled cases", "hardware": "1 or 2 RTX PRO 6000 at 400 W each, standard clocks; 4 DGX Spark", "repetitions": "8 sessions per layout; workload-specific batteries" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Each arm is restarted from its own image pair. - Greedy, sampled, concurrent and target-only cases are distinguished. ## Limits - The v14 decode campaign does not refresh historical v6 startup or prefill tables. - A capped RTX PRO 6000 experiment does not demonstrate physical RTX 5090 behavior. ## License declaration {"declared_spdx": "MIT", "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."} ## Editorial answer Four Spark expert workers are paired with one or two RTX PRO 6000 Blackwell coordinators, each limited to 400 W at standard memory clocks. The report identifies the base checkpoint, runtime source and image IDs. A 400 W RTX PRO 6000 is still that card; the experiment does not establish the behavior of a physical RTX 5090. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What is this hybrid system? Four Spark expert workers are paired with one or two RTX PRO 6000 Blackwell coordinators, each limited to 400 W at standard memory clocks. The report identifies the base checkpoint, runtime source and image IDs. A 400 W RTX PRO 6000 is still that card; the experiment does not establish the behavior of a physical RTX 5090. Sources: https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md#L6 ## How was v14 compared with v13? Each layout runs eight sessions in an interleaved version order, relaunching the appropriate image pair and discarding warmup. Sessions are the independent units. The published ratios use geometric means, with Welch 95% intervals on session log-ratios. The two-coordinator sampled C1 ratio of 1.377 is therefore a within-campaign version comparison, not a ratio against an unrelated desktop. Sources: https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md#L25 ## Why retain content and traffic type? The release table’s weighted C1 medians are 104.69 tokens/s with one coordinator and 129.67 with two. Counting and code have different rates, and the concurrency sweep uses an aggregate interval from earliest output to final completion. That denominator differs from an individual request’s decode or end-to-end served rate. Sources: https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md#L124 ## What are the limitations and disclosed artifacts? Some greedy outputs differ across arms through batch-shape numerics, although the author reports all objective checks passed. A schema-JSON probe initially hit the prompt cache and was rerun with a fresh nonce; the table uses that rerun. Historical startup and prefill tables belong to older campaigns. Keep the fresh decode evidence separate from those retained history sections. Sources: https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md#L79 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "One RTX coordinator + four Sparks, weighted DSpark", "metric": "decode", "value": 104.69, "unit": "tokens/s", "statistic": "median of three repeats", "uncertainty": null, "conditions": { "system": "1 or 2 RTX PRO 6000 Blackwell coordinators plus 4 DGX Spark workers", "node_count": null, "parallelism": "4 Spark expert workers, TP4", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-24", "model_revision": "dba1be0a40aa45a94ad051997016db3960a90277", "runtime_execution_revision": "af9d96923f70af1cd385960e28450e5ec0653ff8", "worker_count": 4, "worker_parallelism": "Spark TP4", "gpu_power_watts_each": 400, "repetitions": 3, "statistic": "median", "concurrency": 1, "coordinator_count": 1 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md#L105", "source_document_sha256": "b1d31e955106b701569c49476026ad674a838d376f93294395ae0b198d4fd6ee" }, { "label": "Two RTX coordinators + four Sparks, weighted DSpark", "metric": "decode", "value": 129.67, "unit": "tokens/s", "statistic": "median of three repeats", "uncertainty": null, "conditions": { "system": "1 or 2 RTX PRO 6000 Blackwell coordinators plus 4 DGX Spark workers", "node_count": null, "parallelism": "4 Spark expert workers, TP4", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-24", "model_revision": "dba1be0a40aa45a94ad051997016db3960a90277", "runtime_execution_revision": "af9d96923f70af1cd385960e28450e5ec0653ff8", "worker_count": 4, "worker_parallelism": "Spark TP4", "gpu_power_watts_each": 400, "repetitions": 3, "statistic": "median", "concurrency": 1, "coordinator_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md#L105", "source_document_sha256": "b1d31e955106b701569c49476026ad674a838d376f93294395ae0b198d4fd6ee" }, { "label": "One RTX coordinator + four Sparks, target only", "metric": "decode", "value": 59.84, "unit": "tokens/s", "statistic": "median of three repeats", "uncertainty": null, "conditions": { "system": "1 or 2 RTX PRO 6000 Blackwell coordinators plus 4 DGX Spark workers", "node_count": null, "parallelism": "4 Spark expert workers, TP4", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-24", "model_revision": "dba1be0a40aa45a94ad051997016db3960a90277", "runtime_execution_revision": "af9d96923f70af1cd385960e28450e5ec0653ff8", "worker_count": 4, "worker_parallelism": "Spark TP4", "gpu_power_watts_each": 400, "repetitions": 3, "statistic": "median", "concurrency": 1, "coordinator_count": 1, "speculative_decoding": false }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md#L105", "source_document_sha256": "b1d31e955106b701569c49476026ad674a838d376f93294395ae0b198d4fd6ee" }, { "label": "Two RTX coordinators, sampled C1 v14/v13", "metric": "decode_ratio", "value": 1.377, "unit": "ratio", "statistic": "geometric mean of session-score ratio", "uncertainty": { "interval_type": "Welch 95% interval on session log-ratios", "lower": 1.369, "upper": 1.385 }, "conditions": { "system": "1 or 2 RTX PRO 6000 Blackwell coordinators plus 4 DGX Spark workers", "node_count": null, "parallelism": "4 Spark expert workers, TP4", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-24", "model_revision": "dba1be0a40aa45a94ad051997016db3960a90277", "runtime_execution_revision": "af9d96923f70af1cd385960e28450e5ec0653ff8", "worker_count": 4, "worker_parallelism": "Spark TP4", "gpu_power_watts_each": 400, "repetitions": "8 interleaved sessions per layout; four per arm", "statistic": "median", "concurrency": 1, "coordinator_count": 2, "temperature": 0.7, "top_p": 0.9 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/tpurtell/ds41rt/blob/3067d0684a0b572d88a17d38ca3f7d10599619b4/docs/release-v14-performance.md#L39", "source_document_sha256": "b1d31e955106b701569c49476026ad674a838d376f93294395ae0b198d4fd6ee" } ] ``` --- # Two-Spark MoE: served-rate and warmup audit Two-node serving study that distinguishes end-to-end served rate from decode rate, and audits cache, warmup and checkpoint identity. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Multiple - Model: Qwen3.8-Flash-Next / GLM-5.3-Flash / DeepSeek-V4-Flash-0731 - Original author: WeZZard - Original project: https://github.com/WeZZard/dgx-spark-bench - Pinned document: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md - Source revision: 25e8dbf1b6113dd0b3f12d27a670fb284944d260 - Source commit time (not experiment date): 2026-09-06T00:35:29Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: vLLM, SGLang - Hardware topics: spark, nvidia - Reported setup: {"system": "2 x DGX Spark; 121 GiB unified memory each; 200 GbE", "node_count": 2, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": 512, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Fixed-length served rate", "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": 512, "concurrency": null, "cache_state": "full-shape warmup; per-node JIT cache; see row prefix-cache controls", "quantization": null, "speculative_decoding": "model-dependent NEXTN / DFlash2 / no-spec arms", "temperature": null, "top_p": null, "measurement_denominator": "whole request wall clock including TTFT" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Each request is forced to emit 512 tokens; served rate includes time to first token. - Every result retains model, weights, KV type, engine, users and KV pool. ## Limits - Text-0731 and Vision-Exp are separate checkpoints. - A draft model consumes KV capacity; compare the exact per-row configuration. ## License declaration {"declared_spdx": "MIT", "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."} ## Editorial answer The report divides all generated tokens by the complete request wall clock, including first-token delay. Every current table request is forced to 512 tokens with ignore_eos. Earlier naturally stopped outputs and 64-token correctness probes have different denominators and remain in the historical report rather than being merged into this cohort. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What does served rate include? The report divides all generated tokens by the complete request wall clock, including first-token delay. Every current table request is forced to 512 tokens with ignore_eos. Earlier naturally stopped outputs and 64-token correctness probes have different denominators and remain in the historical report rather than being merged into this cohort. Sources: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L25 ## Why were earlier results replaced? The author remeasured the campaign on 5 September after discovering a slow-clock GPU and unequal prefix-cache policy. The corrected runs disable prefix caching for all three models, exercise complete workload shapes during warmup and retain per-node JIT caches. Earlier measurements remain visible as history; they should not be treated as the corrected cohort. Sources: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L45 ## What can the long-context rows establish? At one user in the long agentic workload, the reported medians are 22.5, 21.0 and 17.7 served tokens/s, with different between-campaign spreads. The author labels this row not separated by the data. Different draft models and KV pools remain attached to each model, and the high-concurrency rows do not even use the same number of users across models. Sources: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L70 ## Why are failures and checkpoint identity useful? The README reports a Qwen FP8-KV configuration that failed on chunked input and GLM backend attempts that did not complete a forward pass. Its DeepSeek current rows use the text 0731 checkpoint, not Vision-Exp and not V4.1 Flash 1017. These failures and identities explain why a recipe label cannot substitute for the actual six-field serving configuration. Sources: https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L50 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "DeepSeek V4 Flash 0731, long agentic", "metric": "served_rate", "value": 22.5, "unit": "tokens/s", "statistic": "median of three campaigns", "uncertainty": { "spread_ratio": 1.04 }, "conditions": { "system": "2 x DGX Spark; 121 GiB unified memory each; 200 GbE", "node_count": 2, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-05", "output_tokens": 512, "ignore_eos": true, "prefix_cache": false, "warmup": "all workload/user shapes, full generation", "repetitions": 3, "concurrency": 1, "measurement_denominator": "whole request wall clock including TTFT", "workload": "agentic 33k context", "model": "DeepSeek-V4-Flash-0731", "kv": "fp8_ds_mla", "kv_pool_tokens": 517484, "draft": "DSpark k5 probabilistic" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L70", "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469" }, { "label": "Qwen3.8 Flash-Next, long agentic", "metric": "served_rate", "value": 21.0, "unit": "tokens/s", "statistic": "median of three campaigns", "uncertainty": { "spread_ratio": 1.01 }, "conditions": { "system": "2 x DGX Spark; 121 GiB unified memory each; 200 GbE", "node_count": 2, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-05", "output_tokens": 512, "ignore_eos": true, "prefix_cache": false, "warmup": "all workload/user shapes, full generation", "repetitions": 3, "concurrency": 1, "measurement_denominator": "whole request wall clock including TTFT", "workload": "agentic 33k context", "model": "Qwen3.8-Flash-Next NVFP4", "kv": "BF16", "kv_pool_tokens": 161728, "draft": "NEXTN" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L70", "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469" }, { "label": "GLM5.3 Flash, long agentic", "metric": "served_rate", "value": 17.7, "unit": "tokens/s", "statistic": "median of three campaigns", "uncertainty": { "spread_ratio": 1.32 }, "conditions": { "system": "2 x DGX Spark; 121 GiB unified memory each; 200 GbE", "node_count": 2, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-05", "output_tokens": 512, "ignore_eos": true, "prefix_cache": false, "warmup": "all workload/user shapes, full generation", "repetitions": 3, "concurrency": 1, "measurement_denominator": "whole request wall clock including TTFT", "workload": "agentic 33k context", "model": "GLM-5.3-Flash", "kv": "BF16", "kv_pool_tokens": 162688, "draft": "DFlash2" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L70", "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469" } ] ``` --- # Spark community quality cohort: v7, 12 October A dated 80-scenario quality cohort with two repeats, model-card sampling and per-deployment reports. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Multiple - Model: Multiple local deployments and hosted API references - Original author: Weschera - Original project: https://github.com/Weschera/spark-bench - Pinned document: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md - Source revision: 3eaaeb66aaef7ccf96f3b6e39803817e987cf934 - Source commit time (not experiment date): 2026-10-13T03:42:23Z - Checked: 2026-10-17 - Experiment date: 2026-10-12 - Runtime topics: Multiple runtimes - Hardware topics: spark, nvidia, apple - Reported setup: {"system": "DGX Spark / Mac Studio / hosted APIs, per row", "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-10-12"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "v7.0 cohort only", "evidence_url": "https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": "uncapped within each deployment context", "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": "each model card; per deployment", "top_p": null, "repetitions": 2, "seeds": [ 2001, 2002 ], "scenario_count": 80, "domain_count": 13, "thinking": "on/off separate rows", "hardware": "per-row local hardware; hosted hardware unknown" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The cohort uses 13 domains and seeds 2001–2002. - Thinking-on and thinking-off rows are separate. ## Limits - v7 scores are not comparable with v6.x. - Hosted API latency includes the network and is not a controlled hardware comparison. ## License declaration {"declared_spdx": null, "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."} ## Editorial answer The dated comparison contains 33 valid full deployment runs of v7.0, with 80 scenarios across 13 domains and two repeated seeds. A row identifies a model, quantization, engine/recipe, hardware and thinking mode. TrueScore is the project’s suite score, not a percent-correct rate and not a measure of generation speed. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What is the unit of this quality cohort? The dated comparison contains 33 valid full deployment runs of v7.0, with 80 scenarios across 13 domains and two repeated seeds. A row identifies a model, quantization, engine/recipe, hardware and thinking mode. TrueScore is the project’s suite score, not a percent-correct rate and not a measure of generation speed. Sources: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L3 ## Why retain thinking mode and deployment identity? The GLM Flash example is 88.9 with thinking enabled and 84.8 with thinking disabled under the named two-Spark EXL3/TensorFold recipe. Those are two source rows, not a general estimate of the value of reasoning across all tasks or runtimes. Different quantizations, engines or hardware in other rows create additional differences. Sources: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L7 ## How are incomplete and failed runs handled? The cohort requires its golden gate and a transport-error limit below five percent. Three hosted runs each had one timeout. The document excludes incomplete runs and describes reasoning-channel parsing failures separately. Counting only the successful top-line score would discard information the author deliberately retained about the evaluation process. Sources: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L70 ## What comparisons are unsupported? The author says v7 is not comparable with v6.x and notes that easy shared scenarios compress the upper scores. Hosted APIs have uncontrolled hardware and network latency. This page therefore excerpts two contextual examples rather than republishing the full leaderboard or converting it into a site-wide ranking. No license declaration was found in repository metadata; the original report remains the full source. Sources: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L62 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "GLM-5.3 Flash, thinking on", "metric": "TrueScore", "value": 88.9, "unit": "suite score", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "DGX Spark / Mac Studio / hosted APIs, per row", "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-10-12", "suite": "v7.0 full, 80 scenarios / 13 domains", "repetitions": 2, "seeds": [ 2001, 2002 ], "sampling": "model-card settings", "output_budget": "uncapped within each deployment context", "hardware": "2 Sparks TP2", "quantization": "EXL3 4bpw", "runtime": "TensorFold v0.6.0 + DFlash2, Mia recipe v1.8", "thinking": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L12", "source_document_sha256": "10ceff97d360c33a22e002db2e746a7d7de2aa9bd31051813a8d77e18f60c2cc" }, { "label": "GLM-5.3 Flash, thinking off", "metric": "TrueScore", "value": 84.8, "unit": "suite score", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "DGX Spark / Mac Studio / hosted APIs, per row", "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-10-12", "suite": "v7.0 full, 80 scenarios / 13 domains", "repetitions": 2, "seeds": [ 2001, 2002 ], "sampling": "model-card settings", "output_budget": "uncapped within each deployment context", "hardware": "2 Sparks TP2", "quantization": "EXL3 4bpw", "runtime": "TensorFold v0.6.0 + DFlash2, Mia recipe v1.8", "thinking": false }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L26", "source_document_sha256": "10ceff97d360c33a22e002db2e746a7d7de2aa9bd31051813a8d77e18f60c2cc" } ] ``` --- # Qwen3.5 on M4 Pro: dated precision matrix A fixed March M4 Pro report with repeated generation, first-token and memory measurements across available precisions. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen3.5 0.8B / 2B / 4B / 9B / 27B / 35B-A3B - Original author: weklund - Original project: https://github.com/weklund/mlx-coding-bench - Pinned document: https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md - Source revision: 5ae9b491f7a0539d5abef949eded6bd57785a981 - Source commit time (not experiment date): 2026-07-01T19:16:36Z - Checked: 2026-10-17 - Experiment date: 2026-03-06 - Runtime topics: MLX - Hardware topics: apple - Reported setup: {"system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Generation speed", "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": 256, "output_tokens": 100, "concurrency": null, "cache_state": null, "quantization": "int4 / int8 / BF16 as listed per model", "speculative_decoding": null, "temperature": null, "top_p": null, "runtime_version": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1 } }, { "label": "First-token latency and peak memory", "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P%2B4E%2B20GPU_64GB/2026-03-06__19%3A47%3A24/REPORT.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": 4096, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null, "runtime_version": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1 } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Generation and first-token/memory tests use different prompt lengths. - The report states 20 iterations, one warmup and mean/standard deviation. ## Limits - Only listed model/precision cells were tested. - The historical report is retained as dated evidence, not current Qwen3.8 performance. ## License declaration {"declared_spdx": "MIT", "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."} ## Editorial answer The operator report is dated 6 March 2026 and names an M4 Pro with ten performance and four efficiency CPU cores, twenty GPU cores and 64 GB unified memory. It uses MLX 0.31.0 and mlx-lm 0.30.7, one warmup followed by twenty measured iterations. These are historical Qwen3.5 results, not measurements of current Qwen3.8 or another Apple chip. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## Which exact machine and stack were tested? The operator report is dated 6 March 2026 and names an M4 Pro with ten performance and four efficiency CPU cores, twenty GPU cores and 64 GB unified memory. It uses MLX 0.31.0 and mlx-lm 0.30.7, one warmup followed by twenty measured iterations. These are historical Qwen3.5 results, not measurements of current Qwen3.8 or another Apple chip. Sources: https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L3 ## Why are the generation and latency tables separate? Generation uses a 256-token prompt and 100-token output. First-token latency and peak-memory tables use a 4,096-token prompt. The 4B int4 example consequently reports 68.0 ± 0.4 generated tokens/s in one profile and 6.14 ± 0.25 seconds to first token in the other. Those values cannot be assembled into one measured end-to-end 256-token request. Sources: https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L21 ## What does the dense versus MoE example show? The selected int4 rows put 27B at 11.8 ± 0.2 tokens/s and 35B-A3B at 25.6 ± 0.5 under the generation profile. At the longer prompt their reported peaks are 22.1 and 23.7 GiB. That is an observed speed/memory tradeoff on this stack; it does not prove equal coding quality or a broad rule that one architecture is better. Sources: https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L45 ## Which claims are not reproduced here? The source adds qualitative recommendations and secondary-source quality claims after its measured tables. We keep those separate from the directly reported MLX timing and memory cells. Missing BF16 cells stay missing; they are not zero-speed or failed measurements. Weight hashes, cache state and concurrent-user counts are not specified by the selected report. Sources: https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L58 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Qwen3.5-4B int4 generation", "metric": "generation", "value": 68.0, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.4 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-4B int4", "quantization": "int4", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L27", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-4B int4 first token", "metric": "ttft", "value": 6.14, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.25 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-4B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L40", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-4B int4 memory", "metric": "peak_memory", "value": 6.4, "unit": "GiB", "statistic": "reported peak memory", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-4B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L53", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-9B int4 generation", "metric": "generation", "value": 38.5, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.4 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-9B int4", "quantization": "int4", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L28", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-9B int4 first token", "metric": "ttft", "value": 10.88, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.61 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-9B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L41", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-9B int4 memory", "metric": "peak_memory", "value": 9.0, "unit": "GiB", "statistic": "reported peak memory", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-9B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L54", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-27B int4 generation", "metric": "generation", "value": 11.8, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.2 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-27B int4", "quantization": "int4", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L29", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-27B int4 first token", "metric": "ttft", "value": 36.57, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.99 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-27B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L42", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-27B int4 memory", "metric": "peak_memory", "value": 22.1, "unit": "GiB", "statistic": "reported peak memory", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-27B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L55", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-35B-A3B int4 generation", "metric": "generation", "value": 25.6, "unit": "tokens/s", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.5 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-35B-A3B int4", "quantization": "int4", "input_tokens": 256, "output_tokens": 100 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L30", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-35B-A3B int4 first token", "metric": "ttft", "value": 5.52, "unit": "seconds", "statistic": "mean of 20 measurements", "uncertainty": { "standard_deviation": 0.1 }, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-35B-A3B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L43", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" }, { "label": "Qwen3.5-35B-A3B int4 memory", "metric": "peak_memory", "value": 23.7, "unit": "GiB", "statistic": "reported peak memory", "uncertainty": null, "conditions": { "system": "Apple M4 Pro 10P+4E / 20 GPU cores / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-03-06", "runtime": "MLX 0.31.0 / mlx-lm 0.30.7", "repetitions": 20, "warmup_iterations": 1, "concurrency": null, "model_revision": null, "cache_state": null, "model": "Qwen3.5-35B-A3B int4", "quantization": "int4", "input_tokens": 4096, "output_tokens": null }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/weklund/mlx-coding-bench/blob/5ae9b491f7a0539d5abef949eded6bd57785a981/measurements/llm_benchmarks/Apple_M4_Pro_10P+4E+20GPU_64GB/2026-03-06__19:47:24/REPORT.md#L56", "source_document_sha256": "ff21a1f8a2597867e191a376cea28af293d47dfff82bb423193009b05c7ef5a3" } ] ``` --- # llama.cpp on Spark: Q4_K_M prefill and decode Original Spark report separating prompt processing from single-batch generation at different context depths. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Multiple - Model: Qwen3 / Ministral 3 - Original author: DandinPower - Original project: https://github.com/DandinPower/llama.cpp_bench - Pinned document: https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md - Source revision: a1e9514561e9be68982e1905f48808143082013a - Source commit time (not experiment date): 2025-12-05T13:40:35Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: llama.cpp - Hardware topics: spark, nvidia - Reported setup: {"system": "NVIDIA DGX Spark / GB10 / 128 GB LPDDR5x", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": "Q4_K_M", "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Prompt processing", "evidence_url": "https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": "Q4_K_M", "speculative_decoding": null, "temperature": null, "top_p": null, "context_depth_tokens": [ 512, 16384 ] } }, { "label": "Generation", "evidence_url": "https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": "Q4_K_M", "speculative_decoding": null, "temperature": null, "top_p": null, "batch_size": 1, "context_depth_tokens": [ 512, 2048 ] } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Q4_K_M is the selected weight format. - Prefill and decode tables describe different context depths. ## Limits - Exact model and runtime commits are not specified in this report. - A tested context depth is not the machine’s maximum supported context. ## License declaration {"declared_spdx": null, "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."} ## Editorial answer The report studies Qwen3 and Ministral 3 Q4_K_M artifacts on a single GB10 Spark with 128 GB unified memory. Prompt processing is tabulated at context depths 512 and 16,384; single-batch generation at 512 and 2,048. Context depth is the source table’s label, not a specified generated-output budget or a maximum capacity test. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What experiment is being described? The report studies Qwen3 and Ministral 3 Q4_K_M artifacts on a single GB10 Spark with 128 GB unified memory. Prompt processing is tabulated at context depths 512 and 16,384; single-batch generation at 512 and 2,048. Context depth is the source table’s label, not a specified generated-output budget or a maximum capacity test. Sources: https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md#L20 ## What is a useful concrete comparison? The selected Qwen3 30B MoE row reports 89.3 decode tokens/s at depth 512 and 83.8 at 2,048, while the 32B dense row reports 10.7 at depth 512. These results illustrate the author’s bandwidth interpretation for this particular pair of Q4_K_M artifacts. They do not establish equal quality or the same outcome for another MoE family. Sources: https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md#L33 ## What prevents a strong reproduction claim? The report does not identify exact weight hashes, executed runtime commit, output lengths, cache policy or repetitions. Those gaps remain explicit. Without them, a later reader cannot know whether a discrepancy comes from kernels, artifacts or workload. A tested context depth also cannot certify the machine’s maximum context. Sources: https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md#L18 ## How is the source reused? This article gives a small attributed factual selection with an original explanation. Repository metadata does not declare a license, so the full result table and prose are left with the original author. No independent run is claimed, and prefill and decode remain separate measurement types. Sources: https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md#L46 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Qwen3 30B MoE, shallow context", "metric": "decode", "value": 89.3, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "NVIDIA DGX Spark / GB10 / 128 GB LPDDR5x", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null, "runtime": "llama.cpp, executed commit unspecified", "quantization": "Q4_K_M", "batch_size": 1, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "model": "Qwen3 30B MoE", "context_depth_tokens": 512 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md#L40", "source_document_sha256": "44d415d26c12533d1694fcc7b622f8961ea0a320078177e14c663a5b77028c70" }, { "label": "Qwen3 30B MoE, deeper context", "metric": "decode", "value": 83.8, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "NVIDIA DGX Spark / GB10 / 128 GB LPDDR5x", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null, "runtime": "llama.cpp, executed commit unspecified", "quantization": "Q4_K_M", "batch_size": 1, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "model": "Qwen3 30B MoE", "context_depth_tokens": 2048 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md#L40", "source_document_sha256": "44d415d26c12533d1694fcc7b622f8961ea0a320078177e14c663a5b77028c70" }, { "label": "Qwen3 32B dense, shallow context", "metric": "decode", "value": 10.7, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "NVIDIA DGX Spark / GB10 / 128 GB LPDDR5x", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null, "runtime": "llama.cpp, executed commit unspecified", "quantization": "Q4_K_M", "batch_size": 1, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "model": "Qwen3 32B dense", "context_depth_tokens": 512 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/DandinPower/llama.cpp_bench/blob/a1e9514561e9be68982e1905f48808143082013a/dgx_spark/report.md#L41", "source_document_sha256": "44d415d26c12533d1694fcc7b622f8961ea0a320078177e14c663a5b77028c70" } ] ``` --- # MLX LM: M4 Max precision and efficiency reference Maintainer benchmark document with M4 Max hardware and explicit package versions, separating capability and efficiency columns. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: Qwen3-4B / Qwen3-30B-A3B Instruct-2507 - Original author: ml-explore - Original project: https://github.com/ml-explore/mlx-lm - Pinned document: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md - Source revision: 93e6d4a6447b3080861985947f0820e76fcb957d - Source commit time (not experiment date): 2026-10-16T02:48:11Z - Checked: 2026-10-17 - Experiment date: None - Runtime topics: MLX - Hardware topics: apple - Reported setup: {"system": "Apple M4 Max / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Efficiency benchmark", "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md", "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": 2048, "output_tokens": 128, "concurrency": null, "cache_state": null, "quantization": "BF16 / q8 / q6 / q5 / q4, per model row", "speculative_decoding": null, "temperature": null, "top_p": null, "runtime_version": "mlx 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2 / macOS 26.1" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The efficiency command specifies 2048 prompt and 128 generated tokens. - Per-model precision rows retain separate memory and quality measurements. ## Limits - This dated package stack is not a measurement of current MLX or Qwen3.8. - Missing, failed and untested table cells do not mean zero performance. ## License declaration {"declared_spdx": "MIT", "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."} ## Editorial answer The efficiency command uses 2,048 prompt tokens and 128 generated tokens on an M4 Max with 64 GB unified memory. It records a development MLX build, mlx-lm 0.28.2 and macOS 26.1. Its separate evaluation command runs MMLU Pro. Timing and quality therefore come from different programs, even when printed on the same model row. By Run AI Here editorial · 2026-10-17 AI-assisted source reading and original editorial synthesis; no independent model execution. ## What does the maintainer document fix? The efficiency command uses 2,048 prompt tokens and 128 generated tokens on an M4 Max with 64 GB unified memory. It records a development MLX build, mlx-lm 0.28.2 and macOS 26.1. Its separate evaluation command runs MMLU Pro. Timing and quality therefore come from different programs, even when printed on the same model row. Sources: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L14 ## What is the observed precision tradeoff? For Qwen3-4B Instruct-2507, the selected BF16 and q4 rows report 52.47 and 134.52 generation tokens/s. The corresponding MMLU Pro values are 64.05 and 60.72. Faster decoding does not establish unchanged capability, and a task score cannot be interpreted as a decode rate. Group-size-specific q4 and q5 rows are separate artifacts. Sources: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L29 ## How should missing cells be represented? The 30B-A3B BF16 row contains unavailable efficiency cells while retaining a quality value. Those cells remain absent rather than being converted to zero or to a fabricated failure diagnosis. The memory column is labeled GB by the source and is not silently relabeled GiB. Repetitions and exact weight revisions remain unspecified here. Sources: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L51 ## What is the appropriate use of this reference? Use it as a dated maintainer example of a performance/quality tradeoff on an identified stack. It is not a benchmark of current MLX, Qwen3.8 or a different Apple chip. Preserve the conversion repository, precision/group size, model revision and command parameters before comparing another measurement. Sources: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L23 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Qwen3-4B Instruct-2507 BF16", "metric": "generation", "value": 52.47, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "Apple M4 Max / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null, "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2", "os": "macOS 26.1", "input_tokens": 2048, "output_tokens": 128, "repetitions": null, "cache_state": null, "concurrency": null, "model": "Qwen/Qwen3-4B-Instruct-2507", "quantization": "BF16" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L31", "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218" }, { "label": "Qwen3-4B Instruct-2507 q4", "metric": "generation", "value": 134.52, "unit": "tokens/s", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "Apple M4 Max / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null, "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2", "os": "macOS 26.1", "input_tokens": 2048, "output_tokens": 128, "repetitions": null, "cache_state": null, "concurrency": null, "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit", "quantization": "q4" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L37", "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218" }, { "label": "Qwen3-4B Instruct-2507 BF16", "metric": "MMLU_Pro", "value": 64.05, "unit": "reported score", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "Apple M4 Max / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null, "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2", "os": "macOS 26.1", "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "concurrency": null, "model": "Qwen/Qwen3-4B-Instruct-2507", "quantization": "BF16", "harness": "mlx_lm.evaluate --task mmlu_pro" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L31", "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218" }, { "label": "Qwen3-4B Instruct-2507 q4", "metric": "MMLU_Pro", "value": 60.72, "unit": "reported score", "statistic": "reported value", "uncertainty": null, "conditions": { "system": "Apple M4 Max / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null, "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2", "os": "macOS 26.1", "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "concurrency": null, "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit", "quantization": "q4", "harness": "mlx_lm.evaluate --task mmlu_pro" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L37", "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218" } ] ``` --- # DeepSeek 1017: TensorFold on two DGX Sparks A traceable 1017 EXL3 deployment and two named decode configurations, with a separate later prefill cohort and explicit correctness limits. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw - Original author: Elevatormusic - Original project: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark - Pinned document: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/README.md - Source revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - Source commit time (not experiment date): 2026-10-18T05:40:29Z - Checked: 2026-10-18 - Experiment date: 2026-10-17 and 2026-10-18, separate cohorts - Runtime topics: TensorFold, vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "2 x DGX Spark GB10 TP2 over ConnectX-7", "node_count": 2, "parallelism": "TP2", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-10-17 and 2026-10-18, separate cohorts"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "vllm-kit-run01 prose", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json#L6", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "vllm-kit-run01", "engine": "vLLM + exllamav3 overlay", "engine_version": null, "runtime_execution_revision": null, "image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "kit_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "draft_tokens": 3, "configured_max_requests": 1, "context_limit_tokens": 262144, "prefill_chunk_tokens": 1536, "chat_template": "kit template; actual CHAT_TEMPLATE override not read by the source", "workload": "prose", "per_prompt_decode_rates": [ 24.7, 29.3 ], "per_prompt_TTFT_seconds": [ 0.32, 0.28 ], "prompt_count": 2 } }, { "label": "vllm-kit-run01 code", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json#L20", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "vllm-kit-run01", "engine": "vLLM + exllamav3 overlay", "engine_version": null, "runtime_execution_revision": null, "image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "kit_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "draft_tokens": 3, "configured_max_requests": 1, "context_limit_tokens": 262144, "prefill_chunk_tokens": 1536, "chat_template": "kit template; actual CHAT_TEMPLATE override not read by the source", "workload": "code", "per_prompt_decode_rates": [ 40.4, 39.4 ], "per_prompt_TTFT_seconds": [ 0.47, 0.32 ], "prompt_count": 2 } }, { "label": "tensorfold-run11 prose", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json#L6", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "tensorfold-run11", "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": "fd2365042f56fcf726cf4071132c8b779ad2eae0", "image_tag": "tensorfold-dsv41:fd2365042f56", "image_digest": null, "draft_tokens": 5, "configured_max_requests": 4, "context_limit_tokens": 1048576, "prefill_chunk_tokens": 2048, "chat_template": "patched kit template; source infers per-run use from file times", "workload": "prose", "per_prompt_decode_rates": [ 53.4, 60.4 ], "per_prompt_TTFT_seconds": [ 0.12, 0.12 ], "prompt_count": 2 } }, { "label": "tensorfold-run11 code", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json#L20", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "tensorfold-run11", "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": "fd2365042f56fcf726cf4071132c8b779ad2eae0", "image_tag": "tensorfold-dsv41:fd2365042f56", "image_digest": null, "draft_tokens": 5, "configured_max_requests": 4, "context_limit_tokens": 1048576, "prefill_chunk_tokens": 2048, "chat_template": "patched kit template; source infers per-run use from file times", "workload": "code", "per_prompt_decode_rates": [ 91.1, 92.5 ], "per_prompt_TTFT_seconds": [ 0.13, 0.16 ], "prompt_count": 2 } }, { "label": "TensorFold final setup 32K prefill", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L16", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-18", "concurrency": 1, "thinking": false, "temperature": null, "top_p": null, "max_output_tokens": null, "output_tokens": null, "input_tokens": null, "prefix_cache": false, "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "prompt_tokens / TTFT; source final setup and host tuning", "repetitions": 5, "checkpoint_is_latest_revision_verified": false, "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": null, "runtime_revision_prefix": "92aa710b533d", "image_tag": "tensorfold-dsv41:92aa710b533d", "image_digest": null, "reported_prompt_bucket": "32K", "cached_tokens": 0, "per_run_values": [ 2201, 2180, 2065, 2165, 2100 ], "run_ids": [ "hw5", "hw6", "ht1", "ht2", "ht3" ], "draft_tokens": 5, "context_limit_tokens": 1048576, "warmup": "one 8K prompt; subsequent 8K requests before 32K/64K/128K; random-word prompts", "host_tuning_scope": "author applies no engine swap and memory-reclaim/compaction controls; not a universal host-tuning recommendation" } }, { "label": "TensorFold final setup 128K prefill", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L16", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-18", "concurrency": 1, "thinking": false, "temperature": null, "top_p": null, "max_output_tokens": null, "output_tokens": null, "input_tokens": null, "prefix_cache": false, "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "prompt_tokens / TTFT; source final setup and host tuning", "repetitions": 5, "checkpoint_is_latest_revision_verified": false, "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": null, "runtime_revision_prefix": "92aa710b533d", "image_tag": "tensorfold-dsv41:92aa710b533d", "image_digest": null, "reported_prompt_bucket": "128K", "cached_tokens": 0, "per_run_values": [ 1986, 1981, 1914, 1961, 1851 ], "run_ids": [ "hw5", "hw6", "ht1", "ht2", "ht3" ], "draft_tokens": 5, "context_limit_tokens": 1048576, "warmup": "one 8K prompt; subsequent 8K requests before 32K/64K/128K; random-word prompts", "host_tuning_scope": "author applies no engine swap and memory-reclaim/compaction controls; not a universal host-tuning recommendation" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The selected 1017 results describe two complete serving configurations, not an isolated engine-only test. Keep the converted weights and separate Engram revision, distinguish October 17 decode from October 18 prefill, and retain missing image, cache and quality evidence. ## Limits - Source-reported only; no independent site model runs. - Document revision identifies the report, not the executed runtime or weight revision. - Complete measured configurations and missing fields must accompany extracted values. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Repository metadata only; model conversion, base weights, plugin and engine terms retain their own scope."} ## Editorial answer The selected 1017 results describe two complete serving configurations, not an isolated engine-only test. Keep the converted weights and separate Engram revision, distinguish October 17 decode from October 18 prefill, and retain missing image, cache and quality evidence. By Run AI Here editorial · 2026-10-18 Local Spark Qwen draft, source-audited and edited by Run AI Here with Codex; no independent model execution. ## Which 1017 artifact was measured? The report uses an EXL3 2.9bpw conversion at afc7aa0aab98ee735c0f54e0a122019af1ac08f1 plus separate official 1017 Engram tables at 1bae0e1088993b9b82de08f59de5be5a34e42170. The author does not establish the measured conversion is the newest revision. This is a particular converted artifact, not a speed result for unconverted official weights. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/PROVENANCE.md ## What changes between the two decode arms? Both use one request on the same reported model view and Spark pair. The vLLM kit has three drafts and a 262144 context cap; TensorFold run 11 has five drafts and a 1048576 cap. Their batching, configured concurrency and template paths also differ. The selected two-prompt means are observations of complete configurations, not an isolated engine effect. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/BENCHMARKS.md ## Why not use the headline median as six repeats? The report explicitly says its six TensorFold suite runs used different builds. Run 11 identifies fd2365042f56, whereas the current recipe builds 92aa710b533d. Keep those versions apart. The baseline has only one run, and there is no measured multi-user vLLM baseline. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/BENCHMARKS.md ## What does the later prefill cohort establish? Five October 18 source rows use the final setup, random-word prompts and recorded zero cached tokens. The selected 32K and 128K medians retain all five row values and their observed ranges. These are separate from the older repeated-sentence suite, where cache hits could not be excluded; range is not a confidence interval. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md ## Where are the limits? Image input and general accuracy were not measured. Near-million-token needle retrieval does not prove general reasoning, decode speed or simultaneous long-session capacity. Runtime caches, Engram storage and per-node memory headroom need their own checks. Remaining slower states are acknowledged by the author. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/KNOWN-LIMITS.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "vllm-kit-run01 prose", "metric": "decode_rate", "value": 27.0, "unit": "tokens/s", "statistic": "one run; mean of two prompt-level decode rates; rounded by source", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "vllm-kit-run01", "engine": "vLLM + exllamav3 overlay", "engine_version": null, "runtime_execution_revision": null, "image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "kit_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "draft_tokens": 3, "configured_max_requests": 1, "context_limit_tokens": 262144, "prefill_chunk_tokens": 1536, "chat_template": "kit template; actual CHAT_TEMPLATE override not read by the source", "workload": "prose", "per_prompt_decode_rates": [ 24.7, 29.3 ], "per_prompt_TTFT_seconds": [ 0.32, 0.28 ], "prompt_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json#L6", "source_document_sha256": "0cb7c61b5b321ba0b9eec76460e4d9d3d0e759bd09cfb03fc747ef66e7c2ea63", "source_document_id": "ds1017-runs-2026-10-17-01_vllm-miaai-kit_suite-results_32of32.json", "source_line": 6 }, { "label": "vllm-kit-run01 code", "metric": "decode_rate", "value": 39.9, "unit": "tokens/s", "statistic": "one run; mean of two prompt-level decode rates; rounded by source", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "vllm-kit-run01", "engine": "vLLM + exllamav3 overlay", "engine_version": null, "runtime_execution_revision": null, "image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "kit_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "draft_tokens": 3, "configured_max_requests": 1, "context_limit_tokens": 262144, "prefill_chunk_tokens": 1536, "chat_template": "kit template; actual CHAT_TEMPLATE override not read by the source", "workload": "code", "per_prompt_decode_rates": [ 40.4, 39.4 ], "per_prompt_TTFT_seconds": [ 0.47, 0.32 ], "prompt_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json#L20", "source_document_sha256": "0cb7c61b5b321ba0b9eec76460e4d9d3d0e759bd09cfb03fc747ef66e7c2ea63", "source_document_id": "ds1017-runs-2026-10-17-01_vllm-miaai-kit_suite-results_32of32.json", "source_line": 20 }, { "label": "tensorfold-run11 prose", "metric": "decode_rate", "value": 56.9, "unit": "tokens/s", "statistic": "one run; mean of two prompt-level decode rates; rounded by source", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "tensorfold-run11", "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": "fd2365042f56fcf726cf4071132c8b779ad2eae0", "image_tag": "tensorfold-dsv41:fd2365042f56", "image_digest": null, "draft_tokens": 5, "configured_max_requests": 4, "context_limit_tokens": 1048576, "prefill_chunk_tokens": 2048, "chat_template": "patched kit template; source infers per-run use from file times", "workload": "prose", "per_prompt_decode_rates": [ 53.4, 60.4 ], "per_prompt_TTFT_seconds": [ 0.12, 0.12 ], "prompt_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json#L6", "source_document_sha256": "d0ec8c99b0208ac4ddc272379d3cd71c796b5662d09481d6ba1e98fbbf0fa41c", "source_document_id": "ds1017-runs-2026-10-17-11_tensorfold-fd2365042f56_suite-results_32of32.json", "source_line": 6 }, { "label": "tensorfold-run11 code", "metric": "decode_rate", "value": 91.8, "unit": "tokens/s", "statistic": "one run; mean of two prompt-level decode rates; rounded by source", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-17", "concurrency": 1, "thinking": false, "temperature": 1.0, "top_p": 1.0, "max_output_tokens": 700, "output_tokens": null, "input_tokens": null, "prefix_cache": "present; cache-hit counts not logged in this suite", "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "(completion_tokens - 1) / (stream end - first chunk with content, reasoning or tool call)", "repetitions": 1, "checkpoint_is_latest_revision_verified": false, "id": "tensorfold-run11", "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": "fd2365042f56fcf726cf4071132c8b779ad2eae0", "image_tag": "tensorfold-dsv41:fd2365042f56", "image_digest": null, "draft_tokens": 5, "configured_max_requests": 4, "context_limit_tokens": 1048576, "prefill_chunk_tokens": 2048, "chat_template": "patched kit template; source infers per-run use from file times", "workload": "code", "per_prompt_decode_rates": [ 91.1, 92.5 ], "per_prompt_TTFT_seconds": [ 0.13, 0.16 ], "prompt_count": 2 }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json#L20", "source_document_sha256": "d0ec8c99b0208ac4ddc272379d3cd71c796b5662d09481d6ba1e98fbbf0fa41c", "source_document_id": "ds1017-runs-2026-10-17-11_tensorfold-fd2365042f56_suite-results_32of32.json", "source_line": 20 }, { "label": "TensorFold final setup 32K prefill", "metric": "prefill_rate", "value": 2165, "unit": "tokens/s", "statistic": "median of five source rows; individually recoverable in per_run_values", "uncertainty": { "observed_min": 2065, "observed_max": 2201, "interval_kind": "observed range, not confidence interval" }, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-18", "concurrency": 1, "thinking": false, "temperature": null, "top_p": null, "max_output_tokens": null, "output_tokens": null, "input_tokens": null, "prefix_cache": false, "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "prompt_tokens / TTFT; source final setup and host tuning", "repetitions": 5, "checkpoint_is_latest_revision_verified": false, "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": null, "runtime_revision_prefix": "92aa710b533d", "image_tag": "tensorfold-dsv41:92aa710b533d", "image_digest": null, "reported_prompt_bucket": "32K", "cached_tokens": 0, "per_run_values": [ 2201, 2180, 2065, 2165, 2100 ], "run_ids": [ "hw5", "hw6", "ht1", "ht2", "ht3" ], "draft_tokens": 5, "context_limit_tokens": 1048576, "warmup": "one 8K prompt; subsequent 8K requests before 32K/64K/128K; random-word prompts", "host_tuning_scope": "author applies no engine swap and memory-reclaim/compaction controls; not a universal host-tuning recommendation" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L16", "source_document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "source_document_id": "ds1017-runs-2026-10-18-INDEX.md", "source_line": 16 }, { "label": "TensorFold final setup 128K prefill", "metric": "prefill_rate", "value": 1961, "unit": "tokens/s", "statistic": "median of five source rows; individually recoverable in per_run_values", "uncertainty": { "observed_min": 1851, "observed_max": 1986, "interval_kind": "observed range, not confidence interval" }, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "2 x NVIDIA DGX Spark GB10, TP2 over ConnectX-7", "node_count": 2, "driver": "580.178.04", "CUDA": "13.0", "experiment_date": "2026-10-18", "concurrency": 1, "thinking": false, "temperature": null, "top_p": null, "max_output_tokens": null, "output_tokens": null, "input_tokens": null, "prefix_cache": false, "kv_dtype": null, "kv_pool_tokens": null, "measurement_denominator": "prompt_tokens / TTFT; source final setup and host tuning", "repetitions": 5, "checkpoint_is_latest_revision_verified": false, "engine": "TensorFold", "engine_version": "0.6.6", "runtime_execution_revision": null, "runtime_revision_prefix": "92aa710b533d", "image_tag": "tensorfold-dsv41:92aa710b533d", "image_digest": null, "reported_prompt_bucket": "128K", "cached_tokens": 0, "per_run_values": [ 1986, 1981, 1914, 1961, 1851 ], "run_ids": [ "hw5", "hw6", "ht1", "ht2", "ht3" ], "draft_tokens": 5, "context_limit_tokens": 1048576, "warmup": "one 8K prompt; subsequent 8K requests before 32K/64K/128K; random-word prompts", "host_tuning_scope": "author applies no engine swap and memory-reclaim/compaction controls; not a universal host-tuning recommendation" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L16", "source_document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "source_document_id": "ds1017-runs-2026-10-18-INDEX.md", "source_line": 16 } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "ds1017-docs-BENCHMARKS.md", "parent_record_id": "deepseek-1017-tensorfold-spark", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "docs/BENCHMARKS.md", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/BENCHMARKS.md", "document_sha256": "b2d05a077ae8145a2cf2a98ac2ae2eabef87b992230c32398a60e0a2e9174861", "checked_on": "2026-10-18" }, { "document_id": "ds1017-docs-KNOWN-LIMITS.md", "parent_record_id": "deepseek-1017-tensorfold-spark", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "docs/KNOWN-LIMITS.md", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/KNOWN-LIMITS.md", "document_sha256": "eeb6bc6f7a3fce20654f977581e9718b124e70fc01ab358efebefe9afd9dc942", "checked_on": "2026-10-18" }, { "document_id": "ds1017-runs-2026-10-17-PROVENANCE.md", "parent_record_id": "deepseek-1017-tensorfold-spark", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "runs/2026-10-17/PROVENANCE.md", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/PROVENANCE.md", "document_sha256": "aa020bd6aa0fe840c9518c96dbc00f04bbdf6add834f2fe7bac2b076aa91a089", "checked_on": "2026-10-18" }, { "document_id": "ds1017-runs-2026-10-18-INDEX.md", "parent_record_id": "deepseek-1017-tensorfold-spark", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "runs/2026-10-18/INDEX.md", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md", "document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "checked_on": "2026-10-18" }, { "document_id": "ds1017-runs-2026-10-17-01_vllm-miaai-kit_suite-results_32of32.json", "parent_record_id": "deepseek-1017-tensorfold-spark", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/01_vllm-miaai-kit_suite-results_32of32.json", "document_sha256": "0cb7c61b5b321ba0b9eec76460e4d9d3d0e759bd09cfb03fc747ef66e7c2ea63", "checked_on": "2026-10-18" }, { "document_id": "ds1017-runs-2026-10-17-11_tensorfold-fd2365042f56_suite-results_32of32.json", "parent_record_id": "deepseek-1017-tensorfold-spark", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-17/11_tensorfold-fd2365042f56_suite-results_32of32.json", "document_sha256": "d0ec8c99b0208ac4ddc272379d3cd71c796b5662d09481d6ba1e98fbbf0fa41c", "checked_on": "2026-10-18" }, { "document_id": "official-deepseek-ai--DeepSeek-V4.1-Flash-1017", "parent_record_id": "deepseek-1017-tensorfold-spark", "publisher": "deepseek-ai", "repository": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "document_path": "README.md", "pinned_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/blob/1bae0e1088993b9b82de08f59de5be5a34e42170/README.md", "document_sha256": "b72b796e4f4c739eca5d0ad360a7679b39118ca7eb64891f7151a527b292c42a", "checked_on": "2026-10-18" } ] } ``` --- # Qwen3.8-27B: Chisel replay on one DGX Spark A pinned single-Spark NVFP4/DFlash2 recipe with separate speed cohorts, task-level BF16 comparisons and negative non-inferiority flags. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: unsloth/Qwen3.8-27B-NVFP4 - Original author: yuanlin2004 - Original project: https://github.com/yuanlin2004/chisel-spark-recipes - Pinned document: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md - Source revision: 84feafddff1e834bd16b849c827eed11676e5934 - Source commit time (not experiment date): 2026-10-17T23:01:04Z - Checked: 2026-10-18 - Experiment date: 2026-10-15 source cohort; BF16 reference filed 2026-10-09 - Runtime topics: vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "1 x DGX Spark GB10; 128 GB unified memory; 2000 MHz cap in candidate timing cohort", "node_count": 1, "parallelism": "TP1", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-10-15 source cohort; BF16 reference filed 2026-10-09"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Chisel replay mixed-prompt median decode latency", "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L9", "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "thinking": true, "reasoning_effort": "medium", "greedy": true, "temperature": 0, "top_p": null, "concurrency": 1, "input_tokens": null, "max_output_tokens": 256, "output_tokens": null, "prompt_count": 25, "prompt_mix": { "chat": 9, "code": 9, "JSON_schema": 6, "long_story": 1 }, "repeats_per_session": 3, "fresh_start_sessions": 2, "measurement_denominator": "(last streamed token time - first) / (output tokens - 1); median latency, not whole-request served rate" } }, { "label": "Chisel replay mixed-prompt median TTFT", "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L10", "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "thinking": true, "reasoning_effort": "medium", "greedy": true, "temperature": 0, "top_p": null, "concurrency": 1, "input_tokens": null, "max_output_tokens": 256, "output_tokens": null, "prompt_count": 25, "prompt_mix": { "chat": 9, "code": 9, "JSON_schema": 6, "long_story": 1 }, "repeats_per_session": 3, "fresh_start_sessions": 2, "measurement_denominator": "request start to first streamed token" } }, { "label": "Local llama-benchy 16K C1", "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L11", "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "benchmark": "llama-benchy 0.4.0, local execution of spark-arena-v2 shape; not arena scores", "concurrency": 1, "input_tokens": 2048, "retained_context_tokens": 16384, "output_tokens": 128, "prefix_cache": true, "repeats_per_session": 5, "fresh_start_sessions": 2, "thinking": null, "measurement_denominator": "source local llama-benchy generation throughput; total across users for C2" } }, { "label": "Local llama-benchy 16K C2", "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L11", "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "benchmark": "llama-benchy 0.4.0, local execution of spark-arena-v2 shape; not arena scores", "concurrency": 2, "input_tokens": 2048, "retained_context_tokens": 16384, "output_tokens": 128, "prefix_cache": true, "repeats_per_session": 5, "fresh_start_sessions": 2, "thinking": null, "measurement_denominator": "source local llama-benchy generation throughput; total across users for C2" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - The recipe is a concrete deployment candidate, but speed and quality use different request shapes and thinking modes. The source custom quality bar passes while its non-inferiority flags are false; preserve both and leave the executed local image digest unknown. ## Limits - Source-reported only; no independent site model runs. - Document revision identifies the report, not the executed runtime or weight revision. - Complete measured configurations and missing fields must accompany extracted values. ## License declaration {"declared_spdx": "Apache-2.0", "scope": "Repository metadata only; model conversion, base weights, plugin and engine terms retain their own scope."} ## Editorial answer The recipe is a concrete deployment candidate, but speed and quality use different request shapes and thinking modes. The source custom quality bar passes while its non-inferiority flags are false; preserve both and leave the executed local image digest unknown. By Run AI Here editorial · 2026-10-18 Local Spark Qwen draft, source-audited and edited by Run AI Here with Codex; no independent model execution. ## What is pinned and what remains unknown? The recipe names unsloth Qwen3.8-27B NVFP4, ten DFlash2 draft tokens, bf16 GDN state and chisel-passes 0.1.0. The measured target revision is only a short prefix. A recipe base-image digest is available, but the main tests used a local build and the saved server.image is empty; the executed image digest remains unknown. Sources: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/ENVIRONMENT.md ## Which speed cohort is being quoted? The mixed-prompt test reports median decode latency and TTFT with thinking on and medium effort, one request at a time, up to 256 output tokens and two fresh-start sessions. Local llama-benchy results instead use 16K retained context, 128 output tokens and prefix cache on; its two-user throughput is aggregate, not per user. Sources: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/METHOD.md ## How should the quality gate be read? Quality tests use thinking off and compare four tasks against a separately recorded BF16 reference. The custom bar passes, but all four non_inferior flags and the overall all_non_inferior flag are false. Preserve both gates, the raw scores, sample counts, paired differences and intervals. These results do not establish equivalence or general quality preservation. Sources: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json ## How were the intervals computed? The comparison code uses matched per-question differences and a normal approximation: mean difference plus or minus 1.96 sample standard errors. It is not a bootstrap interval. The non-inferiority flag requires the lower bound above a loss of one point, while the author custom bar uses score recovery, a clearly-worse check and answer-length limits. Sources: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/tools/compare_quality.py ## What should an operator verify next? The configured 262144 context cap and four-sequence capacity are launch settings. They do not establish simultaneous near-limit fit. Verify the exact quantized artifact, drafter, plugin and image, then use representative requests and correctness tasks. Do not transfer candidate-specific image, clock or plugin settings to the BF16 reference when they were not reported. Sources: https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.yaml ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Chisel replay mixed-prompt median decode latency", "metric": "decode_latency", "value": 15.35, "unit": "ms/token", "statistic": "source-reported median over two fresh-start sessions", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "thinking": true, "reasoning_effort": "medium", "greedy": true, "temperature": 0, "top_p": null, "concurrency": 1, "input_tokens": null, "max_output_tokens": 256, "output_tokens": null, "prompt_count": 25, "prompt_mix": { "chat": 9, "code": 9, "JSON_schema": 6, "long_story": 1 }, "repeats_per_session": 3, "fresh_start_sessions": 2, "measurement_denominator": "(last streamed token time - first) / (output tokens - 1); median latency, not whole-request served rate" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L9", "source_document_sha256": "5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321", "source_document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.md", "source_line": 9 }, { "label": "Chisel replay mixed-prompt median TTFT", "metric": "ttft", "value": 241, "unit": "ms", "statistic": "source-reported median on the same prompt cohort", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "thinking": true, "reasoning_effort": "medium", "greedy": true, "temperature": 0, "top_p": null, "concurrency": 1, "input_tokens": null, "max_output_tokens": 256, "output_tokens": null, "prompt_count": 25, "prompt_mix": { "chat": 9, "code": 9, "JSON_schema": 6, "long_story": 1 }, "repeats_per_session": 3, "fresh_start_sessions": 2, "measurement_denominator": "request start to first streamed token" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L10", "source_document_sha256": "5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321", "source_document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.md", "source_line": 10 }, { "label": "Local llama-benchy 16K C1", "metric": "generation_throughput", "value": 30.0, "unit": "tokens/s", "statistic": "source-reported summary; separate from mixed-prompt median latency", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "benchmark": "llama-benchy 0.4.0, local execution of spark-arena-v2 shape; not arena scores", "concurrency": 1, "input_tokens": 2048, "retained_context_tokens": 16384, "output_tokens": 128, "prefix_cache": true, "repeats_per_session": 5, "fresh_start_sessions": 2, "thinking": null, "measurement_denominator": "source local llama-benchy generation throughput; total across users for C2" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L11", "source_document_sha256": "5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321", "source_document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.md", "source_line": 11 }, { "label": "Local llama-benchy 16K C2", "metric": "generation_throughput", "value": 58.0, "unit": "tokens/s", "statistic": "source-reported summary; separate from mixed-prompt median latency", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "2026-10-15 source cohort; some recorded sessions cross UTC midnight", "benchmark": "llama-benchy 0.4.0, local execution of spark-arena-v2 shape; not arena scores", "concurrency": 2, "input_tokens": 2048, "retained_context_tokens": 16384, "output_tokens": 128, "prefix_cache": true, "repeats_per_session": 5, "fresh_start_sessions": 2, "thinking": null, "measurement_denominator": "source local llama-benchy generation throughput; total across users for C2" }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.md#L11", "source_document_sha256": "5da566bc197bf4ba29c7c4d343b3c0362577de023425369a1a6ea9e53de95321", "source_document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.md", "source_line": 11 }, { "label": "gsm8k BF16 reference", "metric": "task_score", "value": 90.06823351023503, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "Qwen/Qwen3.8-27B", "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reported_model_revision_prefix": null, "system": null, "node_count": null, "driver": null, "GPU_clock_cap_MHz": null, "engine": null, "executed_engine_version": null, "runtime_execution_revision": null, "recipe_base_image_digest": null, "executed_local_image_digest": null, "executed_image_boundary": "Reference model and bf16 KV are reported; candidate-specific image, plugin and clock settings are not imputed to this arm.", "plugin": null, "draft_model": null, "draft_weight_sha256": null, "draft_tokens": null, "GDN_state_dtype": null, "KV_dtype": null, "configured_context_limit_tokens": null, "configured_max_requests": null, "GPU_memory_utilization": null, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "BF16 reference", "task": "gsm8k", "n": 1319, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "5-shot accuracy", "paired_difference_normal_CI95_points": [ -1.8137451343135735, 0.6007049523575464 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L8", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 8 }, { "label": "gsm8k NVFP4 replay", "metric": "task_score", "value": 89.46171341925702, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "NVFP4 replay", "task": "gsm8k", "n": 1319, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "5-shot accuracy", "paired_difference_normal_CI95_points": [ -1.8137451343135735, 0.6007049523575464 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L9", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 9 }, { "label": "ifeval BF16 reference", "metric": "task_score", "value": 80.96118299445472, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "Qwen/Qwen3.8-27B", "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reported_model_revision_prefix": null, "system": null, "node_count": null, "driver": null, "GPU_clock_cap_MHz": null, "engine": null, "executed_engine_version": null, "runtime_execution_revision": null, "recipe_base_image_digest": null, "executed_local_image_digest": null, "executed_image_boundary": "Reference model and bf16 KV are reported; candidate-specific image, plugin and clock settings are not imputed to this arm.", "plugin": null, "draft_model": null, "draft_weight_sha256": null, "draft_tokens": null, "GDN_state_dtype": null, "KV_dtype": null, "configured_context_limit_tokens": null, "configured_max_requests": null, "GPU_memory_utilization": null, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "BF16 reference", "task": "ifeval", "n": 541, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "prompt-level strict", "paired_difference_normal_CI95_points": [ -1.560611007417151, 3.409039842906985 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L26", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 26 }, { "label": "ifeval NVFP4 replay", "metric": "task_score", "value": 81.88539741219964, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "NVFP4 replay", "task": "ifeval", "n": 541, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "prompt-level strict", "paired_difference_normal_CI95_points": [ -1.560611007417151, 3.409039842906985 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L27", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 27 }, { "label": "humaneval_chat BF16 reference", "metric": "task_score", "value": 95.1219512195122, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "Qwen/Qwen3.8-27B", "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reported_model_revision_prefix": null, "system": null, "node_count": null, "driver": null, "GPU_clock_cap_MHz": null, "engine": null, "executed_engine_version": null, "runtime_execution_revision": null, "recipe_base_image_digest": null, "executed_local_image_digest": null, "executed_image_boundary": "Reference model and bf16 KV are reported; candidate-specific image, plugin and clock settings are not imputed to this arm.", "plugin": null, "draft_model": null, "draft_weight_sha256": null, "draft_tokens": null, "GDN_state_dtype": null, "KV_dtype": null, "configured_context_limit_tokens": null, "configured_max_requests": null, "GPU_memory_utilization": null, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "BF16 reference", "task": "humaneval_chat", "n": 164, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "chat pass@1", "paired_difference_normal_CI95_points": [ -2.6839968720184646, 1.4644846768965132 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L44", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 44 }, { "label": "humaneval_chat NVFP4 replay", "metric": "task_score", "value": 94.51219512195121, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "NVFP4 replay", "task": "humaneval_chat", "n": 164, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "chat pass@1", "paired_difference_normal_CI95_points": [ -2.6839968720184646, 1.4644846768965132 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L45", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 45 }, { "label": "longbench_narrativeqa BF16 reference", "metric": "task_score", "value": 34.83234254995091, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "Qwen/Qwen3.8-27B", "model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reported_model_revision_prefix": null, "system": null, "node_count": null, "driver": null, "GPU_clock_cap_MHz": null, "engine": null, "executed_engine_version": null, "runtime_execution_revision": null, "recipe_base_image_digest": null, "executed_local_image_digest": null, "executed_image_boundary": "Reference model and bf16 KV are reported; candidate-specific image, plugin and clock settings are not imputed to this arm.", "plugin": null, "draft_model": null, "draft_weight_sha256": null, "draft_tokens": null, "GDN_state_dtype": null, "KV_dtype": null, "configured_context_limit_tokens": null, "configured_max_requests": null, "GPU_memory_utilization": null, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "BF16 reference", "task": "longbench_narrativeqa", "n": 200, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "F1, full stories", "paired_difference_normal_CI95_points": [ -1.6421295670363527, 2.4483306144283086 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L62", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 62 }, { "label": "longbench_narrativeqa NVFP4 replay", "metric": "task_score", "value": 35.235443073646884, "unit": "score points", "statistic": "source paired-comparison raw score; retain task-specific definition", "uncertainty": null, "conditions": { "model": "Qwen3.8-27B", "model_artifact": "unsloth/Qwen3.8-27B-NVFP4", "model_revision": null, "reported_model_revision_prefix": "f0b7c9e7", "system": "1 x DGX Spark GB10 sm_121; 128 GB unified memory", "node_count": 1, "driver": "580.178.04", "GPU_clock_cap_MHz": 2000, "engine": "vLLM", "executed_engine_version": "0.31.1rc1.dev380", "runtime_execution_revision": null, "recipe_base_image_digest": "sha256:b77010699ec42ab88b3409f4dd101f2288dfd8895e2778e5bb737e03a20953f0", "executed_local_image_digest": null, "executed_image_boundary": "benchmark and quality used a local build with the same plugin code preinstalled; saved server.image is []; do not substitute the recipe base digest as the executed image digest", "plugin": "chisel-passes 0.1.0", "draft_model": "yuanlin2004/Qwen3.8-27B-Chisel-DFlash2-NVFP4-drafter", "draft_weight_sha256": "8d464c9f597c41a2e8917c585454239be14f254592f114675fd9d5d5dbc92f4c", "draft_tokens": 10, "GDN_state_dtype": "bfloat16", "KV_dtype": null, "configured_context_limit_tokens": 262144, "configured_max_requests": 4, "GPU_memory_utilization": 0.6, "experiment_date": "reference cohort 2026-10-09; candidate cohort filed 2026-10-15; compare file retains earlier original paths", "quality_arm": "NVFP4 replay", "task": "longbench_narrativeqa", "n": 200, "thinking": false, "greedy": true, "concurrency": null, "input_tokens": null, "output_tokens": null, "benchmark": "lm-evaluation-harness 0.4.13; chat template; full task", "reference_model": "Qwen/Qwen3.8-27B", "reference_model_revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "reference_weight_precision": "bfloat16", "reference_KV_dtype": "bfloat16", "metric_definition": "F1, full stories", "paired_difference_normal_CI95_points": [ -1.6421295670363527, 2.4483306144283086 ], "non_inferior": false, "author_custom_bar_passed": true }, "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L63", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 63 } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "chisel-recipes-qwen3.8-27b-chisel-nvfp4-dflash2-replay.yaml", "parent_record_id": "qwen38-chisel-dflash-replay", "publisher": "yuanlin2004", "repository": "yuanlin2004/chisel-spark-recipes", "revision": "84feafddff1e834bd16b849c827eed11676e5934", "document_path": "recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.yaml", "pinned_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/recipes/qwen3.8-27b/chisel-nvfp4-dflash2-replay.yaml", "document_sha256": "028f2badf76407546be67b8bd93e2e81aae8ad5d00076e20b8bd8835021a1ede", "checked_on": "2026-10-18" }, { "document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-ENVIRONMENT.md", "parent_record_id": "qwen38-chisel-dflash-replay", "publisher": "yuanlin2004", "repository": "yuanlin2004/chisel-spark-recipes", "revision": "84feafddff1e834bd16b849c827eed11676e5934", "document_path": "results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/ENVIRONMENT.md", "pinned_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/ENVIRONMENT.md", "document_sha256": "349c8de9b51a1df3ebba677db79d4e62c9a652a6dc225965f7515b80583d471a", "checked_on": "2026-10-18" }, { "document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-METHOD.md", "parent_record_id": "qwen38-chisel-dflash-replay", "publisher": "yuanlin2004", "repository": "yuanlin2004/chisel-spark-recipes", "revision": "84feafddff1e834bd16b849c827eed11676e5934", "document_path": "results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/METHOD.md", "pinned_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/METHOD.md", "document_sha256": "072fb07c8e46bb6e1e7a4c8a9416b94a87863cfc60dfde60f127e9243363dd8e", "checked_on": "2026-10-18" }, { "document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-server.image", "parent_record_id": "qwen38-chisel-dflash-replay", "publisher": "yuanlin2004", "repository": "yuanlin2004/chisel-spark-recipes", "revision": "84feafddff1e834bd16b849c827eed11676e5934", "document_path": "results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/server.image", "pinned_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/server.image", "document_sha256": "37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570", "checked_on": "2026-10-18" }, { "document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "parent_record_id": "qwen38-chisel-dflash-replay", "publisher": "yuanlin2004", "repository": "yuanlin2004/chisel-spark-recipes", "revision": "84feafddff1e834bd16b849c827eed11676e5934", "document_path": "results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json", "pinned_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json", "document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "checked_on": "2026-10-18" }, { "document_id": "chisel-tools-compare-quality", "parent_record_id": "qwen38-chisel-dflash-replay", "publisher": "yuanlin2004", "repository": "yuanlin2004/chisel-spark-recipes", "revision": "84feafddff1e834bd16b849c827eed11676e5934", "document_path": "tools/compare_quality.py", "pinned_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/tools/compare_quality.py", "document_sha256": "ee52180fce64fa6948d7516ac42f7f5fd0ef143584d2c284d6aa19f6fdecb783", "checked_on": "2026-10-18" }, { "document_id": "official-Qwen--Qwen3.8-27B", "parent_record_id": "qwen38-chisel-dflash-replay", "publisher": "Qwen", "repository": "Qwen/Qwen3.8-27B", "revision": "1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", "document_path": "README.md", "pinned_url": "https://huggingface.co/Qwen/Qwen3.8-27B/blob/1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0/README.md", "document_sha256": "57e4bdb258ee1a7d2635c5174ebd4e56abe392505cdb5f8bbb356b0dc4293641", "checked_on": "2026-10-18" } ] } ``` ## Paired source quality comparisons ```json { "quality_comparisons": [ { "task": "gsm8k", "n": 1319, "reference_score": 90.06823351023503, "candidate_score": 89.46171341925702, "diff_points": -0.6065200909780136, "ci95_points": [ -1.8137451343135735, 0.6007049523575464 ], "non_inferior": false, "length_ratio": 1.0086485431409682, "flips_right_to_wrong": 37, "flips_wrong_to_right": 29, "recovery": 0.9932659932659933, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L6", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 6 }, { "task": "ifeval", "n": 541, "reference_score": 80.96118299445472, "candidate_score": 81.88539741219964, "diff_points": 0.9242144177449169, "ci95_points": [ -1.560611007417151, 3.409039842906985 ], "non_inferior": false, "length_ratio": 1.001384198457369, "flips_right_to_wrong": 21, "flips_wrong_to_right": 26, "recovery": 1.0114155251141552, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L24", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 24 }, { "task": "humaneval_chat", "n": 164, "reference_score": 95.1219512195122, "candidate_score": 94.51219512195121, "diff_points": -0.6097560975609756, "ci95_points": [ -2.6839968720184646, 1.4644846768965132 ], "non_inferior": false, "length_ratio": 1.0066489055410106, "flips_right_to_wrong": 2, "flips_wrong_to_right": 1, "recovery": 0.9935897435897434, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L42", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 42 }, { "task": "longbench_narrativeqa", "n": 200, "reference_score": 34.83234254995091, "candidate_score": 35.235443073646884, "diff_points": 0.403100523695978, "ci95_points": [ -1.6421295670363527, 2.4483306144283086 ], "non_inferior": false, "length_ratio": 0.9618899475014583, "recovery": 1.011572593003698, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L60", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 60 } ] } ``` ## quality comparisons ```json { "quality_comparisons": [ { "task": "gsm8k", "n": 1319, "reference_score": 90.06823351023503, "candidate_score": 89.46171341925702, "diff_points": -0.6065200909780136, "ci95_points": [ -1.8137451343135735, 0.6007049523575464 ], "non_inferior": false, "length_ratio": 1.0086485431409682, "flips_right_to_wrong": 37, "flips_wrong_to_right": 29, "recovery": 0.9932659932659933, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L6", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 6 }, { "task": "ifeval", "n": 541, "reference_score": 80.96118299445472, "candidate_score": 81.88539741219964, "diff_points": 0.9242144177449169, "ci95_points": [ -1.560611007417151, 3.409039842906985 ], "non_inferior": false, "length_ratio": 1.001384198457369, "flips_right_to_wrong": 21, "flips_wrong_to_right": 26, "recovery": 1.0114155251141552, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L24", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 24 }, { "task": "humaneval_chat", "n": 164, "reference_score": 95.1219512195122, "candidate_score": 94.51219512195121, "diff_points": -0.6097560975609756, "ci95_points": [ -2.6839968720184646, 1.4644846768965132 ], "non_inferior": false, "length_ratio": 1.0066489055410106, "flips_right_to_wrong": 2, "flips_wrong_to_right": 1, "recovery": 0.9935897435897434, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L42", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 42 }, { "task": "longbench_narrativeqa", "n": 200, "reference_score": 34.83234254995091, "candidate_score": 35.235443073646884, "diff_points": 0.403100523695978, "ci95_points": [ -1.6421295670363527, 2.4483306144283086 ], "non_inferior": false, "length_ratio": 0.9618899475014583, "recovery": 1.011572593003698, "clearly_worse": false, "length_ok": true, "passes": true, "interval_method": "paired per-question differences; mean +/- 1.96 * sample SD / sqrt(n); normal approximation, not bootstrap", "evidence_state": "source_reported", "site_reproduced": false, "evidence_url": "https://github.com/yuanlin2004/chisel-spark-recipes/blob/84feafddff1e834bd16b849c827eed11676e5934/results/qwen3.8-27b/chisel-nvfp4-dflash2-replay/2026-10-15/quality/compare-vs-bf16.json#L60", "source_document_sha256": "46aec52dd7d9d842b58c710904f6af19960f1196a7f05f2c834f4532481121fb", "source_document_id": "chisel-results-qwen3.8-27b-chisel-nvfp4-dflash2-replay-2026-10-15-quality-compare-vs-bf16.json", "source_line": 60 } ] } ``` --- # Qwen Flash-Next: single-Spark memory and cold context The v2.4.0 promotion arm serves the pinned GDN-MSE checkpoint, with an extra MXFP8 prefill shard. NVFP4 expert weights, MXFP8 paths and a refitted four-draft MTP head are separate components. The experimental image tag is retained; the later release-image digest does not prove the exact cold-test image digest. Repository release names are specific to this one-Spark recipe. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE - Original author: ursuciprian - Original project: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark - Pinned document: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/README.md - Source revision: 348ede7e28a01f6aa6b501aaf2392d20a286a012 - Source commit time (not experiment date): 2026-10-16T16:43:24Z - Checked: 2026-10-18 - Experiment date: 2026-10-16 - Runtime topics: vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "kv_dtype": "fp8", "context_limit_tokens": 262144, "experiment_date": "2026-10-16"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "1x v2.4.0 16384-token ttft", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1262", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 16384, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" } }, { "label": "1x v2.4.0 16384-token prefill_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1263", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 16384, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" } }, { "label": "1x v2.4.0 65536-token ttft", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1264", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 65536, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" } }, { "label": "1x v2.4.0 65536-token prefill_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1265", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 65536, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" } }, { "label": "1x v2.4.0 131072-token ttft", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1266", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 131072, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" } }, { "label": "1x v2.4.0 131072-token prefill_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1267", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 131072, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" } }, { "label": "1x v2.4.0 allocated KV pool", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/BENCHMARKS.md#L106", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": null, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": null, "thinking": null, "output_tokens": null, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": null, "memory_kind": "allocated_KV_token_pool_not_measured_peak_RAM", "max_context_simultaneously_validated": false } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document, not an independent experiment count. Preserve measurement type and unknown fields. ## Limits - No independent Run AI Here model execution. - Document revision is distinct from execution and weight identity. - File bytes, host RSS, anonymous memory, MemAvailable, KV allocation and configured context are different quantities. ## License declaration {"declared_spdx": "apache-2.0", "scope": "Source declaration only; base weights, converters, runtimes and source integration material retain their own terms."} ## Editorial answer The v2.4.0 promotion arm serves the pinned GDN-MSE checkpoint, with an extra MXFP8 prefill shard. NVFP4 expert weights, MXFP8 paths and a refitted four-draft MTP head are separate components. The experimental image tag is retained; the later release-image digest does not prove the exact cold-test image digest. Repository release names are specific to this one-Spark recipe. By Run AI Here editorial · 2026-10-18 Original source-grounded editorial synthesis with local Spark Qwen assistance and Codex review; no site model benchmark. ## Which quantization and build were tested? The v2.4.0 promotion arm serves the pinned GDN-MSE checkpoint, with an extra MXFP8 prefill shard. NVFP4 expert weights, MXFP8 paths and a refitted four-draft MTP head are separate components. The experimental image tag is retained; the later release-image digest does not prove the exact cold-test image digest. Repository release names are specific to this one-Spark recipe. Sources: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/VERSIONS.md, https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/arm-p1/recipe.yaml ## What does the long-prompt curve measure? The selected cold-prefix observations cover 16384, 65536 and 131072 input tokens at C1, temperature zero and one requested output token. Values are means of four boots with three cold prompts each; reported SD is between boot means. A cold prefix does not establish cold operating-system page cache. These are prefill/TTFT observations, not long-response decode or broad reasoning accuracy. Sources: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv, https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/BENCHMARKS.md ## How should memory limits be read? The recipe allocates a 14 GiB FP8 KV pool and reports 993754 token capacity, while its request window is 262144 and scheduler ceiling is eight requests. The token pool is shared: these numbers do not demonstrate eight simultaneous full windows. PLE uses NVMe mmap and page cache. The 0.01 GPU-memory-utilization setting is a startup guard with explicit KV bytes, not measured one-percent memory use. Sources: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/README.md, https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/arm-p1/recipe.yaml ## How should two-Spark comparisons be selected? The project numbers releases independently for one-Spark and two-Spark setups. Its capability table also combines dated measurements when a newer build has no corresponding cell. Keep setup, release, workload and original run path together instead of treating every headline cell as one matched scaling experiment. Sources: https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/VERSIONS.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "1x v2.4.0 16384-token ttft", "metric": "ttft", "value": 7.104, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 0.016 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 16384, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1262, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1262" }, { "label": "1x v2.4.0 16384-token prefill_rate", "metric": "prefill_rate", "value": 2306.4, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 5.2 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 16384, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1263, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1263" }, { "label": "1x v2.4.0 65536-token ttft", "metric": "ttft", "value": 30.723, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 0.06 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 65536, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1264, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1264" }, { "label": "1x v2.4.0 65536-token prefill_rate", "metric": "prefill_rate", "value": 2133.2, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 4.2 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 65536, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1265, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1265" }, { "label": "1x v2.4.0 131072-token ttft", "metric": "ttft", "value": 70.0, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 0.137 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 131072, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "ttft_s", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1266, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1266" }, { "label": "1x v2.4.0 131072-token prefill_rate", "metric": "prefill_rate", "value": 1872.5, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "kind": "SD between boot means, not confidence interval", "standard_deviation": 3.7 }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": 1, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": 0, "thinking": null, "output_tokens": 1, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": 131072, "repetitions": "4 boots x 3 cold prompts; SD between boots", "harness": "cold.py (k101)", "source_run_directory": "1x:results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/", "source_metric": "pp", "workload": "long-prompt" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-data-capability.csv", "source_document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "source_line": 1267, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv#L1267" }, { "label": "1x v2.4.0 allocated KV pool", "metric": "kv_pool_tokens", "value": 993754, "unit": "tokens", "statistic": "boot allocation observed on every reported boot", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_artifact": "local-inference-lab/Qwen3.8-Flash-Next-NVFP4", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "weight_precision": "NVFP4 experts; MXFP8 dense/attention; GDN NVFP4 decode and MXFP8 prefill", "system": "one DGX Spark GB10 with 128 GB unified memory, local NVMe", "node_count": 1, "parallelism": "TP1", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "runtime_revision_prefix": "10d0acc8 files over 5dad364d", "b12x_revision_prefix": "a1ad1b69", "executed_image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "release_image_is_exact_cold_test_image": false, "recipe_release": "1x v2.4.0 promotion A/B; k103 cb test arm, not the release-image ship check", "experiment_date": "2026-10-16", "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_tokens": 4, "kv_dtype": "fp8", "GPU_memory_utilization_startup_guard": 0.01, "configured_KV_pool_bytes": 15032385536, "PLE_backing": "NVMe mmap, page cache; reported 26.8 GiB PLE table", "concurrency": null, "cache_state": "cold prefix, no cache hit; not a claimed cold OS page cache", "temperature": null, "thinking": null, "output_tokens": null, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "input_tokens": null, "memory_kind": "allocated_KV_token_pool_not_measured_peak_RAM", "max_context_simultaneously_validated": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen1x-docs-BENCHMARKS.md", "source_document_sha256": "ac2128626b607b2e4571bdb45fb7e2639e5c65c83841a8f28367bb5692986fc0", "source_line": 106, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/BENCHMARKS.md#L106" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "qwen1x-docs-data-capability.csv", "parent_record_id": "qwen38-flash-next-single-spark-context", "publisher": "ursuciprian", "repository": "ursuciprian/qwen3.8-flash-next-1x-dgx-spark", "revision": "348ede7e28a01f6aa6b501aaf2392d20a286a012", "document_path": "docs/data/capability.csv", "pinned_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/data/capability.csv", "document_sha256": "4376de2ea68ca23d5208731ce948a28801cd1da2dc026c9415b2b122e9339205", "checked_on": "2026-10-18" }, { "document_id": "qwen1x-docs-BENCHMARKS.md", "parent_record_id": "qwen38-flash-next-single-spark-context", "publisher": "ursuciprian", "repository": "ursuciprian/qwen3.8-flash-next-1x-dgx-spark", "revision": "348ede7e28a01f6aa6b501aaf2392d20a286a012", "document_path": "docs/BENCHMARKS.md", "pinned_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/docs/BENCHMARKS.md", "document_sha256": "ac2128626b607b2e4571bdb45fb7e2639e5c65c83841a8f28367bb5692986fc0", "checked_on": "2026-10-18" }, { "document_id": "qwen1x-VERSIONS.md", "parent_record_id": "qwen38-flash-next-single-spark-context", "publisher": "ursuciprian", "repository": "ursuciprian/qwen3.8-flash-next-1x-dgx-spark", "revision": "348ede7e28a01f6aa6b501aaf2392d20a286a012", "document_path": "VERSIONS.md", "pinned_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/VERSIONS.md", "document_sha256": "c513523883d95b716707697a0633fb9c5e63c9cf748aeaf37a38d4f771913abd", "checked_on": "2026-10-18" }, { "document_id": "qwen1x-results-k103-1x-prefill-combo-ab-20261016-1206-cb-01-arm-p1-recipe.yaml", "parent_record_id": "qwen38-flash-next-single-spark-context", "publisher": "ursuciprian", "repository": "ursuciprian/qwen3.8-flash-next-1x-dgx-spark", "revision": "348ede7e28a01f6aa6b501aaf2392d20a286a012", "document_path": "results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/arm-p1/recipe.yaml", "pinned_url": "https://github.com/ursuciprian/qwen3.8-flash-next-1x-dgx-spark/blob/348ede7e28a01f6aa6b501aaf2392d20a286a012/results/k103-1x-prefill-combo-ab-20261016-1206/cb-01/arm-p1/recipe.yaml", "document_sha256": "33ecbeea74e85987bd4dfa7418543438e98ff93ea2270c103deccab44a43fa7e", "checked_on": "2026-10-18" } ] } ``` ## memory observations ```json { "memory_observations": [ { "id": "qwen-1x-v2.4-cold", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "checkpoint_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "engine": "vLLM + b12x", "image_tag": "ghcr.io/ursuciprian/spark-vllm-b12x:k103-20261015-a1ad1b69-10d0acc8", "executed_image_digest": null, "release_image_digest": "sha256:30fe678ed7ae5d4deb277be137497bcf597e46e829471b7bed24edd4294a84f1", "context_limit_tokens": 262144, "measured_cold_input_tokens": [ 16384, 65536, 131072 ], "configured_max_requests": 8, "KV_pool_bytes": 15032385536, "KV_pool_tokens": 993754, "MTP_draft_tokens": 4, "PLE_file_reported_GiB": 26.8, "all_eight_full_window_requests_proven": false, "cold_prefix_is_not_cold_page_cache": true } ] } ``` --- # Qwen Flash-Next: quantized PLE and memory residence This experiment changes the n-gram PLE table backing, rather than quantizing every model component. The source compares a BF16 table held in RAM with file-mapped sidecars on one 96 GB RTX PRO 6000 and a 176 GB host. Main-table throughput uses 8K input, 512 requested output tokens and no prefix-cache hits. Full checkpoint revision and executed image digest are not supplied. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 - Original author: primitive-ai - Original project: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant - Pinned document: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md - Source revision: a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 - Source commit time (not experiment date): 2026-09-14T15:58:31.000Z - Checked: 2026-10-18 - Experiment date: None - Runtime topics: vLLM - Hardware topics: nvidia - Reported setup: {"system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": 32768, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "BF16 in RAM table size", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L96", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "BF16 in RAM", "memory_kind": "PLE_artifact_size", "memory_backing": "in RAM" } }, { "label": "BF16 in RAM host RSS", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L96", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "BF16 in RAM", "memory_kind": "host_RSS_not_necessarily_anonymous", "memory_backing": "in RAM" } }, { "label": "INT4 group-16 mmap table size", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L98", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16 mmap", "memory_kind": "PLE_artifact_size", "memory_backing": "reclaimable mapped file pages" } }, { "label": "INT4 group-16 mmap host RSS", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L98", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16 mmap", "memory_kind": "host_RSS_not_necessarily_anonymous", "memory_backing": "reclaimable mapped file pages" } }, { "label": "mmap cold first request", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L68", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": 1, "cache_state": "page cache dropped before each arm", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": null, "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "workload": "first request, exact request shape unreported; optional resident-vs-mmap cohort, not main 8K/512 table", "memory_backing": "NVMe mmap" } }, { "label": "resident cold first request", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L69", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": 1, "cache_state": "page cache dropped before each arm", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": null, "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "workload": "first request, exact request shape unreported; optional resident-vs-mmap cohort, not main 8K/512 table", "memory_backing": "anonymous resident copy" } }, { "label": "INT4 resident sidecar anonymous allocation", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L81", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "anonymous_resident_sidecar_not_total_RSS" } }, { "label": "optional residence cohort KV pool, MTP 0", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L80", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "allocated_KV_token_pool_not_measured_RAM" } }, { "label": "optional residence cohort KV pool, MTP 3", "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L80", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 3, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "allocated_KV_token_pool_not_measured_RAM" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document, not an independent experiment count. Preserve measurement type and unknown fields. ## Limits - No independent Run AI Here model execution. - Document revision is distinct from execution and weight identity. - File bytes, host RSS, anonymous memory, MemAvailable, KV allocation and configured context are different quantities. ## License declaration {"declared_spdx": "apache-2.0", "scope": "Source declaration only; base weights, converters, runtimes and source integration material retain their own terms."} ## Editorial answer This experiment changes the n-gram PLE table backing, rather than quantizing every model component. The source compares a BF16 table held in RAM with file-mapped sidecars on one 96 GB RTX PRO 6000 and a 176 GB host. Main-table throughput uses 8K input, 512 requested output tokens and no prefix-cache hits. Full checkpoint revision and executed image digest are not supplied. By Run AI Here editorial · 2026-10-18 Original source-grounded editorial synthesis with local Spark Qwen assistance and Codex review; no site model benchmark. ## Which allocation is reduced? This experiment changes the n-gram PLE table backing, rather than quantizing every model component. The source compares a BF16 table held in RAM with file-mapped sidecars on one 96 GB RTX PRO 6000 and a 176 GB host. Main-table throughput uses 8K input, 512 requested output tokens and no prefix-cache hits. Full checkpoint revision and executed image digest are not supplied. Sources: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md, https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/ples_int4/META.json ## Why is RSS different from anonymous memory? The selected BF16 and INT4 rows retain artifact size separately from host RSS. File-mapped RSS includes reclaimable page-cache pages; it is not a permanent anonymous allocation. The optional INT4 resident copy instead adds a reported 29.8 GiB of anonymous sidecar memory, above the worker allocation. Source-labelled GB and GiB are not silently equated. Sources: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md ## What remains outside the evidence? The optional cold-residence cohort drops page cache before each arm and has a different, incompletely reported request shape. Its first-request speed is not a warm throughput average. MTP changes the allocated KV token pool; mmap versus resident does not change that pool within each reported MTP setting. Neither the table-size reduction nor suite-level sanity proves general quality equivalence. Sources: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md ## Does the native window fit every launch configuration? The source reports a native-window startup failure with MTP when 7.57 GiB of KV was required but 5.95 GiB remained. Its selected main table used a shorter 32768-token launch window, while a separate field report used 135168. These are configuration-specific observations, not a universal maximum or evidence that every format fits the native window. Sources: https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "BF16 in RAM table size", "metric": "artifact_size", "value": 95.4, "unit": "GB", "statistic": "source-reported PLE table size, not whole model or peak RAM", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "BF16 in RAM", "memory_kind": "PLE_artifact_size", "memory_backing": "in RAM" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 96, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L96" }, { "label": "BF16 in RAM host RSS", "metric": "host_rss", "value": 95, "unit": "GB", "statistic": "approximate source host RSS", "uncertainty": { "approximate": true }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "BF16 in RAM", "memory_kind": "host_RSS_not_necessarily_anonymous", "memory_backing": "in RAM" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 96, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L96" }, { "label": "INT4 group-16 mmap table size", "metric": "artifact_size", "value": 32, "unit": "GB", "statistic": "source-reported PLE table size, not whole model or peak RAM", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16 mmap", "memory_kind": "PLE_artifact_size", "memory_backing": "reclaimable mapped file pages" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 98, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L98" }, { "label": "INT4 group-16 mmap host RSS", "metric": "host_rss", "value": 32.9, "unit": "GB", "statistic": "source host RSS including reclaimable file pages", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": 32768, "configured_max_requests": 36, "input_tokens": 8192, "output_tokens": 512, "actual_output_tokens": null, "concurrency": 1, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": true, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16 mmap", "memory_kind": "host_RSS_not_necessarily_anonymous", "memory_backing": "reclaimable mapped file pages" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 98, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L98" }, { "label": "mmap cold first request", "metric": "decode_rate", "value": 19.0, "unit": "tokens/s", "statistic": "first-request observation; warm asymptote is a separate quantity", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": 1, "cache_state": "page cache dropped before each arm", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": null, "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "workload": "first request, exact request shape unreported; optional resident-vs-mmap cohort, not main 8K/512 table", "memory_backing": "NVMe mmap" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 68, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L68" }, { "label": "resident cold first request", "metric": "decode_rate", "value": 74.3, "unit": "tokens/s", "statistic": "first-request observation; warm asymptote is a separate quantity", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": 1, "cache_state": "page cache dropped before each arm", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": null, "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "workload": "first request, exact request shape unreported; optional resident-vs-mmap cohort, not main 8K/512 table", "memory_backing": "anonymous resident copy" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 69, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L69" }, { "label": "INT4 resident sidecar anonymous allocation", "metric": "anonymous_memory", "value": 29.8, "unit": "GiB", "statistic": "sidecar allocation only, on top of roughly 8 GiB held by worker", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": null, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "anonymous_resident_sidecar_not_total_RSS" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 81, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L81" }, { "label": "optional residence cohort KV pool, MTP 0", "metric": "kv_pool_tokens", "value": 426639, "unit": "tokens", "statistic": "reported identical mmap/resident allocation for this MTP setting", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 0, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "allocated_KV_token_pool_not_measured_RAM" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 80, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L80" }, { "label": "optional residence cohort KV pool, MTP 3", "metric": "kv_pool_tokens", "value": 134192, "unit": "tokens", "statistic": "reported identical mmap/resident allocation for this MTP setting", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8", "model_revision": null, "PLE_artifact": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "PLE_artifact_revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "system": "one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe", "node_count": 1, "engine": "vLLM with exact-image PLE overlay", "engine_version": null, "runtime_execution_revision": null, "image_tag": "vllm/vllm-openai:qwen38-flash-next", "executed_image_digest": null, "experiment_date": null, "source_update_boundary": "2026-09-06 corrections present; individual table trial dates not stated", "model_weight_precision": "mixed NVFP4/FP8 checkpoint; PLE format varies", "context_limit_tokens": null, "configured_max_requests": 36, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "concurrency": null, "cache_state": "prefix-cache-free; mmap page-cache residency not independently logged", "thinking": null, "temperature": null, "top_p": null, "draft_tokens": 3, "MTP_main_table_execution": null, "kv_dtype": null, "throughput_repetitions": "two seeds, a/b; not independent boots", "memory_sampling_instant": null, "reported_size_unit": "GB as labelled by source, byte convention not independently specified", "PLE_format": "INT4 group-16", "memory_kind": "allocated_KV_token_pool_not_measured_RAM" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwenple-readme", "source_document_sha256": "0085c022b212b241827a09f1a315170737f76298ca5af3bd8a46dea2032faab9", "source_line": 80, "evidence_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/README.md#L80" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "qwenple-ples_int4-META.json", "parent_record_id": "qwen38-flash-next-ple-memory", "publisher": "primitive-ai", "repository": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "document_path": "ples_int4/META.json", "pinned_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/ples_int4/META.json", "document_sha256": "e9a1863008450b7ac329d50a3b6b908f21122b8bf5da999bdeff05004161884a", "checked_on": "2026-10-18" }, { "document_id": "qwenple-ples_fp8-META.json", "parent_record_id": "qwen38-flash-next-ple-memory", "publisher": "primitive-ai", "repository": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "document_path": "ples_fp8/META.json", "pinned_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/ples_fp8/META.json", "document_sha256": "de757efa51030ac1e4edb6acdb29507521f8ace860a3be511661a26e5017b632", "checked_on": "2026-10-18" }, { "document_id": "qwenple-ples_nvfp4-META.json", "parent_record_id": "qwen38-flash-next-ple-memory", "publisher": "primitive-ai", "repository": "primitive-ai/Qwen3.8-Flash-Next-PLE-quant", "revision": "a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763", "document_path": "ples_nvfp4/META.json", "pinned_url": "https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-PLE-quant/blob/a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763/ples_nvfp4/META.json", "document_sha256": "5aa4d53703f67c8c664d674c95f16c29c8ce7a94a2d7d9ac27ae8afe4c0b56ef", "checked_on": "2026-10-18" } ] } ``` ## memory observations ```json { "memory_observations": [ { "record_id": "qwen38-flash-next-ple-memory", "kind": "source_main_table_host_RSS", "BF16_in_RAM_GB_approx": 95, "INT4_mmap_GB": 32.9, "mapped_pages_reclaimable": true, "anonymous_memory_inferred_from_RSS": false, "GB_byte_convention_verified": false }, { "record_id": "qwen38-flash-next-ple-memory", "kind": "optional_anonymous_residence", "INT4_sidecar_GiB": 29.8, "additional_worker_GiB_approx": 8, "same_cohort_as_main_table": false } ] } ``` --- # DeepSeek 1017: short-run EXL3 deployment and pool budget The four selected C1 decode observations come from the raw quick-matrix CSVs, with 520-token thinking-off or 546-token effort-75 prompts and 64 actual output tokens. Each is one pass without variance estimates. Their first-to-last streamed-delta rate excludes initial prefill; end-to-end wall throughput remains a different field. This is not a long-context or sustained-response experiment. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw - Original author: x00byte - Original project: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark - Pinned document: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/README.md - Source revision: e85ed7ec3a403d8820a83e5aa33abe0985b58896 - Source commit time (not experiment date): 2026-10-17T18:27:21Z - Checked: 2026-10-18 - Experiment date: None - Runtime topics: TensorFold, vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "kv_dtype": null, "context_limit_tokens": 262144, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "tensorfold C1 thinking off", "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv#L2", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "TensorFold Python 0.6.0 patched", "runtime_execution_revision": "c4646171139ee8a3c38103eaa1699dad226ec12b", "overlay_source_revision": "2dcce0303451da2a59f80e6c7413da8b3b0f7a57", "executed_image_digest": null, "thinking": false, "reasoning_effort": null, "input_tokens": 520, "actual_output_tokens": 64, "batch_wall_seconds": 2.4011132870218717, "end_to_end_output_wall_tps": 26.65430254620761, "TTFT_seconds": 1.313945789996069 } }, { "label": "tensorfold C1 thinking 75", "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv#L10", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "TensorFold Python 0.6.0 patched", "runtime_execution_revision": "c4646171139ee8a3c38103eaa1699dad226ec12b", "overlay_source_revision": "2dcce0303451da2a59f80e6c7413da8b3b0f7a57", "executed_image_digest": null, "thinking": true, "reasoning_effort": 75, "input_tokens": 546, "actual_output_tokens": 64, "batch_wall_seconds": 2.141774678020738, "end_to_end_output_wall_tps": 29.881761446140466, "TTFT_seconds": 1.326543439005036 } }, { "label": "vllm C1 thinking off", "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv#L2", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "vLLM + EXL3 overlay", "runtime_execution_revision": null, "overlay_source_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "executed_image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "thinking": false, "reasoning_effort": null, "input_tokens": 520, "actual_output_tokens": 64, "batch_wall_seconds": 2.5652991679962724, "end_to_end_output_wall_tps": 24.948357212460998, "TTFT_seconds": 0.8555283379973844 } }, { "label": "vllm C1 thinking 75", "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv#L10", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "vLLM + EXL3 overlay", "runtime_execution_revision": null, "overlay_source_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "executed_image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "thinking": true, "reasoning_effort": 75, "input_tokens": 546, "actual_output_tokens": 64, "batch_wall_seconds": 2.4199117650277913, "end_to_end_output_wall_tps": 26.447245277666145, "TTFT_seconds": 0.8985815939959139 } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document, not an independent experiment count. Preserve measurement type and unknown fields. ## Limits - No independent Run AI Here model execution. - Document revision is distinct from execution and weight identity. - File bytes, host RSS, anonymous memory, MemAvailable, KV allocation and configured context are different quantities. ## License declaration {"declared_spdx": null, "scope": "Source declaration only; base weights, converters, runtimes and source integration material retain their own terms."} ## Editorial answer The four selected C1 decode observations come from the raw quick-matrix CSVs, with 520-token thinking-off or 546-token effort-75 prompts and 64 actual output tokens. Each is one pass without variance estimates. Their first-to-last streamed-delta rate excludes initial prefill; end-to-end wall throughput remains a different field. This is not a long-context or sustained-response experiment. By Run AI Here editorial · 2026-10-18 Original source-grounded editorial synthesis with local Spark Qwen assistance and Codex review; no site model benchmark. ## What was measured? The four selected C1 decode observations come from the raw quick-matrix CSVs, with 520-token thinking-off or 546-token effort-75 prompts and 64 actual output tokens. Each is one pass without variance estimates. Their first-to-last streamed-delta rate excludes initial prefill; end-to-end wall throughput remains a different field. This is not a long-context or sustained-response experiment. Sources: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/benchmark-method.md, https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv, https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv ## Which files and memory controls belong to this profile? Both runtimes use the same pinned EXL3 conversion and separately pinned official 1017 Engrams. The TensorFold Python profile has eight request slots, a shared 524288-token pool, a 262144 per-request ceiling and configured 5/4 GiB memory floors. Floors are admission controls, not observed peak memory or a proof of full-window concurrency. Sources: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/manifest.json, https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/deployment.md, https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/tensorfold-knobs.env.example ## Can these rates be used for a VQ or million-token deployment? No such transfer is established. The runtime, quantized tensors, request length and cache policy must stay attached to each observation. The text matrix predates the explicit native-image-routing flag; a later red-image smoke check confirms that path only. No comprehensive image or model-quality benchmark is claimed. Sources: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/benchmark-method.md, https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/deployment.md ## How does storage placement affect the deployment description? The reported worker reads checkpoint files over a read-only NFS share, while active Engram lookups remain rank-local. The storage inventory includes separate source tables and packed files. Disk preparation, model residency and a shared KV budget must be recorded independently rather than adding all on-disk bytes into a claimed live memory requirement. Sources: https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/deployment.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "tensorfold C1 thinking off", "metric": "decode_rate", "value": 58.05409336065285, "unit": "tokens/s", "statistic": "one pass; no run-to-run uncertainty estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "TensorFold Python 0.6.0 patched", "runtime_execution_revision": "c4646171139ee8a3c38103eaa1699dad226ec12b", "overlay_source_revision": "2dcce0303451da2a59f80e6c7413da8b3b0f7a57", "executed_image_digest": null, "thinking": false, "reasoning_effort": null, "input_tokens": 520, "actual_output_tokens": 64, "batch_wall_seconds": 2.4011132870218717, "end_to_end_output_wall_tps": 26.65430254620761, "TTFT_seconds": 1.313945789996069 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsdual-results-tensorfold-quick.csv", "source_document_sha256": "3005b3be00581a962f9a269211eef2d598a6358d83c41e565ee1549b28e03c33", "source_line": 2, "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv#L2" }, { "label": "tensorfold C1 thinking 75", "metric": "decode_rate", "value": 77.49652509362433, "unit": "tokens/s", "statistic": "one pass; no run-to-run uncertainty estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "TensorFold Python 0.6.0 patched", "runtime_execution_revision": "c4646171139ee8a3c38103eaa1699dad226ec12b", "overlay_source_revision": "2dcce0303451da2a59f80e6c7413da8b3b0f7a57", "executed_image_digest": null, "thinking": true, "reasoning_effort": 75, "input_tokens": 546, "actual_output_tokens": 64, "batch_wall_seconds": 2.141774678020738, "end_to_end_output_wall_tps": 29.881761446140466, "TTFT_seconds": 1.326543439005036 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsdual-results-tensorfold-quick.csv", "source_document_sha256": "3005b3be00581a962f9a269211eef2d598a6358d83c41e565ee1549b28e03c33", "source_line": 10, "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv#L10" }, { "label": "vllm C1 thinking off", "metric": "decode_rate", "value": 36.8753802051545, "unit": "tokens/s", "statistic": "one pass; no run-to-run uncertainty estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "vLLM + EXL3 overlay", "runtime_execution_revision": null, "overlay_source_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "executed_image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "thinking": false, "reasoning_effort": null, "input_tokens": 520, "actual_output_tokens": 64, "batch_wall_seconds": 2.5652991679962724, "end_to_end_output_wall_tps": 24.948357212460998, "TTFT_seconds": 0.8555283379973844 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsdual-results-vllm-quick.csv", "source_document_sha256": "befbf211d4f143798e12053ec21291cc77e0893284ccdb8d5a0aa19ef4722c6e", "source_line": 2, "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv#L2" }, { "label": "vllm C1 thinking 75", "metric": "decode_rate", "value": 41.447290817386964, "unit": "tokens/s", "statistic": "one pass; no run-to-run uncertainty estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_artifact": "deepseek-ai/DeepSeek-V4.1-Flash-1017", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "Engram_shard_SHA256": { "model-00047-of-00048.safetensors": "76a05106ece259be83d1bfea0dcb54f23377864ee60fd321690f35e6698a2af5", "model-00048-of-00048.safetensors": "9dc51433e195bc78b5ad846b68643cc0351d3b174e5e4309222d3b7402c83afd" }, "weight_precision": "EXL3 2.9bpw; 6-bit dense MTP handling in vLLM overlay", "system": "two DGX Spark GB10, 128 GB unified memory each, TP2, both available RoCE links", "node_count": 2, "parallelism": "TP2", "experiment_date": null, "context_limit_tokens": 262144, "configured_max_requests": 8, "draft_depth_cap": 3, "concurrency": 1, "input_lines": 27, "output_tokens": 64, "max_output_tokens": 64, "repetitions": 1, "cache_state": "startup and earlier generation checks; no per-case warmup; TensorFold session/prepared caches disabled", "temperature": null, "top_p": null, "kv_dtype": null, "measurement_denominator": "(completion_tokens - 1) / first-to-last streamed delta interval; excludes initial prefill", "quality_evaluation": false, "long_response_steady_state_measured": false, "native_image_routing_in_text_matrix": false, "engine": "vLLM + EXL3 overlay", "runtime_execution_revision": null, "overlay_source_revision": "6f7d1590ad49a2b8995188e45d7b9db31e677452", "executed_image_digest": "sha256:2f0cf3adc0f989c1d446be274df864eb799630175f604c3b22b71b7205971dce", "thinking": true, "reasoning_effort": 75, "input_tokens": 546, "actual_output_tokens": 64, "batch_wall_seconds": 2.4199117650277913, "end_to_end_output_wall_tps": 26.447245277666145, "TTFT_seconds": 0.8985815939959139 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsdual-results-vllm-quick.csv", "source_document_sha256": "befbf211d4f143798e12053ec21291cc77e0893284ccdb8d5a0aa19ef4722c6e", "source_line": 10, "evidence_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv#L10" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "dsdual-config-manifest.json", "parent_record_id": "deepseek-1017-dual-spark-quick", "publisher": "x00byte", "repository": "x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark", "revision": "e85ed7ec3a403d8820a83e5aa33abe0985b58896", "document_path": "config/manifest.json", "pinned_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/manifest.json", "document_sha256": "6b3da5443112253502befe2a866983a31d188af234aef5e03865a724dbe9b65a", "checked_on": "2026-10-18" }, { "document_id": "dsdual-config-tensorfold-knobs.env.example", "parent_record_id": "deepseek-1017-dual-spark-quick", "publisher": "x00byte", "repository": "x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark", "revision": "e85ed7ec3a403d8820a83e5aa33abe0985b58896", "document_path": "config/tensorfold-knobs.env.example", "pinned_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/tensorfold-knobs.env.example", "document_sha256": "e26dc4761458be238d048c5a03b524993e7c632c27f8457527e3ba0030bc58c0", "checked_on": "2026-10-18" }, { "document_id": "dsdual-config-vllm-knobs.env.example", "parent_record_id": "deepseek-1017-dual-spark-quick", "publisher": "x00byte", "repository": "x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark", "revision": "e85ed7ec3a403d8820a83e5aa33abe0985b58896", "document_path": "config/vllm-knobs.env.example", "pinned_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/config/vllm-knobs.env.example", "document_sha256": "ee4eb1535b45d28c47fdd5b136e4b04268a7955d2ac9ca83174afeb35c0d62d8", "checked_on": "2026-10-18" }, { "document_id": "dsdual-docs-deployment.md", "parent_record_id": "deepseek-1017-dual-spark-quick", "publisher": "x00byte", "repository": "x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark", "revision": "e85ed7ec3a403d8820a83e5aa33abe0985b58896", "document_path": "docs/deployment.md", "pinned_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/deployment.md", "document_sha256": "72fb210773604c15560e73d6db8ada144b0e953354201e42b3b7aebd4c018799", "checked_on": "2026-10-18" }, { "document_id": "dsdual-docs-benchmark-method.md", "parent_record_id": "deepseek-1017-dual-spark-quick", "publisher": "x00byte", "repository": "x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark", "revision": "e85ed7ec3a403d8820a83e5aa33abe0985b58896", "document_path": "docs/benchmark-method.md", "pinned_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/docs/benchmark-method.md", "document_sha256": "33d83fe761ea2eaafaadee937fa212ffcec19d69a2528614776ad7fb1139ba3b", "checked_on": "2026-10-18" }, { "document_id": "dsdual-results-README.md", "parent_record_id": "deepseek-1017-dual-spark-quick", "publisher": "x00byte", "repository": "x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark", "revision": "e85ed7ec3a403d8820a83e5aa33abe0985b58896", "document_path": "results/README.md", "pinned_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/README.md", "document_sha256": "380393cc1e86d175f21af1c216b613a2aad741ed90c851b7e8f491bed5231acf", "checked_on": "2026-10-18" }, { "document_id": "dsdual-results-tensorfold-quick.csv", "parent_record_id": "deepseek-1017-dual-spark-quick", "publisher": "x00byte", "repository": "x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark", "revision": "e85ed7ec3a403d8820a83e5aa33abe0985b58896", "document_path": "results/tensorfold-quick.csv", "pinned_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/tensorfold-quick.csv", "document_sha256": "3005b3be00581a962f9a269211eef2d598a6358d83c41e565ee1549b28e03c33", "checked_on": "2026-10-18" }, { "document_id": "dsdual-results-vllm-quick.csv", "parent_record_id": "deepseek-1017-dual-spark-quick", "publisher": "x00byte", "repository": "x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark", "revision": "e85ed7ec3a403d8820a83e5aa33abe0985b58896", "document_path": "results/vllm-quick.csv", "pinned_url": "https://github.com/x00byte/DeepSeek-V4.1-Flash-1017-Dual-Spark/blob/e85ed7ec3a403d8820a83e5aa33abe0985b58896/results/vllm-quick.csv", "document_sha256": "befbf211d4f143798e12053ec21291cc77e0893284ccdb8d5a0aa19ef4722c6e", "checked_on": "2026-10-18" } ] } ``` --- # DeepSeek 1017: near-million-token needle and memory floor Two saved runs found one exact-string needle at 63 percent depth in random filler, with 1034753 and 1035020 prompt tokens. Each completion is seven tokens, with zero cached prompt tokens. These are two different build/setup observations, not repeated measurements of one unchanged configuration or evidence of general long-document reasoning. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw - Original author: Elevatormusic - Original project: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark - Pinned document: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md - Source revision: 9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32 - Source commit time (not experiment date): 2026-10-18T05:40:29Z - Checked: 2026-10-18 - Experiment date: 2026-10-18T01:51:54Z - Runtime topics: TensorFold - Hardware topics: spark, nvidia - Reported setup: {"system": "two DGX Spark GB10, TP2", "node_count": 2, "parallelism": null, "kv_dtype": null, "context_limit_tokens": 1048576, "experiment_date": "2026-10-18T01:51:54Z"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "fd2365042f56 near-1M first token", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json#L8", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "fd2365042f56", "input_tokens": 1034753, "output_tokens": 7, "cached_tokens": 0, "cache_state": "zero prefix-cache hit tokens; not a cold OS page-cache guarantee", "experiment_date": "2026-10-18T01:51:54Z", "needle_found": true, "reasoning_tokens": 0, "total_seconds": 835.8 } }, { "label": "92aa710b533d near-1M first token", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json#L8", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cached_tokens": 0, "cache_state": "zero prefix-cache hit tokens; not a cold OS page-cache guarantee", "experiment_date": "2026-10-18T04:47:07Z", "needle_found": true, "reasoning_tokens": 0, "total_seconds": 825.9 } }, { "label": "near-1M final head minimum MemAvailable", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L59", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cache_state": "same final needle; cached_tokens 0", "experiment_date": "2026-10-18", "memory_kind": "minimum_host_MemAvailable_during_prefill", "node": "head", "earlyoom_limit_MiB_at_measurement": 1536, "earlyoom_later_limit_MiB": 1024, "OS_memory_controls_changed": true, "ordinary_host_baseline_inferred": false } }, { "label": "near-1M final worker minimum MemAvailable", "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L59", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cache_state": "same final needle; cached_tokens 0", "experiment_date": "2026-10-18", "memory_kind": "minimum_host_MemAvailable_during_prefill", "node": "worker", "earlyoom_limit_MiB_at_measurement": 1536, "earlyoom_later_limit_MiB": 1024, "OS_memory_controls_changed": true, "ordinary_host_baseline_inferred": false } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document, not an independent experiment count. Preserve measurement type and unknown fields. ## Limits - No independent Run AI Here model execution. - Document revision is distinct from execution and weight identity. - File bytes, host RSS, anonymous memory, MemAvailable, KV allocation and configured context are different quantities. ## License declaration {"declared_spdx": null, "scope": "Source declaration only; base weights, converters, runtimes and source integration material retain their own terms."} ## Editorial answer Two saved runs found one exact-string needle at 63 percent depth in random filler, with 1034753 and 1035020 prompt tokens. Each completion is seven tokens, with zero cached prompt tokens. These are two different build/setup observations, not repeated measurements of one unchanged configuration or evidence of general long-document reasoning. By Run AI Here editorial · 2026-10-18 Original source-grounded editorial synthesis with local Spark Qwen assistance and Codex review; no site model benchmark. ## What does success mean here? Two saved runs found one exact-string needle at 63 percent depth in random filler, with 1034753 and 1035020 prompt tokens. Each completion is seven tokens, with zero cached prompt tokens. These are two different build/setup observations, not repeated measurements of one unchanged configuration or evidence of general long-document reasoning. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json, https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json ## How much latency and headroom were observed? The first token took 830.4 seconds in the earlier run and 817.7 seconds in the final setup. During the final fill, minimum host MemAvailable was 1779 MiB on the head and 2542 MiB on the worker. The source describes engine swap policy, host reclaim settings, Engram page dropping and earlyoom thresholds. This is a tuned narrow-margin observation, not a safe default capacity. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md ## What is still unknown? Exact executed image digests and request sampling values are absent from the selected raw records. The author also reports slower prefill states and final-image runs below the best earlier result. One depth, one string and a seven-token output do not validate multiple needles, long generation, multimodal context, concurrent near-million-token requests or general answer correctness. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/KNOWN-LIMITS.md, https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md ## What do the saved usage counts establish? The two saved JSON records retain the actual prompt count, seven completion tokens, zero cached prompt tokens and zero reported reasoning tokens. These values establish the size and outcome of the performed requests. Zero reasoning tokens does not identify an unlogged request thinking flag, and a configured context limit is not an additional completed request. Sources: https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "fd2365042f56 near-1M first token", "metric": "ttft", "value": 830.4, "unit": "s", "statistic": "single prompt, no variance estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "fd2365042f56", "input_tokens": 1034753, "output_tokens": 7, "cached_tokens": 0, "cache_state": "zero prefix-cache hit tokens; not a cold OS page-cache guarantee", "experiment_date": "2026-10-18T01:51:54Z", "needle_found": true, "reasoning_tokens": 0, "total_seconds": 835.8 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsneedle-runs-2026-10-18-needle-1m.json", "source_document_sha256": "9b81971c1378ab5b0bef466219c3a7aa7e2f78315bddbfc5bd0837cfec2fde40", "source_line": 8, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json#L8" }, { "label": "92aa710b533d near-1M first token", "metric": "ttft", "value": 817.7, "unit": "s", "statistic": "single prompt, no variance estimate", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cached_tokens": 0, "cache_state": "zero prefix-cache hit tokens; not a cold OS page-cache guarantee", "experiment_date": "2026-10-18T04:47:07Z", "needle_found": true, "reasoning_tokens": 0, "total_seconds": 825.9 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsneedle-runs-2026-10-18-needle-1m-final.json", "source_document_sha256": "c476006b019d46c68a334ad3763f3e8e0f0532d01dfc6fb800c043c225709ab1", "source_line": 8, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json#L8" }, { "label": "near-1M final head minimum MemAvailable", "metric": "minimum_memavailable", "value": 1779, "unit": "MiB", "statistic": "minimum observed in this one fill, not a safe capacity guarantee", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cache_state": "same final needle; cached_tokens 0", "experiment_date": "2026-10-18", "memory_kind": "minimum_host_MemAvailable_during_prefill", "node": "head", "earlyoom_limit_MiB_at_measurement": 1536, "earlyoom_later_limit_MiB": 1024, "OS_memory_controls_changed": true, "ordinary_host_baseline_inferred": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsneedle-runs-2026-10-18-INDEX.md", "source_document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "source_line": 59, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L59" }, { "label": "near-1M final worker minimum MemAvailable", "metric": "minimum_memavailable", "value": 2542, "unit": "MiB", "statistic": "minimum observed in this one fill, not a safe capacity guarantee", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw", "model_revision": "afc7aa0aab98ee735c0f54e0a122019af1ac08f1", "Engram_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "weight_precision": "EXL3 2.9bpw", "system": "two DGX Spark GB10, TP2", "node_count": 2, "engine": "TensorFold Python 0.6.6", "runtime_execution_revision": null, "executed_image_digest": null, "context_limit_tokens": 1048576, "concurrency": 1, "needle_depth_fraction": 0.63, "workload": "one exact-string needle in random filler lines", "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "general_quality_evaluation": false, "multi_request_long_context_validated": false, "runtime_revision_prefix": "92aa710b533d", "input_tokens": 1035020, "output_tokens": 7, "cache_state": "same final needle; cached_tokens 0", "experiment_date": "2026-10-18", "memory_kind": "minimum_host_MemAvailable_during_prefill", "node": "worker", "earlyoom_limit_MiB_at_measurement": 1536, "earlyoom_later_limit_MiB": 1024, "OS_memory_controls_changed": true, "ordinary_host_baseline_inferred": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "dsneedle-runs-2026-10-18-INDEX.md", "source_document_sha256": "e69d8c270e6407ddbaacb82da36cc48994b6f20fd4300d176753371b4e23fb8e", "source_line": 59, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/INDEX.md#L59" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "dsneedle-runs-2026-10-18-needle-1m.json", "parent_record_id": "deepseek-1017-million-token-needle", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "runs/2026-10-18/needle-1m.json", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json", "document_sha256": "9b81971c1378ab5b0bef466219c3a7aa7e2f78315bddbfc5bd0837cfec2fde40", "checked_on": "2026-10-18" }, { "document_id": "dsneedle-runs-2026-10-18-needle-1m-final.json", "parent_record_id": "deepseek-1017-million-token-needle", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "runs/2026-10-18/needle-1m-final.json", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json", "document_sha256": "c476006b019d46c68a334ad3763f3e8e0f0532d01dfc6fb800c043c225709ab1", "checked_on": "2026-10-18" }, { "document_id": "dsneedle-docs-KNOWN-LIMITS.md", "parent_record_id": "deepseek-1017-million-token-needle", "publisher": "Elevatormusic", "repository": "Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark", "revision": "9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32", "document_path": "docs/KNOWN-LIMITS.md", "pinned_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/docs/KNOWN-LIMITS.md", "document_sha256": "eeb6bc6f7a3fce20654f977581e9718b124e70fc01ab358efebefe9afd9dc942", "checked_on": "2026-10-18" } ] } ``` ## memory observations ```json { "memory_observations": [ { "record_id": "deepseek-1017-million-token-needle", "kind": "minimum_host_MemAvailable_during_final_fill", "head_MiB": 1779, "worker_MiB": 2542, "host_tuning_required_in_source": true, "safe_capacity_guarantee": false } ] } ``` ## long context cases ```json { "long_context_cases": [ { "id": "dsneedle-runs-2026-10-18-needle-1m.json", "classification": "source_reported_single_needle_success", "configured_limit_tokens": 1048576, "actual_input_tokens": 1034753, "actual_output_tokens": 7, "needle_depth_fraction": 0.63, "exact_match": true, "TTFT_seconds": 830.4, "cached_tokens": 0, "runtime_revision_prefix": "fd2365042f56", "executed_image_digest": null, "sampling": null, "quality_scope": "one needle at one depth; not broad reasoning, multi-needle, long-response or concurrency validation", "source_document_id": "dsneedle-runs-2026-10-18-needle-1m.json", "source_document_sha256": "9b81971c1378ab5b0bef466219c3a7aa7e2f78315bddbfc5bd0837cfec2fde40", "source_line": 3, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m.json#L3" }, { "id": "dsneedle-runs-2026-10-18-needle-1m-final.json", "classification": "source_reported_single_needle_success", "configured_limit_tokens": 1048576, "actual_input_tokens": 1035020, "actual_output_tokens": 7, "needle_depth_fraction": 0.63, "exact_match": true, "TTFT_seconds": 817.7, "cached_tokens": 0, "runtime_revision_prefix": "92aa710b533d", "executed_image_digest": null, "sampling": null, "quality_scope": "one needle at one depth; not broad reasoning, multi-needle, long-response or concurrency validation", "source_document_id": "dsneedle-runs-2026-10-18-needle-1m-final.json", "source_document_sha256": "c476006b019d46c68a334ad3763f3e8e0f0532d01dfc6fb800c043c225709ab1", "source_line": 3, "evidence_url": "https://github.com/Elevatormusic/DeepSeek-V4.1-Flash-1017-EXL3-TensorFold-2x-DGX-Spark/blob/9c6ab02340b770c6c3e1a9a4d5e70a56c5829d32/runs/2026-10-18/needle-1m-final.json#L3" } ] } ``` --- # DeepSeek 1017: VQ file identity before runtime claims The converter reports a 1000-tensor base, its exact assembled file size and a successful loader binding check. The separate vision sidecar and official 1017 Engram files retain individual identities. These are artifact and loader observations. The selected source explicitly has no 1017-specific serving speed, peak RAM or model-quality measurement. - Type: Deployment recipe - Evidence: source_reported; site reproduced: false - Model family: DeepSeek - Model: Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF - Original author: Baekpica - Original project: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF - Pinned document: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md - Source revision: f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5 - Source commit time (not experiment date): 2026-10-17T11:05:38.000Z - Checked: 2026-10-18 - Experiment date: 2026-10-17 status in card - Runtime topics: ds4-dfm-rs - Hardware topics: spark, nvidia - Reported setup: {"system": null, "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": 1048576, "experiment_date": "2026-10-17 status in card"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "1017 VQ base GGUF", "evidence_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md#L97", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "model_revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "base_model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "quantizer_revision": "7fb26cc332d740416792204a29c7d13d2986caf5", "vision_converter_revision": "e4ea07889144d85acd33b2490ec4f44032644fb0", "quantization": "YoungAi vq8sh14-q4k-mtpnative; routed-expert VQ, q4_K skeleton; native MTP towers", "measurement_kind": "publisher artifact layout inspection, not runtime evaluation", "experiment_date": "2026-10-17 status in card", "system": null, "engine": "target ds4-dfm-rs; loader check passed, serving unmeasured on 1017", "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "context_limit_tokens": 1048576, "context_limit_origin": "model metadata, not measured serving result", "speed_1017_measured": false, "quality_1017_measured": false, "peak_runtime_memory_measured": false, "base_model_speed_inherited": false, "filename": "DeepSeek-V4.1-Flash-vq8sh14-q4k-mtpnative.gguf", "memory_kind": "artifact_disk_file_bytes", "Engram_included": false } }, { "label": "1017 BF16 vision sidecar", "evidence_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md#L98", "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "model_revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "base_model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "quantizer_revision": "7fb26cc332d740416792204a29c7d13d2986caf5", "vision_converter_revision": "e4ea07889144d85acd33b2490ec4f44032644fb0", "quantization": "YoungAi vq8sh14-q4k-mtpnative; routed-expert VQ, q4_K skeleton; native MTP towers", "measurement_kind": "publisher artifact layout inspection, not runtime evaluation", "experiment_date": "2026-10-17 status in card", "system": null, "engine": "target ds4-dfm-rs; loader check passed, serving unmeasured on 1017", "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "context_limit_tokens": 1048576, "context_limit_origin": "model metadata, not measured serving result", "speed_1017_measured": false, "quality_1017_measured": false, "peak_runtime_memory_measured": false, "base_model_speed_inherited": false, "filename": "DeepSeek-V4.1-Flash.vision.gguf", "memory_kind": "artifact_disk_file_bytes", "Engram_included": false } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document, not an independent experiment count. Preserve measurement type and unknown fields. ## Limits - No independent Run AI Here model execution. - Document revision is distinct from execution and weight identity. - File bytes, host RSS, anonymous memory, MemAvailable, KV allocation and configured context are different quantities. ## License declaration {"declared_spdx": "mit", "scope": "Source declaration only; base weights, converters, runtimes and source integration material retain their own terms."} ## Editorial answer The converter reports a 1000-tensor base, its exact assembled file size and a successful loader binding check. The separate vision sidecar and official 1017 Engram files retain individual identities. These are artifact and loader observations. The selected source explicitly has no 1017-specific serving speed, peak RAM or model-quality measurement. By Run AI Here editorial · 2026-10-18 Original source-grounded editorial synthesis with local Spark Qwen assistance and Codex review; no site model benchmark. ## Which facts are confirmed by the publisher? The converter reports a 1000-tensor base, its exact assembled file size and a successful loader binding check. The separate vision sidecar and official 1017 Engram files retain individual identities. These are artifact and loader observations. The selected source explicitly has no 1017-specific serving speed, peak RAM or model-quality measurement. Sources: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md, https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/SHA256SUMS ## Why are old names unsafe identifiers? The 1017 Engram shard names and byte sizes match the older release, while their hashes differ. The vision sidecar also keeps its name and size with different contents. Use revision and SHA-256 alongside paths. The quantizer and vision converter have separate pinned commits, and domain sidecars trained against the older base are not treated as interchangeable. Sources: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/SOURCE.md, https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/RECIPE.md ## Does a small file prove a small runtime footprint? The base excludes its Engram files and does not describe all live buffers, caches, page residency or service overhead. Its native context field comes from model metadata. Old V4.1 speed and fidelity numbers in the card remain old-checkpoint reference results; they are not published as this 1017 artifact’s measured performance. Sources: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md, https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/RECIPE.md ## Does the GGUF suffix certify generic loader compatibility? This artifact uses custom VQ payload storage and targets the named ds4-dfm-rs or YoungAi path. The file extension alone does not show that an arbitrary llama.cpp or Transformers installation can load it. Keep the converter commits, recipe string and stated target runtime rather than generalizing compatibility from the container name. Sources: https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/RECIPE.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "1017 VQ base GGUF", "metric": "artifact_size", "value": 113556639424, "unit": "bytes", "statistic": "source-reported file bytes; not a RAM requirement", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "model_revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "base_model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "quantizer_revision": "7fb26cc332d740416792204a29c7d13d2986caf5", "vision_converter_revision": "e4ea07889144d85acd33b2490ec4f44032644fb0", "quantization": "YoungAi vq8sh14-q4k-mtpnative; routed-expert VQ, q4_K skeleton; native MTP towers", "measurement_kind": "publisher artifact layout inspection, not runtime evaluation", "experiment_date": "2026-10-17 status in card", "system": null, "engine": "target ds4-dfm-rs; loader check passed, serving unmeasured on 1017", "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "context_limit_tokens": 1048576, "context_limit_origin": "model metadata, not measured serving result", "speed_1017_measured": false, "quality_1017_measured": false, "peak_runtime_memory_measured": false, "base_model_speed_inherited": false, "filename": "DeepSeek-V4.1-Flash-vq8sh14-q4k-mtpnative.gguf", "memory_kind": "artifact_disk_file_bytes", "Engram_included": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "ds1017vq-readme", "source_document_sha256": "b9f85bae000a8aa0e270555d17b1f9bd859a79bc3f5f2d7fde7eb00607cc990f", "source_line": 97, "evidence_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md#L97" }, { "label": "1017 BF16 vision sidecar", "metric": "artifact_size", "value": 971066016, "unit": "bytes", "statistic": "source-reported file bytes; not a RAM requirement", "uncertainty": null, "conditions": { "model": "DeepSeek-V4.1-Flash-1017", "model_artifact": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "model_revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "base_model_revision": "1bae0e1088993b9b82de08f59de5be5a34e42170", "quantizer_revision": "7fb26cc332d740416792204a29c7d13d2986caf5", "vision_converter_revision": "e4ea07889144d85acd33b2490ec4f44032644fb0", "quantization": "YoungAi vq8sh14-q4k-mtpnative; routed-expert VQ, q4_K skeleton; native MTP towers", "measurement_kind": "publisher artifact layout inspection, not runtime evaluation", "experiment_date": "2026-10-17 status in card", "system": null, "engine": "target ds4-dfm-rs; loader check passed, serving unmeasured on 1017", "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "context_limit_tokens": 1048576, "context_limit_origin": "model metadata, not measured serving result", "speed_1017_measured": false, "quality_1017_measured": false, "peak_runtime_memory_measured": false, "base_model_speed_inherited": false, "filename": "DeepSeek-V4.1-Flash.vision.gguf", "memory_kind": "artifact_disk_file_bytes", "Engram_included": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "ds1017vq-readme", "source_document_sha256": "b9f85bae000a8aa0e270555d17b1f9bd859a79bc3f5f2d7fde7eb00607cc990f", "source_line": 98, "evidence_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/README.md#L98" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "ds1017vq-RECIPE.md", "parent_record_id": "deepseek-1017-vq-artifact-inspection", "publisher": "Baekpica", "repository": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "document_path": "RECIPE.md", "pinned_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/RECIPE.md", "document_sha256": "1b66bfc44386bd6b708b1d4250cdd0dd825d2b45eee18c74d2bebfbff9e62486", "checked_on": "2026-10-18" }, { "document_id": "ds1017vq-SOURCE.md", "parent_record_id": "deepseek-1017-vq-artifact-inspection", "publisher": "Baekpica", "repository": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "document_path": "SOURCE.md", "pinned_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/SOURCE.md", "document_sha256": "d039fb41a620380e352de9794271a6b2855e5d6931979d43695f4d3a236d443e", "checked_on": "2026-10-18" }, { "document_id": "ds1017vq-SHA256SUMS", "parent_record_id": "deepseek-1017-vq-artifact-inspection", "publisher": "Baekpica", "repository": "Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF", "revision": "f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5", "document_path": "SHA256SUMS", "pinned_url": "https://huggingface.co/Baekpica/DeepSeek-V4.1-Flash-1017-VQ-GGUF/blob/f81ee2e9801fcc58fd51ae75dd827467ffc0b9e5/SHA256SUMS", "document_sha256": "6b98a0a8923558f69ad53a9d60c29332b984a59f8109a6115c7c88e684763b19", "checked_on": "2026-10-18" } ] } ``` ## memory observations ```json { "memory_observations": [ { "record_id": "deepseek-1017-vq-artifact-inspection", "kind": "file_inventory_not_RAM", "base_bytes": 113556639424, "vision_bytes": 971066016, "Engram_shard_bytes": [ 101535150936, 101537926640 ], "actual_peak_RAM": null, "speed_and_quality_1017": null } ] } ``` --- # Qwen Flash-Next: SGLang concurrency and latency One 128 GB unified-memory DGX Spark serves senfu NVFP4 at its full recorded revision through SGLang commit 4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b. The recipe keeps PLE on NVMe through mmap, uses an FP8 dense path and an NVFP4 LM head, and disables MTP. The repository documentation pin is separate from the engine and weight revisions. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: senfu/Qwen3.8-Flash-Next-NVFP4 - Original author: nabe2030 - Original project: https://github.com/nabe2030/flashnext-dgx-spark - Pinned document: https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md - Source revision: 8a161f2bee2e2d85b689a2d457ae68bead3929a4 - Source commit time (not experiment date): 2026-09-21T10:13:50Z - Checked: 2026-10-18 - Experiment date: None - Runtime topics: SGLang - Hardware topics: spark, nvidia - Reported setup: {"system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "single Spark senfu C1 aggregate_output_rate", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" } }, { "label": "single Spark senfu C1 median_ttft", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" } }, { "label": "single Spark senfu C1 median_time_per_output_token", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" } }, { "label": "single Spark senfu C8 aggregate_output_rate", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" } }, { "label": "single Spark senfu C8 median_ttft", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" } }, { "label": "single Spark senfu C8 median_time_per_output_token", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" } }, { "label": "single Spark senfu C16 aggregate_output_rate", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" } }, { "label": "single Spark senfu C16 median_ttft", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" } }, { "label": "single Spark senfu C16 median_time_per_output_token", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" } }, { "label": "single Spark senfu C32 aggregate_output_rate", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" } }, { "label": "single Spark senfu C32 median_ttft", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" } }, { "label": "single Spark senfu C32 median_time_per_output_token", "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - One source document can contain different experiments and versions; not an independent experiment count. ## Limits - No independent site model execution. - Missing sampling, exact executed image digests and measurement boundaries remain null. - Configured window, completed input, scheduler ceiling and measured full-window concurrency are separate quantities. ## License declaration {"declared_spdx": "apache-2.0", "scope": "Source declaration; models, runtime and dependencies retain their exact terms."} ## Editorial answer One 128 GB unified-memory DGX Spark serves senfu NVFP4 at its full recorded revision through SGLang commit 4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b. The recipe keeps PLE on NVMe through mmap, uses an FP8 dense path and an NVFP4 LM head, and disables MTP. The repository documentation pin is separate from the engine and weight revisions. By Run AI Here editorial · 2026-10-18 Original source-based interpretation; local Spark Qwen draft with Codex factual review, no site model benchmark. ## Which checkpoint and runtime were tested? One 128 GB unified-memory DGX Spark serves senfu NVFP4 at its full recorded revision through SGLang commit 4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b. The recipe keeps PLE on NVMe through mmap, uses an FP8 dense path and an NVFP4 LM head, and disables MTP. The repository documentation pin is separate from the engine and weight revisions. Sources: https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md ## What workload defines the concurrency curve? The selected ShareGPT-200 sweep uses vllm bench serve, thinking off and explicitly disabled radix caching, after one discarded warmup request. C1 is the median of three trials; C8, C16 and C32 are single trials. Exact prompt/output distributions, sampling and per-sweep window are not reported. All selected senfu trials completed 200 requests without errors. Sources: https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md ## How should throughput and latency be read together? Aggregate output rises with concurrency, while the reported median TTFT and time per output token also rise. Read all three metrics with their own timing definitions. Dividing aggregate rate by concurrency does not reconstruct a measured individual stream. Single high-load trials provide no variability estimate, and results from another checkpoint or engine cannot isolate a pure scheduler effect. Sources: https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md ## Does the recipe prove simultaneous long windows? The 49152-token window, 393216 total tokens and eight slots describe one published long-context configuration, not a saved all-slots-full workload. The separate 3.04 GB KV allocation at 128K keeps the original GB label; it is not whole-model peak RAM. The source reports 128K stability but supplies no complete long-request latency or memory series for every concurrency level. Sources: https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "single Spark senfu C1 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 26.97, "unit": "tokens/s", "statistic": "median of three trials", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 9, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9" }, { "label": "single Spark senfu C1 median_ttft", "metric": "median_ttft", "value": 221.23, "unit": "ms", "statistic": "median of three trials", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 9, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9" }, { "label": "single Spark senfu C1 median_time_per_output_token", "metric": "median_time_per_output_token", "value": 36.14, "unit": "ms", "statistic": "median of three trials", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 1, "repetitions": 3, "summary_kind": "median of three trials" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 9, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L9" }, { "label": "single Spark senfu C8 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 67.68, "unit": "tokens/s", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 10, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10" }, { "label": "single Spark senfu C8 median_ttft", "metric": "median_ttft", "value": 526.78, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 10, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10" }, { "label": "single Spark senfu C8 median_time_per_output_token", "metric": "median_time_per_output_token", "value": 113.46, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 8, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 10, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L10" }, { "label": "single Spark senfu C16 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 77.48, "unit": "tokens/s", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 11, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11" }, { "label": "single Spark senfu C16 median_ttft", "metric": "median_ttft", "value": 686.27, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 11, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11" }, { "label": "single Spark senfu C16 median_time_per_output_token", "metric": "median_time_per_output_token", "value": 198.0, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 16, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 11, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L11" }, { "label": "single Spark senfu C32 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 89.35, "unit": "tokens/s", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 12, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12" }, { "label": "single Spark senfu C32 median_ttft", "metric": "median_ttft", "value": 1017.98, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 12, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12" }, { "label": "single Spark senfu C32 median_time_per_output_token", "metric": "median_time_per_output_token", "value": 343.57, "unit": "ms", "statistic": "single trial", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "senfu/Qwen3.8-Flash-Next-NVFP4", "model_revision": "5d37b3b3711d8406174b96ff950c0aa16324b266", "engine": "SGLang", "runtime_execution_revision": "4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b", "executed_image_digest": null, "system": "one DGX Spark GB10, 128 GB unified memory; NVMe file-backed PLE", "node_count": 1, "weight_precision": "NVFP4 experts and LM head; FP8 dense path", "kv_dtype": null, "draft_tokens": 0, "thinking": false, "temperature": null, "top_p": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "request_count": 200, "cache_state": "radix/prefix cache explicitly disabled; one warmup discarded", "harness": "vllm bench serve, ShareGPT 200 prompts", "experiment_date": null, "context_limit_tokens": null, "executed_context_for_each_sweep_unreported": true, "PLE_backing": "NVMe mmap", "exact_prompt_and_output_distributions_unreported": true, "concurrency": 32, "repetitions": 1, "summary_kind": "single trial" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-sglang-results", "source_document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "source_line": 12, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md#L12" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "qwen-sglang-results", "parent_record_id": "qwen38-flash-next-sglang-concurrency", "publisher": "nabe2030", "repository": "nabe2030/flashnext-dgx-spark", "revision": "8a161f2bee2e2d85b689a2d457ae68bead3929a4", "document_path": "results/speed_senfu_vs_radixark.md", "pinned_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/results/speed_senfu_vs_radixark.md", "document_sha256": "cf77f1feb1a37ab7fcdd8073ca0ff3b5fe78b8c47cf261fd917ae69fcb018297", "checked_on": "2026-10-18" } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "source_reported_KV_allocation_at_context", "context_limit_tokens": 131072, "KV_source_GB": 3.04, "GB_byte_convention_verified": false, "whole_model_peak_RAM": null, "source_document_id": "qwen-sglang-readme", "source_document_sha256": "8a27445102511702ff42fa60cfcc469583de37c4c0bd031c43cd40db7c0db9f4", "source_line": 19, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md#L19" }, { "kind": "source_recipe_slots_not_a_completed_long_batch", "per_request_window_tokens": 49152, "shared_total_tokens": 393216, "request_slots": 8, "all_slots_full_simultaneously_proven": false, "source_document_id": "qwen-sglang-readme", "source_document_sha256": "8a27445102511702ff42fa60cfcc469583de37c4c0bd031c43cd40db7c0db9f4", "source_line": 134, "evidence_url": "https://github.com/nabe2030/flashnext-dgx-spark/blob/8a161f2bee2e2d85b689a2d457ae68bead3929a4/README.md#L134" } ] } ``` --- # Qwen Flash-Next: TP2 concurrency and cold-prefix cohorts The k76 v2.0.0 concurrency series uses the earlier 16c9bd54788d12390838a65ce4a4ecda97fa5f1d GDN-MSE checkpoint, while v2.2.0 cold-prefix measurements use 03f4a0570496bbe741189f4a2a7d611679e85406. Both employ two directly connected Sparks, vLLM plus b12x and mixed NVFP4/MXFP8 paths. Release image digests and abbreviated engine commits do not establish an exact executed image digest. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE - Original author: ursuciprian - Original project: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2 - Pinned document: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/README.md - Source revision: b384f4a0e3ff0d563980befe52eb3e6964a5b6b1 - Source commit time (not experiment date): 2026-10-17T10:48:19Z - Checked: 2026-10-18 - Experiment date: 2026-10-10 - Runtime topics: vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "kv_dtype": "fp8", "context_limit_tokens": 262144, "experiment_date": "2026-10-10"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "TP2 v2.0.0 C1 2048 per_request_decode_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1142", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 1, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.0.0 C4 2048 per_request_decode_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1146", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 4, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.0.0 C8 2048 per_request_decode_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1148", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 8, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.0.0 C16 2048 per_request_decode_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1150", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 16, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.0.0 C1 2048 aggregate_output_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1152", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 1, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.0.0 C4 2048 aggregate_output_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1156", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 4, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.0.0 C8 2048 aggregate_output_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1158", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 8, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.0.0 C16 2048 aggregate_output_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1160", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 16, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.2.0 C1 16384 ttft", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1262", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 16384, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.2.0 C1 16384 prefill_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1263", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 16384, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.2.0 C1 65536 ttft", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1264", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 65536, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.2.0 C1 65536 prefill_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1265", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 65536, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.2.0 C1 131072 ttft", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1266", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 131072, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false } }, { "label": "TP2 v2.2.0 C1 131072 prefill_rate", "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1267", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 131072, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - One source document can contain different experiments and versions; not an independent experiment count. ## Limits - No independent site model execution. - Missing sampling, exact executed image digests and measurement boundaries remain null. - Configured window, completed input, scheduler ceiling and measured full-window concurrency are separate quantities. ## License declaration {"declared_spdx": "apache-2.0", "scope": "Source declaration; models, runtime and dependencies retain their exact terms."} ## Editorial answer The k76 v2.0.0 concurrency series uses the earlier 16c9bd54788d12390838a65ce4a4ecda97fa5f1d GDN-MSE checkpoint, while v2.2.0 cold-prefix measurements use 03f4a0570496bbe741189f4a2a7d611679e85406. Both employ two directly connected Sparks, vLLM plus b12x and mixed NVFP4/MXFP8 paths. Release image digests and abbreviated engine commits do not establish an exact executed image digest. By Run AI Here editorial · 2026-10-18 Original source-based interpretation; local Spark Qwen draft with Codex factual review, no site model benchmark. ## Why keep releases separate? The k76 v2.0.0 concurrency series uses the earlier 16c9bd54788d12390838a65ce4a4ecda97fa5f1d GDN-MSE checkpoint, while v2.2.0 cold-prefix measurements use 03f4a0570496bbe741189f4a2a7d611679e85406. Both employ two directly connected Sparks, vLLM plus b12x and mixed NVFP4/MXFP8 paths. Release image digests and abbreviated engine commits do not establish an exact executed image digest. Sources: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/VERSIONS.md ## What does the k76 concurrency series measure? The selected cells use llama-benchy 0.4.1.dev4+g0d4de4271, 2048 input, 512 requested output tokens, temperature 1.0, top-p 0.95, top-k 20 and thinking on. Prefix caching is enabled. Means and SD cover three runs in one boot, not independent boots. Aggregate output includes prompt-processing wall time; individual decode begins after the first token. Sources: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv, https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/BENCHMARKS.md ## What does the later cold-prefix arm measure? At C1 and temperature zero, v2.2.0 requests one output token after 16384, 65536 or 131072 input tokens. Four boots each contain three cold prompts. The SD describes variation between boot means. These observations measure first-token latency and prefill, not sustained generation, broad long-document accuracy or multi-user full-window serving. Sources: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv, https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/BENCHMARKS.md ## Which conclusions are outside these data? The two arms cannot be joined into a single unchanged deployment curve. Current request slots and KV capacity are configuration limits, and cold prefix does not mean cold filesystem pages or cold compilation state. Complete executed digests and some runtime identities remain null. The publisher is already represented by a separate one-Spark document in the catalog; this new document is not a new independent researcher. Sources: https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/README.md, https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/VERSIONS.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "TP2 v2.0.0 C1 2048 per_request_decode_rate", "metric": "per_request_decode_rate", "value": 88.697, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 2.562, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 1, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1142, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1142" }, { "label": "TP2 v2.0.0 C4 2048 per_request_decode_rate", "metric": "per_request_decode_rate", "value": 48.631, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 1.533, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 4, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1146, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1146" }, { "label": "TP2 v2.0.0 C8 2048 per_request_decode_rate", "metric": "per_request_decode_rate", "value": 33.236, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 0.977, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 8, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1148, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1148" }, { "label": "TP2 v2.0.0 C16 2048 per_request_decode_rate", "metric": "per_request_decode_rate", "value": 22.142, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 0.362, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_req", "source_campaign": "k76-capability-matrix", "concurrency": 16, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1150, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1150" }, { "label": "TP2 v2.0.0 C1 2048 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 88.697, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 2.562, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 1, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1152, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1152" }, { "label": "TP2 v2.0.0 C4 2048 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 156.645, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 4.558, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 4, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1156, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1156" }, { "label": "TP2 v2.0.0 C8 2048 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 172.213, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 16.348, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 8, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1158, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1158" }, { "label": "TP2 v2.0.0 C16 2048 aggregate_output_rate", "metric": "aggregate_output_rate", "value": 231.294, "unit": "tokens/s", "statistic": "mean ± sd of 3 runs, one boot", "uncertainty": { "standard_deviation": 4.91, "kind": "SD across runs in one boot" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "16c9bd54788d12390838a65ce4a4ecda97fa5f1d", "recipe_release": "v2.0.0", "experiment_date": "2026-10-10", "harness": "llama-benchy 0.4.1.dev4+g0d4de4271", "source_run": "2x:results/k76-capability-matrix-20261010-1059/2x/d0.json", "source_metric": "tg_total", "source_campaign": "k76-capability-matrix", "concurrency": 16, "input_tokens": 2048, "output_tokens": 512, "temperature": 1.0, "top_p": 0.95, "top_k": 20, "thinking": true, "cache_state": "prefix caching enabled; fresh 2048-token task, not a cold-cache campaign", "repetitions": "mean ± sd of 3 runs, one boot", "aggregate_denominator": "all output tokens / wall time including prompt processing", "per_request_denominator": "one output stream after first token", "release_image_digest": "sha256:8abb60eeeb79ca13680341c17d87975801da1218c06df2505856b23da7ed1821", "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1160, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1160" }, { "label": "TP2 v2.2.0 C1 16384 ttft", "metric": "ttft", "value": 5.487, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 0.045, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 16384, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1262, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1262" }, { "label": "TP2 v2.2.0 C1 16384 prefill_rate", "metric": "prefill_rate", "value": 2986.0, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 24.6, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 16384, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1263, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1263" }, { "label": "TP2 v2.2.0 C1 65536 ttft", "metric": "ttft", "value": 24.041, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 0.057, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 65536, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1264, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1264" }, { "label": "TP2 v2.2.0 C1 65536 prefill_rate", "metric": "prefill_rate", "value": 2726.1, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 6.4, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 65536, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1265, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1265" }, { "label": "TP2 v2.2.0 C1 131072 ttft", "metric": "ttft", "value": 54.037, "unit": "s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 0.256, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "ttft_s", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 131072, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1266, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1266" }, { "label": "TP2 v2.2.0 C1 131072 prefill_rate", "metric": "prefill_rate", "value": 2425.6, "unit": "tokens/s", "statistic": "mean of 4 boots x 3 cold prompts, sd between boots", "uncertainty": { "standard_deviation": 11.5, "kind": "SD between boot means; four boots x three cold prompts" }, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE", "engine": "vLLM + b12x", "engine_version": null, "runtime_execution_revision": null, "executed_image_digest": null, "system": "two DGX Spark GB10, 128 GB unified memory each; ConnectX-7 direct-cabled", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVFP4 routed experts and decode GDN; MXFP8 dense/attention and prefill GDN copies", "extra_prefill_revision": "7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd", "kv_dtype": "fp8", "draft_tokens": 4, "context_limit_tokens": 262144, "configured_max_requests": 16, "actual_output_tokens_per_request": null, "GPU_clock_cap_MHz": null, "model_revision": "03f4a0570496bbe741189f4a2a7d611679e85406", "recipe_release": "v2.2.0", "experiment_date": "2026-10-15", "harness": "cold.py (k101)", "source_run": "1x:results/k101s-1x-gdn-prefill-views-ab-20261015-1600/tp2/2x/", "source_metric": "pp", "source_campaign": "promotion A/B", "concurrency": 1, "input_tokens": 131072, "output_tokens": 1, "temperature": 0, "top_p": null, "top_k": null, "thinking": null, "cache_state": "cold prefix with no hit, not a cold OS page-cache claim", "repetitions": "mean of 4 boots x 3 cold prompts, sd between boots", "aggregate_denominator": null, "per_request_denominator": null, "release_image_digest": null, "release_digest_is_proven_executed_digest": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-tp2-csv", "source_document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "source_line": 1267, "evidence_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv#L1267" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "qwen-tp2-csv", "parent_record_id": "qwen38-flash-next-tp2-concurrency-context", "publisher": "ursuciprian", "repository": "ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2", "revision": "b384f4a0e3ff0d563980befe52eb3e6964a5b6b1", "document_path": "docs/data/capability.csv", "pinned_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/data/capability.csv", "document_sha256": "a82bffe4f0e33f4243db9549c02fb8e9379e5faa62825a35a7b087cd30b4996e", "checked_on": "2026-10-18" }, { "document_id": "qwen-tp2-method", "parent_record_id": "qwen38-flash-next-tp2-concurrency-context", "publisher": "ursuciprian", "repository": "ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2", "revision": "b384f4a0e3ff0d563980befe52eb3e6964a5b6b1", "document_path": "docs/BENCHMARKS.md", "pinned_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/docs/BENCHMARKS.md", "document_sha256": "5645ff4e129b3e2766672e092c65333b073b5550ad89aa6374cbd81ce1da7776", "checked_on": "2026-10-18" }, { "document_id": "qwen-tp2-versions", "parent_record_id": "qwen38-flash-next-tp2-concurrency-context", "publisher": "ursuciprian", "repository": "ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2", "revision": "b384f4a0e3ff0d563980befe52eb3e6964a5b6b1", "document_path": "VERSIONS.md", "pinned_url": "https://github.com/ursuciprian/qwen3.8-flash-next-dgx-spark-tp-2/blob/b384f4a0e3ff0d563980befe52eb3e6964a5b6b1/VERSIONS.md", "document_sha256": "eb9ec6435802430a1be7302b445c67ceb48da9cf5e1ba32e939b5f7aaf9b100e", "checked_on": "2026-10-18" } ] } ``` --- # Qwen Flash-Next: RTX 5090 CPU offload and long-context gates The pinned MXFP4 GGUF runs through ik_llama.cpp merge d854dfd36a7262fc46cb7542716b04a9f6d4b47d on one RTX 5090 with WSL2, a 128 GiB-class host and 96 GiB VM ceiling. The baseline puts 38 expert layers on CPU, keeps mmap, disables the CUDA pinned-host override and uses Q8_0 KV. One request and one slot are used throughout; this is not full-model GPU residence. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF - Original author: kutaelee - Original project: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe - Pinned document: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md - Source revision: fca75dd108ce04b6a79db3b45934ed58bba893bb - Source commit time (not experiment date): 2026-09-18T12:12:22Z - Checked: 2026-10-18 - Experiment date: None - Runtime topics: llama.cpp - Hardware topics: nvidia - Reported setup: {"system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "parallelism": null, "kv_dtype": "Q8_0", "context_limit_tokens": 32768, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "32K needle gate prefill_rate", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L53", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "workload": "32K needle gate", "memory_cohort": "completed 32K monitoring" } }, { "label": "32K needle gate decode_rate", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L53", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "workload": "32K needle gate", "memory_cohort": "completed 32K monitoring" } }, { "label": "128K server smoke prefill_rate", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L54", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 61, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K server smoke", "memory_cohort": "no paired 128K telemetry published" } }, { "label": "128K server smoke decode_rate", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L54", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 61, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K server smoke", "memory_cohort": "no paired 128K telemetry published" } }, { "label": "128K long-context needle prefill_rate", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L55", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 104805, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K long-context needle", "memory_cohort": "no paired 128K telemetry published" } }, { "label": "128K long-context needle decode_rate", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L55", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 104805, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K long-context needle", "memory_cohort": "no paired 128K telemetry published" } }, { "label": "32K telemetry startup", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "startup_duration", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false } }, { "label": "32K telemetry host_rss", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "peak_host_RSS", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false } }, { "label": "32K telemetry minimum_memavailable", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "minimum_MemAvailable_in_reported_monitor_scope", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false } }, { "label": "32K telemetry swap_use", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "maximum_swap_use", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false } }, { "label": "32K telemetry GPU_memory", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "peak_GPU_memory", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false } }, { "label": "canonical MTP n1 peak GPU memory", "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L61", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 1, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": null, "context_limit_tokens": 131072, "cache_state": null, "memory_cohort": "canonical MTP n1 short gates; not baseline or long-context peak", "memory_kind": "peak_GPU_memory" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - One source document can contain different experiments and versions; not an independent experiment count. ## Limits - No independent site model execution. - Missing sampling, exact executed image digests and measurement boundaries remain null. - Configured window, completed input, scheduler ceiling and measured full-window concurrency are separate quantities. ## License declaration {"declared_spdx": "mit", "scope": "Source declaration; models, runtime and dependencies retain their exact terms."} ## Editorial answer The pinned MXFP4 GGUF runs through ik_llama.cpp merge d854dfd36a7262fc46cb7542716b04a9f6d4b47d on one RTX 5090 with WSL2, a 128 GiB-class host and 96 GiB VM ceiling. The baseline puts 38 expert layers on CPU, keeps mmap, disables the CUDA pinned-host override and uses Q8_0 KV. One request and one slot are used throughout; this is not full-model GPU residence. By Run AI Here editorial · 2026-10-18 Original source-based interpretation; local Spark Qwen draft with Codex factual review, no site model benchmark. ## Which memory placement belongs to the report? The pinned MXFP4 GGUF runs through ik_llama.cpp merge d854dfd36a7262fc46cb7542716b04a9f6d4b47d on one RTX 5090 with WSL2, a 128 GiB-class host and 96 GiB VM ceiling. The baseline puts 38 expert layers on CPU, keeps mmap, disables the CUDA pinned-host override and uses Q8_0 KV. One request and one slot are used throughout; this is not full-model GPU residence. Sources: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md, https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/configs/ik-llama-baseline-128k.example.sh ## What input was actually completed? The source reports a 26253-token needle under a 32768 window, a 61-token short smoke under a 131072 window, and a 104805-token exact needle under that longer window. Throughput comes from server timing. Completion lengths, needle depth, per-gate sampling and repeated-run dispersion are absent. A short smoke with a large configured ceiling is not a full-window input test. Sources: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md, https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/methodology.md ## Which memory observations are paired with which gate? Startup, peak RSS, minimum MemAvailable, swap and peak GPU memory are reported for the completed 32K monitoring gate. The longer 104805-token gate has no paired peak telemetry in these selected documents. The MemAvailable sampling boundary is unspecified, so its relationship to the WSL ceiling cannot be assumed. Keep original KiB and MiB labels and do not sum overlapping allocations. Sources: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md, https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/limitations.md ## What happened to speculation? MTP n=4 and canonical n=1 passed the tested correctness checks but failed speed promotion on this particular CPU-expert split. The canonical MTP short gates reached 31775 MiB peak GPU memory. Ngram speculative decoding was not run because its prerequisite failed; it has no measured benefit here. This is separate from the Q8 n-gram embedding table present in the artifact. Sources: https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md, https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/limitations.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "32K needle gate prefill_rate", "metric": "prefill_rate", "value": 271.64, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "workload": "32K needle gate", "memory_cohort": "completed 32K monitoring" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 53, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L53" }, { "label": "32K needle gate decode_rate", "metric": "decode_rate", "value": 16.53, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "workload": "32K needle gate", "memory_cohort": "completed 32K monitoring" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 53, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L53" }, { "label": "128K server smoke prefill_rate", "metric": "prefill_rate", "value": 17.02, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 61, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K server smoke", "memory_cohort": "no paired 128K telemetry published" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 54, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L54" }, { "label": "128K server smoke decode_rate", "metric": "decode_rate", "value": 15.26, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 61, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K server smoke", "memory_cohort": "no paired 128K telemetry published" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 54, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L54" }, { "label": "128K long-context needle prefill_rate", "metric": "prefill_rate", "value": 245.72, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 104805, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K long-context needle", "memory_cohort": "no paired 128K telemetry published" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 55, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L55" }, { "label": "128K long-context needle decode_rate", "metric": "decode_rate", "value": 13.94, "unit": "tokens/s", "statistic": "one reported completed gate; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 104805, "context_limit_tokens": 131072, "cache_state": null, "workload": "128K long-context needle", "memory_cohort": "no paired 128K telemetry published" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 55, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L55" }, { "label": "32K telemetry startup", "metric": "startup", "value": 25.15, "unit": "s", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "startup_duration", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57" }, { "label": "32K telemetry host_rss", "metric": "host_rss", "value": 53901956, "unit": "KiB", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "peak_host_RSS", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57" }, { "label": "32K telemetry minimum_memavailable", "metric": "minimum_memavailable", "value": 86701016, "unit": "KiB", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "minimum_MemAvailable_in_reported_monitor_scope", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57" }, { "label": "32K telemetry swap_use", "metric": "swap_use", "value": 63708, "unit": "KiB", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "maximum_swap_use", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57" }, { "label": "32K telemetry GPU_memory", "metric": "GPU_memory", "value": 25813, "unit": "MiB", "statistic": "source monitored peak/minimum; no distribution", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 0, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": 26253, "context_limit_tokens": 32768, "cache_state": null, "memory_cohort": "completed 32K gate only; not paired with the 104805-token gate", "memory_kind": "peak_GPU_memory", "MemAvailable_sampling_scope": null, "VM_ceiling_and_host_MemAvailable_same_scope_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 57, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L57" }, { "label": "canonical MTP n1 peak GPU memory", "metric": "GPU_memory", "value": 31775, "unit": "MiB", "statistic": "maximum in completed MTP short gates", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF", "model_revision": "4a768bf4458a83e1c80667b4d818ec721080546e", "artifact_SHA256": "5a4236a6eeda73d23c151f8ed19e5457b0bf1935e1c02c1d42d98a0e26dadde2", "engine": "ik_llama.cpp", "runtime_execution_revision": "d854dfd36a7262fc46cb7542716b04a9f6d4b47d", "MTP_patch_head": "108dfaa77fdaa138c46b043bfff0e94fbe2adff7", "system": "single RTX 5090 32607 MiB; 16-core/32-thread desktop CPU; 128 GiB-class host; Ubuntu WSL2 with 96 GiB VM ceiling", "node_count": 1, "concurrency": 1, "configured_max_requests": 1, "weight_precision": "MXFP4 GGUF, Q8 n-gram table", "CPU_expert_layers": 38, "defer_experts": true, "mmap": true, "GGML_CUDA_NO_PINNED": true, "kv_dtype": "Q8_0", "draft_tokens": 1, "ngram_speculation": false, "ngram_embedding_table_present": true, "temperature": null, "top_p": null, "thinking": null, "output_tokens": null, "actual_output_tokens": null, "repetitions": null, "experiment_date": null, "throughput_origin": "server timing fields, not a client estimate", "input_tokens": null, "context_limit_tokens": 131072, "cache_state": null, "memory_cohort": "canonical MTP n1 short gates; not baseline or long-context peak", "memory_kind": "peak_GPU_memory" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 61, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L61" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "qwen5090-method", "parent_record_id": "qwen38-flash-next-5090-offload-gates", "publisher": "kutaelee", "repository": "kutaelee/qwen38-flash-next-5090-128k-runtime-recipe", "revision": "fca75dd108ce04b6a79db3b45934ed58bba893bb", "document_path": "docs/methodology.md", "pinned_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/methodology.md", "document_sha256": "54d61a37de6873bcebe34c2ad34978a9b0a7191232fb865c364540cdd3e4c706", "checked_on": "2026-10-18" }, { "document_id": "qwen5090-limits", "parent_record_id": "qwen38-flash-next-5090-offload-gates", "publisher": "kutaelee", "repository": "kutaelee/qwen38-flash-next-5090-128k-runtime-recipe", "revision": "fca75dd108ce04b6a79db3b45934ed58bba893bb", "document_path": "docs/limitations.md", "pinned_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/docs/limitations.md", "document_sha256": "029f3cfc9a4b8e2e03eba92137e4df3ca722782b13af4dd990295a8fe1c270d2", "checked_on": "2026-10-18" }, { "document_id": "qwen5090-config", "parent_record_id": "qwen38-flash-next-5090-offload-gates", "publisher": "kutaelee", "repository": "kutaelee/qwen38-flash-next-5090-128k-runtime-recipe", "revision": "fca75dd108ce04b6a79db3b45934ed58bba893bb", "document_path": "configs/ik-llama-baseline-128k.example.sh", "pinned_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/configs/ik-llama-baseline-128k.example.sh", "document_sha256": "fc5de996c4e03b80de59f21597ae543a37ff829822641bf2117426b9c6f09f28", "checked_on": "2026-10-18" } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "unpaired_memory_telemetry", "measured_context_limit_tokens": 32768, "measured_input_tokens": 26253, "longer_needle_input_tokens": 104805, "longer_needle_peak_RAM": null, "host_MemAvailable_scope": null, "CPU_expert_layers": 38, "source_GGUF_bytes": 126846153856 } ] } ``` ## long context cases ```json { "long_context_cases": [ { "classification": "source_reported_exact_needle_success", "actual_input_tokens": 26253, "configured_limit_tokens": 32768, "actual_output_tokens": null, "exact_match": true, "needle_depth_fraction": null, "general_reasoning_quality_proven": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 53, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L53" }, { "classification": "source_reported_short_smoke", "actual_input_tokens": 61, "configured_limit_tokens": 131072, "actual_output_tokens": null, "exact_match": true, "needle_depth_fraction": null, "general_reasoning_quality_proven": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 54, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L54" }, { "classification": "source_reported_exact_needle_success", "actual_input_tokens": 104805, "configured_limit_tokens": 131072, "actual_output_tokens": null, "exact_match": true, "needle_depth_fraction": null, "general_reasoning_quality_proven": false, "source_document_id": "qwen5090-readme", "source_document_sha256": "0668bd3d2e2ca6ed48b4ef13e9d9c56e4ce5c37c45ebbf8dc54e359bd9575de0", "source_line": 55, "evidence_url": "https://github.com/kutaelee/qwen38-flash-next-5090-128k-runtime-recipe/blob/fca75dd108ce04b6a79db3b45934ed58bba893bb/README.md#L55" } ] } ``` --- # Qwen Long Context: Concurrency, Memory and Failure Evidence Read 128K and 490K Qwen batches, a four-GPU 128K matrix, sampled peak VRAM and a Mamba crash with original versions and measurement limits. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: RadixArk/Qwen3.8-Flash-Next-NVFP4 - Original author: jpezzulli - Original project: https://github.com/jpezzulli/sglang-rtxpro6000 - Pinned document: https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md - Source revision: ca9be12f9be920e916d02954a41702a981b18823 - Source commit time (not experiment date): 2026-10-17T16:41:23Z - Checked: 2026-10-18 - Experiment date: 2026-09-11 - Runtime topics: SGLang, vLLM - Hardware topics: nvidia - Reported setup: {"system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "kv_dtype": "FP8", "context_limit_tokens": 524288, "experiment_date": "2026-09-11"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "128K C4 option off", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L231", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K four-request three-key retrieval", "input_length_source_label": "128K", "concurrency": 4, "precision_option": "option off", "actual_output_tokens_group": 15725, "group_wall_seconds": 42.833, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": null, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false } }, { "label": "128K C4 online FP8", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L231", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K four-request three-key retrieval", "input_length_source_label": "128K", "concurrency": 4, "precision_option": "online FP8", "actual_output_tokens_group": 15898, "group_wall_seconds": 37.67, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": 30.152, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false } }, { "label": "490K C4 option off", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L232", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K four-request three-key retrieval", "input_length_source_label": "490K", "concurrency": 4, "precision_option": "option off", "actual_output_tokens_group": 15455, "group_wall_seconds": 65.3, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": null, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false } }, { "label": "490K C4 online FP8", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L232", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K four-request three-key retrieval", "input_length_source_label": "490K", "concurrency": 4, "precision_option": "online FP8", "actual_output_tokens_group": 15127, "group_wall_seconds": 45.977, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": 24.85, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false } }, { "label": "128K online FP8 C1 decode_rate", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L229", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "128K", "metric_denominator": "client generation after first token, excluding TTFT" } }, { "label": "128K online FP8 C1 ttft", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L229", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "128K", "metric_denominator": "client generation after first token, excluding TTFT" } }, { "label": "490K online FP8 C1 decode_rate", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L230", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "490K", "metric_denominator": "client generation after first token, excluding TTFT" } }, { "label": "490K online FP8 C1 ttft", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L230", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "490K", "metric_denominator": "client generation after first token, excluding TTFT" } }, { "label": "Post-graph free VRAM", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L243", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "concurrency": null, "memory_kind": "post_graph_available_not_peak_used", "memory_cohort": "startup", "paired_long_text_batch_peak": false } }, { "label": "Media-window minimum free VRAM", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L263", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "concurrency": null, "memory_kind": "sampled_minimum_free_not_per_long_text_batch_peak", "memory_cohort": "ten selected GPU-media checks; ordinary/large JPEG, concurrent images and video history", "paired_long_text_batch_peak": false } }, { "label": "3.0 cold 64K prefill_rate", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L44", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 63921, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "native mean input rate" } }, { "label": "3.0 cold 64K ttft", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L44", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 63921, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "client first-token latency" } }, { "label": "3.0 cold 490K prefill_rate", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L45", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 489915, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "native mean input rate" } }, { "label": "3.0 cold 490K ttft", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L45", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 489915, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "client first-token latency" } }, { "label": "3.0 short C1 native mean", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L26", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": 0.8, "top_p": 0.95, "thinking": "xhigh", "repetitions": 3, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 128, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "regular warmed screens", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "top_k": 40, "min_p": 0.05, "metric_denominator": "elapsed-weighted complete active native log intervals; excludes idle/prefill crossing and partial tail; aggregate if C4" } }, { "label": "3.0 short C4 native mean", "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L27", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": 0.8, "top_p": 0.95, "thinking": "xhigh", "repetitions": 3, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 128, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "regular warmed screens", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 4, "top_k": 40, "min_p": 0.05, "metric_denominator": "elapsed-weighted complete active native log intervals; excludes idle/prefill crossing and partial tail; aggregate if C4" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document and its selected result points are not independent experiment counts. ## Limits - No independent Run AI Here model test. - Executed identity, timing definition, cache state, repetition unit and missingness must be read per result. - No cross-hardware or unmatched model score ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer Four submitted requests do not guarantee four running requests. The sources contain completed long-context batches, but their cache states, timing denominators and memory windows differ. Keep the four-GPU campaign peak separate from per-cell latency, and retain the original scheduler failure. By Run AI Here editorial · 2026-10-18 Local Spark Qwen writing draft; Codex source audit and rewrite. No site model benchmark. ## Which long-context batches were measured? Penny Royal v2.5.0 tests one RTX PRO 6000 at TP1 with RadixArk NVFP4, RAM PLE, FP8 KV, FR-Spec and an 824384-token pool. Selected 128K and 490K C4 results divide all output by synchronized group wall time. Each cell is one batch, with stop or a 4096-token cap and differing completion totals. The online-FP8 arm records actual four-way overlap; exact prefix sharing and full checkpoint revision remain unreported here. Sources: https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md ## What does the newer release add? October 10 Penny Royal 3.0 has separate short regular screens and actual 63921/489915-token cold prompts. It uses another checkpoint, an adaptive policy and a larger token pool. The regular rate is an elapsed-weighted native active-interval mean, not whole task wall time. Public accepted source and local diagnostic checkout are separately identified. These cells do not replace the older long C4 experiment. Sources: https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md, https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.json ## How are memory and sustained-load evidence bounded? The external four-GPU report supplies a sampled summed used-VRAM peak for its entire matrix, not a per-cell peak or host peak. Penny Royal post-graph free VRAM and minimum free VRAM in a GPU-media window are different observations. A separate credited field report submits four approximately 240K requests over ten rounds, with queuing and 57 percent peak pool occupancy; HTTP 200 alone is not retrieval correctness. Sources: https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell, https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md, https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md ## Which failure remains useful? A first-person SGLang incident has approximately 140K cold input, HiCache write-through, 51 Mamba slots and only two running requests when Mamba usage reaches 1.00. The scheduler asserts while full token usage is 0.46. PR37619 is still open at this check; its CPU tests and proposed fallback do not establish that the GPU reproducer or all affected workloads are fixed. Sources: https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113 ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "128K C4 option off", "metric": "aggregate_output_rate", "value": 367.127, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K four-request three-key retrieval", "input_length_source_label": "128K", "concurrency": 4, "precision_option": "option off", "actual_output_tokens_group": 15725, "group_wall_seconds": 42.833, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": null, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 231, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L231" }, { "label": "128K C4 online FP8", "metric": "aggregate_output_rate", "value": 422.037, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K four-request three-key retrieval", "input_length_source_label": "128K", "concurrency": 4, "precision_option": "online FP8", "actual_output_tokens_group": 15898, "group_wall_seconds": 37.67, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": 30.152, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 231, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L231" }, { "label": "490K C4 option off", "metric": "aggregate_output_rate", "value": 236.678, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K four-request three-key retrieval", "input_length_source_label": "490K", "concurrency": 4, "precision_option": "option off", "actual_output_tokens_group": 15455, "group_wall_seconds": 65.3, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": null, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 232, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L232" }, { "label": "490K C4 online FP8", "metric": "aggregate_output_rate", "value": 329.01, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K four-request three-key retrieval", "input_length_source_label": "490K", "concurrency": 4, "precision_option": "online FP8", "actual_output_tokens_group": 15127, "group_wall_seconds": 45.977, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": 24.85, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 232, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L232" }, { "label": "128K online FP8 C1 decode_rate", "metric": "decode_rate", "value": 195.634, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "128K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 229, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L229" }, { "label": "128K online FP8 C1 ttft", "metric": "ttft", "value": 12.906, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "128K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 229, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L229" }, { "label": "490K online FP8 C1 decode_rate", "metric": "decode_rate", "value": 172.644, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "490K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 230, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L230" }, { "label": "490K online FP8 C1 ttft", "metric": "ttft", "value": 63.296, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "490K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 230, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L230" }, { "label": "Post-graph free VRAM", "metric": "available_GPU_memory", "value": 7.52, "unit": "GiB", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "concurrency": null, "memory_kind": "post_graph_available_not_peak_used", "memory_cohort": "startup", "paired_long_text_batch_peak": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 243, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L243" }, { "label": "Media-window minimum free VRAM", "metric": "available_GPU_memory", "value": 1897, "unit": "MiB", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "concurrency": null, "memory_kind": "sampled_minimum_free_not_per_long_text_batch_peak", "memory_cohort": "ten selected GPU-media checks; ordinary/large JPEG, concurrent images and video history", "paired_long_text_batch_peak": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 263, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L263" }, { "label": "3.0 cold 64K prefill_rate", "metric": "prefill_rate", "value": 17235.9, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 63921, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "native mean input rate" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 44, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L44" }, { "label": "3.0 cold 64K ttft", "metric": "ttft", "value": 4.71, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 63921, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "client first-token latency" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 44, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L44" }, { "label": "3.0 cold 490K prefill_rate", "metric": "prefill_rate", "value": 8839.75, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 489915, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "native mean input rate" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 45, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L45" }, { "label": "3.0 cold 490K ttft", "metric": "ttft", "value": 60.37, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 489915, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "client first-token latency" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 45, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L45" }, { "label": "3.0 short C1 native mean", "metric": "native_decode_rate", "value": 218.63, "unit": "tokens/s", "statistic": "native mean; three regular repeats, no SD", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": 0.8, "top_p": 0.95, "thinking": "xhigh", "repetitions": 3, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 128, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "regular warmed screens", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "top_k": 40, "min_p": 0.05, "metric_denominator": "elapsed-weighted complete active native log intervals; excludes idle/prefill crossing and partial tail; aggregate if C4" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 26, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L26" }, { "label": "3.0 short C4 native mean", "metric": "native_decode_rate", "value": 489.35, "unit": "tokens/s", "statistic": "native mean; three regular repeats, no SD", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": 0.8, "top_p": 0.95, "thinking": "xhigh", "repetitions": 3, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 128, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "regular warmed screens", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 4, "top_k": 40, "min_p": 0.05, "metric_denominator": "elapsed-weighted complete active native log intervals; excludes idle/prefill crossing and partial tail; aggregate if C4" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 27, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L27" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "penny-readme", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "README.md", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/README.md", "document_sha256": "7645e7297eb1326393e8e4106fb73334980dfe44540b45e51655bc1dadd295ad", "last_commit_at": "2026-10-17T16:41:23Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "penny-community-results", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "COMMUNITY-RESULTS.md", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md", "document_sha256": "2bbe9428f0701dc597dd0417193c12e72157dff2aca3ee6481b3104755a1301c", "last_commit_at": "2026-10-17T16:41:23Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "penny-release-3.0", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "release-3.0.json", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.json", "document_sha256": "1799eca8496bf143fc48bbdc83a4f6ccc34b8773c43083b47e29721b1ceb3dcf", "last_commit_at": "2026-10-17T16:41:23Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "penny-release-3.0.2", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "release-3.0.2.json", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.2.json", "document_sha256": "4843d621fa7f735a4c95d6bc79c9582d465cfaecef6b473689158d8c190920e2", "last_commit_at": "2026-10-17T16:41:23Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "qwen-cstech-4gpu", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "CS Technology Services", "repository": null, "revision": "snapshot-2026-10-18", "document_path": "qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell", "pinned_url": "https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell", "document_sha256": "b81878ba45f2f6b7a6de06b0bf930bd1dfa3cb497202ed3fcd368efe66065930", "last_commit_at": "2026-08-26", "checked_on": "2026-10-18", "public_URL_is_immutable": false, "revision_kind": "Mutable original report; retrieval snapshot date and SHA256, no Git revision" }, { "document_id": "qwen-mamba-incident", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "leehom0123", "repository": "sgl-project/sglang", "revision": "comment-5606493113", "document_path": "/issues/37619/comments/5606493113", "pinned_url": "https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113", "document_sha256": "d36cc21fa7b9577474ba20c9fe26becbb3c61bab9b9c4a5c421447b9d905c3f4", "last_commit_at": "2026-09-09T18:05:35Z", "checked_on": "2026-10-18", "public_URL_is_immutable": false, "revision_kind": "GitHub issue comment ID; mutable body with saved SHA256" } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "sampled_peak_summed_GPU_used_over_entire_matrix", "value": 374976, "unit": "MiB", "sample_count": 501, "campaign_seconds": 1206, "paired_per_cell_peak": null, "host_peak_RAM": null }, { "kind": "v2.5.0_GPU_media_window_minimum_free", "value": 1897, "unit": "MiB", "long_text_C4_peak": null, "source_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md" } ] } ``` ## long context cases ```json { "long_context_cases": [ { "classification": "source_reported_four_submitted_stress", "recipe_release": "v2.5.0", "contributor": "BESTIA / Prometeus field report; separate contributor from runtime author", "source_url": "https://www.reddit.com/r/BlackwellPerformance/comments/1weaxdz/comment/p9i76fb/", "source_summary_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md", "context_limit_tokens": 524288, "KV_pool_tokens": 1114304, "input_tokens": null, "input_length_source_label": "approximately 240K", "submitted_concurrency": 4, "simultaneous_running_concurrency": null, "rounds": 10, "total_requests": 40, "output_tokens_reported": 8192, "round_seconds_range": [ 93, 107 ], "HTTP_200": 40, "OOMs_reported": 0, "retractions_reported": 0, "aborts_reported": 0, "peak_pool_occupancy_percent": 57, "peak_memory_bytes": null, "queuing_observed": true, "retrieval_correctness": null } ] } ``` ## case reports ```json { "case_reports": [ { "id": "qwen-cstech-128k-matrix", "classification": "source_reported_duration_matrix", "source_url": "https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell", "source_snapshot_sha256": "b81878ba45f2f6b7a6de06b0bf930bd1dfa3cb497202ed3fcd368efe66065930", "public_URL_is_immutable": false, "source_published_on": "2026-08-26", "model": "Qwen/Qwen3.8-Flash-Next-FP8", "model_revision": null, "engine": "vLLM 0.1.dev20073+g8e685d198", "executed_image_digest": null, "runtime_execution_revision": null, "harness": "llm-inference-bench v0.4.29", "system": "4 RTX PRO 6000 Blackwell Max-Q, 300 W/card, Threadripper, PCIe Gen4, no NVLink", "parallelism": "TP4", "context_limit_tokens": 262144, "KV_pool_tokens": 2662752, "configured_max_requests": 256, "draft_tokens": 3, "temperature": null, "top_p": null, "thinking": null, "input_length_source_label": "128k nominal matrix context; exact per-cell prompt token counts unreported", "duration_seconds_per_cell": 30, "output_tokens_cap": 8192, "ignore_eos": true, "effective_concurrency_validated_by_source": true, "cache_state": "duration-mode integrated scout warms shared prefix; matrix-start TTFT includes cache hit and queueing", "cells": [ { "concurrency": 1, "aggregate_output_tokens_per_second": 128.5, "TTFT_seconds": 1.41 }, { "concurrency": 8, "aggregate_output_tokens_per_second": 505.0, "TTFT_seconds": 2.02 }, { "concurrency": 16, "aggregate_output_tokens_per_second": 729.9, "TTFT_seconds": 3.42 } ], "omitted_cell": { "concurrency": 32, "status": "not_run_capacity", "aggregate_output_tokens_per_second": null, "observed_runtime_crash": false }, "memory": { "kind": "sampled_peak_summed_GPU_used_over_entire_matrix", "value": 374976, "unit": "MiB", "sample_count": 501, "campaign_seconds": 1206, "paired_per_cell_peak": null, "host_peak_RAM": null }, "cold_C1": { "actual_input_tokens": 128021, "TTFT_seconds": 24.25, "prefill_tokens_per_second": 5279 }, "per_user_table_is_derived_aggregate_divided_by_streams": true }, { "id": "qwen-mamba-long-prefill-crash", "classification": "source_reported_scheduler_assertion", "source_url": "https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113", "source_snapshot_sha256": "d36cc21fa7b9577474ba20c9fe26becbb3c61bab9b9c4a5c421447b9d905c3f4", "source_updated_at": "2026-09-09T18:05:35Z", "public_URL_is_immutable": false, "model": "Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "engine": "SGLang 0.0.0.dev1+g4ccff141d", "runtime_execution_revision": null, "executed_image_digest": null, "weight_precision": "modelopt_mixed", "kv_dtype": "fp8_e4m3", "context_length_source_label": "262k", "actual_prompt_tokens": null, "prompt_length_source_label": "approximately 140k", "chunked_prefill_size": 4096, "mamba_track_interval": 64, "mamba_slots": 51, "configured_max_requests": 12, "observed_running_requests": 2, "observed_queued_requests_reporter": 1, "queued_requests_at_cited_log_line": 0, "mamba_usage_before": 0.18, "mamba_usage_at_failure": 1.0, "full_token_usage_at_failure": 0.46, "HiCache": true, "hicache_ratio": 2.0, "write_policy": "write_through", "speculation": false, "error": "AssertionError: Can not alloc mamba cache", "outcome": "scheduler child exited; container auto-restarted, service outage during reload", "fix_reference": "https://github.com/sgl-project/sglang/pull/37619", "fix_PR_state": "open", "fix_PR_merged": false, "fix_PR_head_revision": "cd530ac217ac0a67b913f7083679c0890670d228", "author_GPU_reproducer_rerun": false, "universal_fix_verified": false, "peak_RAM": null } ] } ``` --- # GLM-5.3-Flash on Two Sparks: Speed Meets Quality Gates A pinned NVFP4 target, DFlash2 draft and community image reach sustained decode and 120124-token retrieval, while strict output failures remain visible. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: GLM - Model: nvidia/GLM-5.3-Flash-NVFP4 - Original author: Weschera - Original project: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark - Pinned document: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/README.md - Source revision: 281dc9a982bca17e944b322746658250a670ae73 - Source commit time (not experiment date): 2026-09-10T13:54:03Z - Checked: 2026-10-18 - Experiment date: None - Runtime topics: vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "kv_dtype": "fp8_e4m3", "context_limit_tokens": 131072, "experiment_date": null} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "baseline median decode", "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L18", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 0, "drafter": null, "drafter_revision": null, "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 4356, 2355, 10853, 2238 ], "all_speed_requests_natural_stop": true, "quality_passed": false } }, { "label": "dflash2-7 median decode", "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L98", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 7, "drafter": "incoai/GLM-5.3-Flash-DFlash2", "drafter_revision": "bf582e4eacc1810f76656d1811693ff6c6737d2a", "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 7396, 2272, 2292, 2158 ], "all_speed_requests_natural_stop": true, "quality_passed": false } }, { "label": "dflash2-3 median decode", "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L173", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 3, "drafter": "incoai/GLM-5.3-Flash-DFlash2", "drafter_revision": "bf582e4eacc1810f76656d1811693ff6c6737d2a", "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 2822, 2194, 6331, 2105 ], "all_speed_requests_natural_stop": true, "quality_passed": false } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document and its selected result points are not independent experiment counts. ## Limits - No independent Run AI Here model test. - Executed identity, timing definition, cache state, repetition unit and missingness must be read per result. - No cross-hardware or unmatched model score ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer The selected seven-token draft arm is faster than the baseline in this source, but every arm retains quality_passed=false. The fastest three-token diagnostic failed the response contract before code execution. This is a text-only community experiment with useful negative results. By Run AI Here editorial · 2026-10-18 Local Spark Qwen writing draft; Codex source audit and rewrite. No site model benchmark. ## What is actually deployed? The target and DFlash2 draft have separate full weight revisions, and the community image is pinned by digest. NVFP4 storage uses Marlin W4A16 MoE compute on GB10. The text-only TP2/RoCE setup is eager, uses FP8 E4M3 KV with an explicit 6 GiB allocation, a 131072 window and one request slot. It is not an NVIDIA-published GB10 production recipe, and the target export has no MTP head. Sources: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/README.md, https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/SOURCE_MANIFEST.json ## What workload defines the speed values? Each arm has four naturally stopping requests: two English technical tutorials and two Korean essays. Input lengths are 78 or 92 tokens and generated lengths vary greatly. Median estimated decode excludes measured TTFT. One boot per arm and different output texts prevent a matched task-time or uncertainty claim. Saved request controls include temperature zero, top-p one and seed 37. Sources: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json, https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/dflash2-7/speed-prose-0-request.json ## Why keep false quality flags? The depth-7 arm passes the executed semantic checks but still returns a fenced block where bare Python was required and explanation where exactly 703 was required. The baseline and depth-3 arm fail the early one-code-block contract; later code checks are skipped rather than proven algorithmically wrong. All quality_passed flags remain false. Sources: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json, https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/dflash2-7/quality-report.json ## What do context and thinking checks establish? All arms retrieve three beginning/middle/end markers from 120124 actual input tokens. That is smaller than the configured window and is not broad long-document reasoning. The custom forced-OFF template keeps maximum-effort system wording and leaks planning into final text, so native OFF is unqualified. The final restart has a health probe, not a repeated quality suite; whole-model peak RAM is unreported. Sources: https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/README.md, https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/REPRODUCE.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "baseline median decode", "metric": "decode_rate", "value": 15.181243819412654, "unit": "tokens/s", "statistic": "median of four different prompts in one arm boot; not repeated identical task", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 0, "drafter": null, "drafter_revision": null, "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 4356, 2355, 10853, 2238 ], "all_speed_requests_natural_stop": true, "quality_passed": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-weschera-extra3", "source_document_sha256": "36dc46ce255d484340558a64b6b45ce2b79f644471d91cbe6dc0e7a82de2d461", "source_line": 18, "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L18" }, { "label": "dflash2-7 median decode", "metric": "decode_rate", "value": 20.50006471628756, "unit": "tokens/s", "statistic": "median of four different prompts in one arm boot; not repeated identical task", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 7, "drafter": "incoai/GLM-5.3-Flash-DFlash2", "drafter_revision": "bf582e4eacc1810f76656d1811693ff6c6737d2a", "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 7396, 2272, 2292, 2158 ], "all_speed_requests_natural_stop": true, "quality_passed": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-weschera-extra3", "source_document_sha256": "36dc46ce255d484340558a64b6b45ce2b79f644471d91cbe6dc0e7a82de2d461", "source_line": 98, "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L98" }, { "label": "dflash2-3 median decode", "metric": "decode_rate", "value": 24.77753188791761, "unit": "tokens/s", "statistic": "median of four different prompts in one arm boot; not repeated identical task", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "nvidia/GLM-5.3-Flash-NVFP4", "model_revision": "423acf37583782c51c142d145aef733d72943d93", "runtime_execution_revision": null, "executed_image_digest": "sha256:4def0ef644cb2e9814136dcffd5e385e21bc594f48f3b292234051904abe85a6", "runtime_patch_source": "050081dc41ce6edd4d3f15fa19dc3410ba4210e3", "engine": "vLLM community GB10 image with Marlin W4A16 MoE", "system": "two 128 GB DGX Spark GB10, TP2 over RoCE; text only", "node_count": 2, "parallelism": "TP2", "weight_precision": "NVIDIA NVFP4 storage; Marlin W4A16 MoE compute", "kv_dtype": "fp8_e4m3", "KV_allocation_GiB": 6, "context_limit_tokens": 131072, "configured_max_requests": 1, "concurrency": 1, "prefill_batch_tokens": 128, "enforce_eager": true, "cache_state": "prefix caching disabled", "thinking": "custom forced-OFF template; does not qualify as native OFF", "temperature": 0, "top_p": 1, "seed": 37, "repetitions": "four different sustained prompts in one boot per arm", "experiment_date": null, "metric_denominator": "completion tokens / (wall minus measured TTFT); estimated decode", "output_tokens": null, "actual_output_tokens": null, "peak_runtime_RAM": null, "draft_tokens": 3, "drafter": "incoai/GLM-5.3-Flash-DFlash2", "drafter_revision": "bf582e4eacc1810f76656d1811693ff6c6737d2a", "input_tokens": [ 78, 92 ], "actual_completion_tokens_by_prompt": [ 2822, 2194, 6331, 2105 ], "all_speed_requests_natural_stop": true, "quality_passed": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-weschera-extra3", "source_document_sha256": "36dc46ce255d484340558a64b6b45ce2b79f644471d91cbe6dc0e7a82de2d461", "source_line": 173, "evidence_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json#L173" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "glm-weschera-extra0", "parent_record_id": "glm53-flash-nvfp4-dflash-quality", "publisher": "Weschera", "repository": "Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark", "revision": "281dc9a982bca17e944b322746658250a670ae73", "document_path": "docs/RESULTS.md", "pinned_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/docs/RESULTS.md", "document_sha256": "ff3e50dd4f7a65232e37c01e4dd79ca5bee10c9ba66dbbec0f2cf1a7b5a4c3d5", "last_commit_at": "2026-09-10T13:54:03Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "glm-weschera-extra1", "parent_record_id": "glm53-flash-nvfp4-dflash-quality", "publisher": "Weschera", "repository": "Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark", "revision": "281dc9a982bca17e944b322746658250a670ae73", "document_path": "REPRODUCE.md", "pinned_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/REPRODUCE.md", "document_sha256": "89dd3e9334b41901f2ae8dc1d92ec8437e2ed90c143e4026c28c0978242f13c2", "last_commit_at": "2026-09-10T13:54:03Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "glm-weschera-extra2", "parent_record_id": "glm53-flash-nvfp4-dflash-quality", "publisher": "Weschera", "repository": "Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark", "revision": "281dc9a982bca17e944b322746658250a670ae73", "document_path": "SOURCE_MANIFEST.json", "pinned_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/SOURCE_MANIFEST.json", "document_sha256": "7065d7de93ef78feb6023048fefde4506c097148cfc2733bfe65257bcbac1692", "last_commit_at": "2026-09-10T13:54:03Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "glm-weschera-extra3", "parent_record_id": "glm53-flash-nvfp4-dflash-quality", "publisher": "Weschera", "repository": "Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark", "revision": "281dc9a982bca17e944b322746658250a670ae73", "document_path": "evidence/text128-dflash-resume/comparison.json", "pinned_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/comparison.json", "document_sha256": "36dc46ce255d484340558a64b6b45ce2b79f644471d91cbe6dc0e7a82de2d461", "last_commit_at": "2026-09-10T13:54:03Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "glm-weschera-extra5", "parent_record_id": "glm53-flash-nvfp4-dflash-quality", "publisher": "Weschera", "repository": "Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark", "revision": "281dc9a982bca17e944b322746658250a670ae73", "document_path": "evidence/text128-dflash-resume/dflash2-7/quality-report.json", "pinned_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/dflash2-7/quality-report.json", "document_sha256": "239adbba20e74bd853f04af15301d6c5b4254512b5ee9e4b79a532b64bac0957", "last_commit_at": "2026-09-10T13:54:03Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "glm-method-0", "parent_record_id": "glm53-flash-nvfp4-dflash-quality", "publisher": "Weschera", "repository": "Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark", "revision": "281dc9a982bca17e944b322746658250a670ae73", "document_path": "evidence/text128-dflash-resume/dflash2-7/speed-prose-0-request.json", "pinned_url": "https://github.com/Weschera/GLM-5.3-Flash-NVIDIA-NVFP4-2x-DGX-Spark/blob/281dc9a982bca17e944b322746658250a670ae73/evidence/text128-dflash-resume/dflash2-7/speed-prose-0-request.json", "document_sha256": "46e917cee3ce542c4b710c92cdb0f22fa98281177d0405f94e57af3f6fd89ff6", "last_commit_at": null, "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" } ] } ``` ## Paired source quality comparisons ```json { "quality_comparisons": [ { "variant": "baseline", "speculative_tokens": 0, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": false, "long_context_passed": true, "strict_failures": [ { "probe": "short-quality", "category": "semantic", "passed": false, "error": "Expected one code block" } ] }, { "variant": "dflash2-7", "speculative_tokens": 7, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": true, "long_context_passed": true, "strict_failures": [ { "probe": "code", "requirement": "Only Python code, no Markdown fence", "passed": false }, { "probe": "thinking-on", "requirement": "Final content exactly 703", "passed": false, "observed": "**Reasoning:** The sum of the first n positive integers is given by the formula n(n+1)/2. For n = 37:\n\nSum = 37 × 38 / 2 = 37 × 19 = 703\n\n**Answer:** 703" } ] }, { "variant": "dflash2-3", "speculative_tokens": 3, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": false, "long_context_passed": true, "strict_failures": [ { "probe": "short-quality", "category": "semantic", "passed": false, "error": "Expected one code block" } ] } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "explicit_KV_allocation", "value": 6, "unit": "GiB", "whole_model_peak_RAM": null } ] } ``` ## long context cases ```json { "long_context_cases": [ { "classification": "source_reported_three_marker_success", "actual_input_tokens": 120124, "configured_limit_tokens": 131072, "arms": [ "baseline", "dflash2-7", "dflash2-3" ], "exact_all_three_markers": true, "full_window_tested": false, "general_long_context_reasoning_proven": false } ] } ``` ## quality comparisons ```json { "quality_comparisons": [ { "variant": "baseline", "speculative_tokens": 0, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": false, "long_context_passed": true, "strict_failures": [ { "probe": "short-quality", "category": "semantic", "passed": false, "error": "Expected one code block" } ] }, { "variant": "dflash2-7", "speculative_tokens": 7, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": true, "long_context_passed": true, "strict_failures": [ { "probe": "code", "requirement": "Only Python code, no Markdown fence", "passed": false }, { "probe": "thinking-on", "requirement": "Final content exactly 703", "passed": false, "observed": "**Reasoning:** The sum of the first n positive integers is given by the formula n(n+1)/2. For n = 37:\n\nSum = 37 × 38 / 2 = 37 × 19 = 703\n\n**Answer:** 703" } ] }, { "variant": "dflash2-3", "speculative_tokens": 3, "quality_passed": false, "runtime_checks_passed": true, "semantic_checks_passed": false, "long_context_passed": true, "strict_failures": [ { "probe": "short-quality", "category": "semantic", "passed": false, "error": "Expected one code block" } ] } ] } ``` --- # MiniMax-M3 on Two Sparks: The Measured W4A16 Lane Separate a July lane-A reproduction from proposed speed/context recipes, retaining request timing, startup memory brackets and a negative NCCL result. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: MiniMax - Model: Sebesky/MiniMax-M3-W4A16-GPTQ - Original author: tonyd2wild - Original project: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s - Pinned document: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/README.md - Source revision: a8101fa423cac8140016e45110a2ccc8a0a72162 - Source commit time (not experiment date): 2026-07-13T01:19:48Z - Checked: 2026-10-18 - Experiment date: 2026-07-05 - Runtime topics: vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "kv_dtype": "b12x NVFP4", "context_limit_tokens": 196608, "experiment_date": "2026-07-05"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "JSON thinking False", "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L38", "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "JSON", "thinking": false } }, { "label": "JSON thinking adaptive", "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L38", "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "JSON", "thinking": "adaptive" } }, { "label": "code thinking False", "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L39", "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "code", "thinking": false } }, { "label": "code thinking adaptive", "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L39", "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "code", "thinking": "adaptive" } }, { "label": "Lane A KV pool", "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L31", "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "thinking": null } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document and its selected result points are not independent experiment counts. ## Limits - No independent Run AI Here model test. - Executed identity, timing definition, cache state, repetition unit and missingness must be read per result. - No cross-hardware or unmatched model score ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer Only lane A is reproduced by this reporter: W4A16 GPTQ with EAGLE3 and NVFP4 KV on two Sparks. Its four short workload observations use whole request wall time. Lanes B and C remain another author’s references under test, and the source supplies no quality score or measured peak RAM. By Run AI Here editorial · 2026-10-18 Local Spark Qwen writing draft; Codex source audit and rewrite. No site model benchmark. ## How recent are these particular measurements? The community report verifies lane A on July 5, 2026, with two 128 GB DGX Sparks, TP2 and direct 200G RoCE. Keep that date even when reading it as a current MiniMax-family deployment reference. The target is Sebesky W4A16 GPTQ, the draft is EAGLE3 RTN INT4 and the engine uses b12x; exact executed image digest and full weight revisions are not supplied. Sources: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/README.md, https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md ## Which numbers were reproduced? Lane A uses three draft tokens, NVFP4 KV, a 196608-token context ceiling and a 208128-token pool. JSON and code probes are single stream, temperature zero and warm engine, with 256 requested output tokens. The reported denominator is completion_tokens divided by whole request wall. Thinking off and adaptive values remain separate workloads; exact inputs, actual completion counts and dispersion are missing. Sources: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md ## Why is the startup bracket informative? GMU 0.93 misses startup allocation by 0.02 GiB in this boot state; 0.925 starts with insufficient KV for the target context; 0.929 reaches the desired pool with about 100 MB startup margin. These are allocation observations rather than a peak-RAM measurement or a universal safe fraction. High-context and concurrent transient demand remain unmeasured. Sources: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/DEFAULT-CONFIG.md, https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md ## Which experiments are unfinished or negative? The alternative speed and context lanes are explicitly under test by this reporter and retain their original author attribution. The LL protocol plus two NCCL channels reduces code rate from 31.8 to 28.8 in the tested lane. The proposed model-class explanation is a hypothesis. No concurrency-above-one sweep or general quality score is present. Sources: https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "JSON thinking False", "metric": "request_output_rate", "value": 36.6, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "JSON", "thinking": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 38, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L38" }, { "label": "JSON thinking adaptive", "metric": "request_output_rate", "value": 35.4, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "JSON", "thinking": "adaptive" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 38, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L38" }, { "label": "code thinking False", "metric": "request_output_rate", "value": 31.8, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "code", "thinking": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 39, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L39" }, { "label": "code thinking adaptive", "metric": "request_output_rate", "value": 29.2, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "workload": "code", "thinking": "adaptive" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 39, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L39" }, { "label": "Lane A KV pool", "metric": "KV_pool", "value": 208128, "unit": "tokens", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "MiniMax-M3", "model_artifact": "Sebesky/MiniMax-M3-W4A16-GPTQ", "model_revision": null, "runtime_execution_revision": null, "runtime_source_prefix": "979b56a66c96", "executed_image_digest": null, "image_tag": "vllm-node-minimax-m3-b12x", "engine": "vLLM + b12x", "system": "two DGX Spark GB10, TP2 over direct 200G RoCE", "node_count": 2, "parallelism": "TP2", "weight_precision": "W4A16 GPTQ", "kv_dtype": "b12x NVFP4", "drafter": "Sebesky/MiniMax-M3-EAGLE3-RTN-INT4", "drafter_revision": null, "draft_tokens": 3, "context_limit_tokens": 196608, "KV_pool_tokens": 208128, "configured_max_requests": 1, "concurrency": 1, "input_tokens": null, "output_tokens": 256, "actual_output_tokens": null, "temperature": 0, "top_p": null, "cache_state": "warm engine; first JIT request excluded", "repetitions": null, "experiment_date": "2026-07-05", "GMU": 0.929, "quality_score": null, "peak_runtime_RAM": null, "metric_denominator": "completion_tokens / whole request wall including prefill", "thinking": null }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "minimax-tp2-extra1", "source_document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "source_line": 31, "evidence_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md#L31" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "minimax-tp2-extra0", "parent_record_id": "minimax-m3-tp2-lane-a", "publisher": "tonyd2wild", "repository": "tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s", "revision": "a8101fa423cac8140016e45110a2ccc8a0a72162", "document_path": "DEFAULT-CONFIG.md", "pinned_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/DEFAULT-CONFIG.md", "document_sha256": "04d26375babfc741c02dac6d1d8c0f0004152284f4843c2ca1efe997d2282287", "last_commit_at": "2026-07-13T01:19:48Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "minimax-tp2-extra1", "parent_record_id": "minimax-m3-tp2-lane-a", "publisher": "tonyd2wild", "repository": "tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s", "revision": "a8101fa423cac8140016e45110a2ccc8a0a72162", "document_path": "benchmarks/RESULTS.md", "pinned_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/benchmarks/RESULTS.md", "document_sha256": "9f3d73849567604e7dc04c19900c1e26343fb1193efb7b5df35fa2ade9838cad", "last_commit_at": "2026-07-13T01:19:48Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "KV_pool_tokens_not_peak_RAM", "value": 208128, "unit": "tokens", "configured_limit_tokens": 196608, "whole_model_peak_RAM": null } ] } ``` ## case reports ```json { "case_reports": [ { "id": "minimax-lane-a-startup-bracket", "classification": "source_reported_startup_capacity_failure", "GMU_failed": 0.93, "shortfall_GiB": 0.02, "GMU_boots_but_KV_below_requested": 0.925, "GMU_selected": 0.929, "startup_margin_source_label": "approximately 100 MB", "runtime_peak_RAM": null, "source_url": "https://github.com/tonyd2wild/MiniMax-M3-2x-DGX-Spark-36-tok-s/blob/a8101fa423cac8140016e45110a2ccc8a0a72162/DEFAULT-CONFIG.md" } ] } ``` --- # Kimi IQ1_S on Three Sparks: Scores, Retries and Missing Pins Audit a publisher-labelled K3 layer-split deployment, approximate memory and small GSM8K/HumanEval subsets, retaining first-pass failures and timeout retries. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Kimi - Model: k3-neuron-iq1s, nine GGUF shards - Original author: chishiki37 - Original project: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark - Pinned document: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md - Source revision: de1fa14ea798512f54dafc8b8795eb3ef1a6c367 - Source commit time (not experiment date): 2026-08-07T15:45:04Z - Checked: 2026-10-18 - Experiment date: 2026-08-07 - Runtime topics: llama.cpp - Hardware topics: spark, nvidia - Reported setup: {"system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "kv_dtype": "f16", "context_limit_tokens": 1048576, "experiment_date": "2026-08-07"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "steady decode", "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L43", "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "steady decode", "input_length_source_label": null } }, { "label": "8K prefill", "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L44", "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "8K prefill", "input_length_source_label": "8K" } }, { "label": "32K prefill", "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L45", "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "32K prefill", "input_length_source_label": "32K" } }, { "label": "131K prefill", "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L46", "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "131K prefill", "input_length_source_label": "131K" } }, { "label": "GSM8K local subset", "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json#L5", "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "GSM8K canonical test subset", "correct": 49, "total": 50, "subset_ids": "50 source-indexed questions; fixed subset, not full benchmark", "timeout_seconds": 1800, "retry_policy": "GLM comparison changed output cap; local retries unreported" } }, { "label": "HumanEval initial local subset", "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json#L6", "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval subset IDs 0 through 19", "correct": 14, "total": 20, "subset_ids": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "timeout_seconds": 600, "timeouts": 4, "retry_policy": "initial run; four timeouts retained as failures" } }, { "label": "HumanEval timeout retries", "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json#L6", "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval timeout-only subset", "correct": 4, "total": 4, "subset_ids": [ 1, 6, 10, 17 ], "timeout_seconds": null, "retry_policy": "rerun only four timed-out problems with higher limits; not fresh 20-problem pass" } }, { "label": "HumanEval local after selected retries", "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L83", "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval subset IDs 0 through 19", "correct": 18, "total": 20, "subset_ids": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "timeout_seconds": null, "retry_policy": "14 initial passes plus 4 successful timeout-only retries; not first-pass 18/20", "failed_ids": [ 2, 7 ] } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - A source document and its selected result points are not independent experiment counts. ## Limits - No independent Run AI Here model test. - Executed identity, timing definition, cache state, repetition unit and missingness must be read per result. - No cross-hardware or unmatched model score ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer The source reports 49/50 on a GSM8K subset and a HumanEval composite of 18/20 after retrying four timed-out problems. The initial HumanEval result was 14/20. Unequal cloud sampling, unpinned weight identity and small subsets do not prove lossless IQ1_S quantization or full 1M-input capability. By Run AI Here editorial · 2026-10-18 Local Spark Qwen writing draft; Codex source audit and rewrite. No site model benchmark. ## Which identity and topology are known? The August 7 publisher labels the artifact Kimi K3/Kimi-Linear IQ1_S with nine GGUF shards. No exact upstream checkpoint revision or weight hash is given, so the label is not an independently verified official artifact. Three DGX Sparks use layer-split RPC through a pwilkin llama.cpp fork, not TP3; the actual transport configuration uses MTU4096, while the fabric overview separately says MTU9000. Sources: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md ## What is measured versus configured? The report gives approximate steady decode and prefill summaries through a 131K-labelled input. The 1048576-token window and one slot are configured limits. About 119–120 GB per node is an approximate occupancy report with no peak-telemetry method; it cannot establish peak headroom at a completed million-token input. Exact engine commit and runtime image are also absent. Sources: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md ## How are scores and retries retained? The local GSM8K subset has 49 correct of 50. Raw HumanEval results show 14 correct of 20 initially, including four 600-second timeouts. The selected timeout-only rerun passes all four, producing the reported composite 18/20. The two remaining assertion failures are IDs2 and7. Do not turn the composite into a first-pass result or call every failure a floating-point issue. Sources: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json, https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json, https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json ## Why is this not a quantization equivalence test? Cloud Kimi uses temperature1 and a smaller GSM8K output cap while local uses temperature0.2; the GLM comparison raises its cap after output exhaustion. These are small fixed subsets with no unquantized local control. Matching selected scores and failure IDs does not prove equivalence, zero quantization cost or a general model ranking. No standardized full-suite leaderboard score is created here. Sources: https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md, https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/cloud-humaneval-logic-k3-cloud.json, https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-cloud.json ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "steady decode", "metric": "decode_rate", "value": 2.9, "unit": "tokens/s", "statistic": "approximate source summary", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "steady decode", "input_length_source_label": null }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 43, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L43" }, { "label": "8K prefill", "metric": "prefill_rate", "value": 79.6, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "8K prefill", "input_length_source_label": "8K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 44, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L44" }, { "label": "32K prefill", "metric": "prefill_rate", "value": 43.9, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "32K prefill", "input_length_source_label": "32K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 45, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L45" }, { "label": "131K prefill", "metric": "prefill_rate", "value": 48.5, "unit": "tokens/s", "statistic": "source summary; exact per-run token counts not supplied", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": null, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "workload": "131K prefill", "input_length_source_label": "131K" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 46, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L46" }, { "label": "GSM8K local subset", "metric": "accuracy", "value": 98.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "GSM8K canonical test subset", "correct": 49, "total": 50, "subset_ids": "50 source-indexed questions; fixed subset, not full benchmark", "timeout_seconds": 1800, "retry_policy": "GLM comparison changed output cap; local retries unreported" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra0", "source_document_sha256": "2d297e3f59c7beb393fa72e6297edec8af0b6525139057ee0e26b425d071f2be", "source_line": 5, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json#L5" }, { "label": "HumanEval initial local subset", "metric": "accuracy", "value": 70.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval subset IDs 0 through 19", "correct": 14, "total": 20, "subset_ids": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "timeout_seconds": 600, "timeouts": 4, "retry_policy": "initial run; four timeouts retained as failures" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra1", "source_document_sha256": "29a942e4a2194c737b8492b59ab752015896fc8e3180cc85135c33c6b249f590", "source_line": 6, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json#L6" }, { "label": "HumanEval timeout retries", "metric": "accuracy", "value": 100.0, "unit": "percent", "statistic": "source fixed-subset score, no confidence interval", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval timeout-only subset", "correct": 4, "total": 4, "subset_ids": [ 1, 6, 10, 17 ], "timeout_seconds": null, "retry_policy": "rerun only four timed-out problems with higher limits; not fresh 20-problem pass" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-extra2", "source_document_sha256": "469ff6a70092ca219c70e3dc2d6967729133f74e97ed0b9cd56e68323b2f2053", "source_line": 6, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json#L6" }, { "label": "HumanEval local after selected retries", "metric": "accuracy", "value": 90.0, "unit": "percent", "statistic": "source composite score after retries", "uncertainty": null, "conditions": { "model": "Kimi K3 IQ1_S (publisher label; exact upstream checkpoint not pinned)", "model_artifact": "k3-neuron-iq1s, nine GGUF shards", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "pwilkin llama.cpp fork, build-rdma", "system": "three DGX Spark GB10, 128 GB unified memory each", "node_count": 3, "parallelism": "layer-split RPC; two remote workers, not TP3", "weight_precision": "IQ1_S", "kv_dtype": "f16", "context_limit_tokens": 1048576, "configured_max_requests": 1, "concurrency": 1, "load_mode": "none", "actual_transport_MTU": 4096, "fabric_overview_MTU": 9000, "temperature": 0.2, "top_p": null, "thinking": null, "input_tokens": null, "output_tokens": 8192, "actual_output_tokens": null, "cache_state": null, "repetitions": null, "experiment_date": "2026-08-07", "peak_runtime_RAM": null, "benchmark": "HumanEval subset IDs 0 through 19", "correct": 18, "total": 20, "subset_ids": [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ], "timeout_seconds": null, "retry_policy": "14 initial passes plus 4 successful timeout-only retries; not first-pass 18/20", "failed_ids": [ 2, 7 ] }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "kimi-iq1s-readme", "source_document_sha256": "4bccb5ffc313f08ca53f62b6c8fc880c1317c07f79fed53c522bfc8e99bea20d", "source_line": 83, "evidence_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/README.md#L83" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "kimi-iq1s-extra0", "parent_record_id": "kimi-k3-iq1s-three-spark-scores", "publisher": "chishiki37", "repository": "chishiki37/kimi-k3-iq1s-3node-dgx-spark", "revision": "de1fa14ea798512f54dafc8b8795eb3ef1a6c367", "document_path": "benchmarks/canon-gsm8k-k3-local.json", "pinned_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-local.json", "document_sha256": "2d297e3f59c7beb393fa72e6297edec8af0b6525139057ee0e26b425d071f2be", "last_commit_at": "2026-08-07T15:45:04Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "kimi-iq1s-extra1", "parent_record_id": "kimi-k3-iq1s-three-spark-scores", "publisher": "chishiki37", "repository": "chishiki37/kimi-k3-iq1s-3node-dgx-spark", "revision": "de1fa14ea798512f54dafc8b8795eb3ef1a6c367", "document_path": "benchmarks/humaneval-p18081.json", "pinned_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-p18081.json", "document_sha256": "29a942e4a2194c737b8492b59ab752015896fc8e3180cc85135c33c6b249f590", "last_commit_at": "2026-08-07T15:45:04Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "kimi-iq1s-extra2", "parent_record_id": "kimi-k3-iq1s-three-spark-scores", "publisher": "chishiki37", "repository": "chishiki37/kimi-k3-iq1s-3node-dgx-spark", "revision": "de1fa14ea798512f54dafc8b8795eb3ef1a6c367", "document_path": "benchmarks/humaneval-retry-p18081.json", "pinned_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/humaneval-retry-p18081.json", "document_sha256": "469ff6a70092ca219c70e3dc2d6967729133f74e97ed0b9cd56e68323b2f2053", "last_commit_at": "2026-08-07T15:45:04Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "kimi-iq1s-extra3", "parent_record_id": "kimi-k3-iq1s-three-spark-scores", "publisher": "chishiki37", "repository": "chishiki37/kimi-k3-iq1s-3node-dgx-spark", "revision": "de1fa14ea798512f54dafc8b8795eb3ef1a6c367", "document_path": "benchmarks/cloud-humaneval-logic-k3-cloud.json", "pinned_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/cloud-humaneval-logic-k3-cloud.json", "document_sha256": "8d42fd9033e794d5a8a7330d5c56707128b521f869c46be071a364c5f45fa22c", "last_commit_at": "2026-08-07T15:45:04Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "kimi-iq1s-extra4", "parent_record_id": "kimi-k3-iq1s-three-spark-scores", "publisher": "chishiki37", "repository": "chishiki37/kimi-k3-iq1s-3node-dgx-spark", "revision": "de1fa14ea798512f54dafc8b8795eb3ef1a6c367", "document_path": "benchmarks/canon-gsm8k-k3-cloud.json", "pinned_url": "https://github.com/chishiki37/kimi-k3-iq1s-3node-dgx-spark/blob/de1fa14ea798512f54dafc8b8795eb3ef1a6c367/benchmarks/canon-gsm8k-k3-cloud.json", "document_sha256": "636e0ee2c31986d961ffe637927bcb95aa1d3d82d78dea08f7efe0807f29971b", "last_commit_at": "2026-08-07T15:45:04Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" } ] } ``` ## Paired source quality comparisons ```json { "quality_comparisons": [ { "benchmark": "GSM8K", "subset_size": 50, "local_correct": 49, "cloud_Kimi_correct": 49, "cloud_GLM52_correct": 49, "local_temperature": 0.2, "cloud_Kimi_temperature": 1.0, "local_output_cap": 8192, "cloud_Kimi_output_cap": 2048, "GLM_cap_changed_after_exhaustion": true, "matched_accuracy_proves_equivalence": false }, { "benchmark": "HumanEval", "subset_size": 20, "initial_local_correct": 14, "initial_timeouts": 4, "retry_subset_ids": [ 1, 6, 10, 17 ], "retry_correct": 4, "final_local_correct": 18, "remaining_failure_ids": [ 2, 7 ], "first_pass_18_of_20": false } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "approximate_per_node_occupancy_unknown_measurement_method", "source_range_GB": [ 119, 120 ], "unit": "GB (source label; byte convention unspecified)", "node_count": 3, "is_measured_peak": false, "peak_RAM": null } ] } ``` ## long context cases ```json { "long_context_cases": [ { "classification": "configured_context_not_completed_full_window", "configured_limit_tokens": 1048576, "longest_performance_source_label": "131K", "actual_full_window_prompt_tokens": null } ] } ``` ## quality comparisons ```json { "quality_comparisons": [ { "benchmark": "GSM8K", "subset_size": 50, "local_correct": 49, "cloud_Kimi_correct": 49, "cloud_GLM52_correct": 49, "local_temperature": 0.2, "cloud_Kimi_temperature": 1.0, "local_output_cap": 8192, "cloud_Kimi_output_cap": 2048, "GLM_cap_changed_after_exhaustion": true, "matched_accuracy_proves_equivalence": false }, { "benchmark": "HumanEval", "subset_size": 20, "initial_local_correct": 14, "initial_timeouts": 4, "retry_subset_ids": [ 1, 6, 10, 17 ], "retry_correct": 4, "final_local_correct": 18, "remaining_failure_ids": [ 2, 7 ], "first_pass_18_of_20": false } ] } ``` --- # Qwen Long Concurrency: Paired Memory Peaks and OOM Recovery Read two 250K-request memory controls, a 500K single-request peak, a production OOM and invalidated/repeated recovery tests without mixing hardware or cohorts. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: Qwen - Model: community NVFP4; exact upstream artifact not identified in the selected report - Original author: nanogenomic - Original project: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000 - Pinned document: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md - Source revision: 11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2 - Source commit time (not experiment date): 2026-10-01T04:27:43Z - Checked: 2026-10-18 - Experiment date: 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported - Runtime topics: SGLang - Hardware topics: nvidia - Reported setup: {"system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "kv_dtype": "FP8", "context_limit_tokens": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "on-GPU control minimum free VRAM", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" } }, { "label": "on-GPU control non-pool peak over idle", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" } }, { "label": "host KV minimum free VRAM", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" } }, { "label": "host KV non-pool peak over idle", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" } }, { "label": "524288 FP8 old gather isolated transient", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "old gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true } }, { "label": "524288 FP8 bounded gather isolated transient", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "bounded gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true } }, { "label": "500K old gather device peak", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "old gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 } }, { "label": "500K bounded gather device peak", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "bounded gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 } }, { "label": "2x250K host-KV old gather non-pool peak", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "old gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false } }, { "label": "2x250K host-KV bounded gather non-pool peak", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "bounded gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false } }, { "label": "Production OOM failed_allocation", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L639", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" } }, { "label": "Production OOM available_GPU_memory", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L640", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" } }, { "label": "Production OOM process_device_memory", "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L642", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Selected values are source excerpts, not independent experiment counts. ## Limits - No independent Run AI Here model test. - Read source date, exact execution identity, denominator, memory scope and retries per observation. - No unmatched quality ranking or hardware speed ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite original factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer A bounded gather lowers the isolated transient and the 500K single-request device peak, but the paired two-250K host-KV cell reports the same +1248 MiB non-pool peak on both trees. Production failure, development recovery and the smaller ship-tree rerun have different conditions. By Run AI Here editorial · 2026-10-18 Local Spark Qwen draft; Codex source audit and rewrite. No site model benchmark. ## Which memory observations are directly paired? The original report compares cold 2x250K prefill on the same Server Edition 600 W card. Minimum free device memory is 3414 MiB on the on-GPU control versus 6962 MiB with host KV; non-pool peaks over idle are 1090 versus 1248 MiB. Pools differ, so this is an operational configuration comparison. Another same-host-KV old/new gather comparison reports +1248 MiB in both arms, with no measurable peak benefit in that concurrent cell. Sources: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md, https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/README.md ## Where does bounded gather reduce the peak? The isolated 524288-token FP8 kernel reports 1024 to 32 MiB transient allocation, excluding output, with bit-exact checks in two runs. The in-engine single-request 500000-token sweep reports 97237 to 96093 MiB device used peak at the same 689728-token pool and 94661 MiB idle. These are different measurement scopes and cannot replace the concurrent cell. Host-UVA timing fails to reproduce consistently. Sources: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md ## What failed in production? The September 30 Max-Q Workstation 300 W lane fails an eager-prefill 560 MiB allocation with 280.94 MiB free and 93.92 GiB process device usage. At failure one request runs, fourteen queue, token pool usage is 0.73 and Mamba usage 0.06. The outage is 12m54s. Monitoring existed but was not consumed by admission. No total host peak or exact failing request input is published. Sources: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md ## How much recovery was actually retested? The host-KV-off development branch has 45 and 42 recoveries across two injection/real-pressure batteries. The ship tree first has an invalid harness run with zero injections; its corrected rerun has two injections and two recoveries. Decode OOM and driver errors remain fatal. The composite gate lacks NIAH cells in its own bundle even though a separate invocation passes nine needles. Sources: https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "on-GPU control minimum free VRAM", "metric": "available_GPU_memory", "value": 3414, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 870, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870" }, { "label": "on-GPU control non-pool peak over idle", "metric": "non_pool_memory_over_idle", "value": 1090, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 871, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871" }, { "label": "host KV minimum free VRAM", "metric": "available_GPU_memory", "value": 6962, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 870, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870" }, { "label": "host KV non-pool peak over idle", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 871, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871" }, { "label": "524288 FP8 old gather isolated transient", "metric": "kernel_transient_memory", "value": 1024.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "old gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 954, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954" }, { "label": "524288 FP8 bounded gather isolated transient", "metric": "kernel_transient_memory", "value": 32.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "bounded gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 954, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954" }, { "label": "500K old gather device peak", "metric": "peak_device_used_memory", "value": 97237, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "old gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 972, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972" }, { "label": "500K bounded gather device peak", "metric": "peak_device_used_memory", "value": 96093, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "bounded gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 972, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972" }, { "label": "2x250K host-KV old gather non-pool peak", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "old gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 981, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981" }, { "label": "2x250K host-KV bounded gather non-pool peak", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "bounded gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 981, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981" }, { "label": "Production OOM failed_allocation", "metric": "failed_allocation", "value": 560.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 639, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L639" }, { "label": "Production OOM available_GPU_memory", "metric": "available_GPU_memory", "value": 280.94, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 640, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L640" }, { "label": "Production OOM process_device_memory", "metric": "process_device_memory", "value": 93.92, "unit": "GiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 642, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L642" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "qwen-nanogen-readme", "parent_record_id": "qwen-flash-next-nanogen-memory-recovery", "publisher": "nanogenomic", "repository": "nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000", "revision": "11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2", "document_path": "README.md", "pinned_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/README.md", "document_sha256": "bcfd94276d2bebef681e8a7737444ecec9814d6c73dce9ecf95a0b9aa437a3ab", "last_commit_at": "2026-10-01T04:27:43Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "qwen-nanogen-commits", "parent_record_id": "qwen-flash-next-nanogen-memory-recovery", "publisher": "nanogenomic", "repository": "nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000", "revision": "11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2", "document_path": "COMMIT-MAP.tsv", "pinned_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/COMMIT-MAP.tsv", "document_sha256": "7a6b188d895f6d99fe46ca650832e7d1d8f182aa0b2c59e1442bd7794042fa92", "last_commit_at": "2026-10-01T04:27:43Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" } ] } ``` ## memory observations ```json { "memory_observations": [ { "label": "on-GPU control minimum free VRAM", "value": 3414, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870" }, { "label": "on-GPU control non-pool peak over idle", "value": 1090, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871" }, { "label": "host KV minimum free VRAM", "value": 6962, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870" }, { "label": "host KV non-pool peak over idle", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871" }, { "label": "524288 FP8 old gather isolated transient", "value": 1024.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "old gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954" }, { "label": "524288 FP8 bounded gather isolated transient", "value": 32.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "bounded gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954" }, { "label": "500K old gather device peak", "value": 97237, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "old gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972" }, { "label": "500K bounded gather device peak", "value": 96093, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "bounded gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972" }, { "label": "2x250K host-KV old gather non-pool peak", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "old gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981" }, { "label": "2x250K host-KV bounded gather non-pool peak", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "bounded gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981" }, { "label": "Production OOM failed_allocation", "value": 560.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L639" }, { "label": "Production OOM available_GPU_memory", "value": 280.94, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L640" }, { "label": "Production OOM process_device_memory", "value": 93.92, "unit": "GiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L642" } ] } ``` ## long context cases ```json { "long_context_cases": [ { "id": "host-KV-two-mains-turn3", "input_length_source_labels": [ "250K", "200K" ], "additional_subagents": 2, "host_KV_pool_tokens": 1310720, "control_pool_tokens": 451840, "host_KV_TTFT_seconds": [ 1.2, 1.0 ], "control_TTFT_seconds": [ 22.9, 41.1 ], "host_KV_cache_hit_rate": 0.9949, "control_cache_hit_rate": 0, "control_requests_completed": true, "host_peak_RAM": null, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 833, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L833" } ] } ``` ## case reports ```json { "case_reports": [ { "id": "deep-queue-production-OOM", "classification": "source_reported_real_allocator_failure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 633, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L633", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "downtime_seconds": 774, "error": "torch.OutOfMemoryError in eager prefill; scheduler SIGQUIT and process-tree termination", "monitoring_alert_before_crash_seconds": 53, "admission_consumed_alert": false }, { "id": "recovery-branch-versus-ship", "classification": "source_reported_fault_injection_and_real_pressure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 999, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L999", "branch": "oom-retract, host KV off", "branch_pool_tokens": [ 451712, 451840 ], "runs": [ { "forced": 39, "real": 6, "recoveries": 45 }, { "forced": 39, "real": 3, "recoveries": 42 } ], "engine_alive": true, "scheduler_suite": { "initial_harness_errors": 9, "initial_error": "runner IndexError", "rerun_passed": 100 }, "ship_tree": { "initial_valid": false, "initial_error": "BatchEncoding not JSON serializable", "initial_injects": 0, "initial_recoveries": 0, "initial_verdict": false, "rerun_injections": 2, "rerun_recoveries": 2, "rerun_all_needles_correct": true, "engine_alive": true, "host_KV_pool_tokens": 1310720 }, "recovery_predicates": [ "overlap enabled", "EXTEND", "TP1", "PP1", "no MLP sync", "no disaggregation", "HiSparse off", "PDMux off", "no diffusion config", "device-wide drain completes" ], "max_retractions": 3, "exhaustion_status": 503, "next_request_status": 200, "decode_OOM_recovered": false, "driver_error_recovered": false, "peer_outputs_bit_identical": false }, { "id": "host-UVA-regression", "classification": "unresolved_nonreproducing_microbenchmark", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 992, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L992", "first_new_ms": [ 341, 376 ], "later_new_ms": [ 4.99, 5.68 ], "later_new_peak_MiB": [ 159, 189 ], "first_touch_explanation_proven": false, "kernel_time_improvement_claim": false }, { "id": "composite-gate-missing-same-bundle-NIAH", "classification": "harness_composition_failure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 873, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L873", "decode_path_gate": true, "standalone_needle_correct": 9, "standalone_needle_total": 9, "same_bundle_NIAH_cells": 0, "long_context_gate": false, "overall_gate": false, "model_quality_failure_established": false } ] } ``` --- # Full GLM-5.3 on Four Sparks: Cache Layout, Speed and Fidelity Audit the 753B non-Flash EXL3 TP4 recipe, replicated and shared Q5 cache, million-token retrieval, BF16-reference fidelity and later RDMA fixes. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: GLM - Model: bertholomus/GLM-5.3-EXL3-3.0bpw - Original author: bertholomus - Original project: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10 - Pinned document: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md - Source revision: 9a5c1100c68cd8ea832803c120ee63313049fc21 - Source commit time (not experiment date): 2026-10-09T05:31:45Z - Checked: 2026-10-18 - Experiment date: 2026-10-04 - Runtime topics: TensorFold - Hardware topics: spark, nvidia - Reported setup: {"system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "context_limit_tokens": 65536, "experiment_date": "2026-10-04"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "replicated Q5 four code streams", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L57", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 4, "context_limit_tokens": 65536, "cache_state": "replicated", "metric_denominator": "source reports HTTP prompt time included; release concurrent raw usage/harness not retained", "token_accounting_verified": false, "available_serial_stream_probe_counts_nonempty_SSE_chunks": true, "release_was_computed_by_that_probe": null, "value_is_approximate": false } }, { "label": "DCP4 shared Q5 four code streams", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L68", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 4, "context_limit_tokens": 1048576, "cache_state": "four streams share one DCP4 pool", "metric_denominator": "source reports HTTP prompt time included; release concurrent raw usage/harness not retained", "token_accounting_verified": false, "available_serial_stream_probe_counts_nonempty_SSE_chunks": true, "release_was_computed_by_that_probe": null, "value_is_approximate": true } }, { "label": "32K-labelled prefill", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L71", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 1, "input_length_source_label": "32K", "context_limit_tokens": 1048576, "cache_state": null, "metric_denominator": "source HTTP prefill measurement, exact prompt count unreported" } }, { "label": "forward KL", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L167", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "forward KL" } }, { "label": "reverse KL", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L168", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "reverse KL" } }, { "label": "top-1 agreement", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L169", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "top-1 agreement" } }, { "label": "quant perplexity", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L170", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "quant perplexity" } }, { "label": "BF16 perplexity", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L170", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "BF16 perplexity" } }, { "label": "served engine prompt-path KL", "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L173", "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "served path; panel-specific cache details not retained", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "served engine" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Selected values are source excerpts, not independent experiment counts. ## Limits - No independent Run AI Here model test. - Read source date, exact execution identity, denominator, memory scope and retries per observation. - No unmatched quality ranking or hardware speed ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite original factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer The source runs full GLM-5.3 on four GB10 nodes with EXL3 and native MTP-3. Its two cache layouts trade reported aggregate code throughput for a larger shared window. Peak RAM and a task-accuracy suite are absent; the published quality panel measures quantization fidelity. By Run AI Here editorial · 2026-10-18 Local Spark Qwen draft; Codex source audit and rewrite. No site model benchmark. ## What must the deployment provide? Four 128 GB GB10 nodes use TP4, direct dual-port 200G RoCE, TensorFold 0.6.0 plus the model branch, EXL3 average 3.04 bpw and native MTP-3. It is text-only and does not use DFlash. The approximately 69.86 GiB per-rank weight fit is shape-derived and excludes whole-run cache and working memory. Sources: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md ## Which release produced the speed table? October 4 release numbers are attributed to engine prefix 3eb35dd; the later head ddfad35 adds per-device RDMA completion matching and graph prewarm. The full executed commit and checkpoint revision are unreported. Replicated Q5 at 65536/stream reports 82 aggregate code tokens/s for four streams; shared DCP4 Q5 at a 1048576-token pool reports about 64. Sources: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md ## How are HTTP accounting and fidelity limited? The README calls the rates HTTP measurements with prompt time included. Its available serial streaming probe counts nonempty SSE chunks rather than verified tokenizer/usage tokens, while the exact concurrent release harness/raw accounting is not retained here. The fidelity panel uses BF16 activations and no cache quantization, with forward KL 0.109, reverse KL 0.123 and 90.2 percent top-1 agreement. It is not a task-accuracy test. Sources: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md, https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_bench.py, https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_greedy.py ## What does the long-window result establish? The source finds a needle at 1039064 actual input tokens, reports 402 tokens/s at nominal 32K and about 76s first-token latency near 31K. Those do not demonstrate four full-million-token requests running together. Repetition counts, long-input peak RAM and raw per-request release accounting remain missing. Sources: https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md, https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_needle.py ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "replicated Q5 four code streams", "metric": "aggregate_output_rate", "value": 82, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 4, "context_limit_tokens": 65536, "cache_state": "replicated", "metric_denominator": "source reports HTTP prompt time included; release concurrent raw usage/harness not retained", "token_accounting_verified": false, "available_serial_stream_probe_counts_nonempty_SSE_chunks": true, "release_was_computed_by_that_probe": null, "value_is_approximate": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 57, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L57" }, { "label": "DCP4 shared Q5 four code streams", "metric": "aggregate_output_rate", "value": 64, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 4, "context_limit_tokens": 1048576, "cache_state": "four streams share one DCP4 pool", "metric_denominator": "source reports HTTP prompt time included; release concurrent raw usage/harness not retained", "token_accounting_verified": false, "available_serial_stream_probe_counts_nonempty_SSE_chunks": true, "release_was_computed_by_that_probe": null, "value_is_approximate": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 68, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L68" }, { "label": "32K-labelled prefill", "metric": "prefill_rate", "value": 402, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "Q5 MLA latent cache; FP8 indexer", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": "2026-10-04", "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": 1, "input_length_source_label": "32K", "context_limit_tokens": 1048576, "cache_state": null, "metric_denominator": "source HTTP prefill measurement, exact prompt count unreported" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 71, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L71" }, { "label": "forward KL", "metric": "KL_reference_to_quant", "value": 0.109, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "forward KL" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 167, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L167" }, { "label": "reverse KL", "metric": "KL_quant_to_reference", "value": 0.123, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "reverse KL" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 168, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L168" }, { "label": "top-1 agreement", "metric": "top1_agreement", "value": 90.2, "unit": "percent", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "top-1 agreement" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 169, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L169" }, { "label": "quant perplexity", "metric": "perplexity", "value": 2.911, "unit": "perplexity", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "quant perplexity" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 170, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L170" }, { "label": "BF16 perplexity", "metric": "perplexity", "value": 2.75, "unit": "perplexity", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "no cache quantization", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "BF16 perplexity" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 170, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L170" }, { "label": "served engine prompt-path KL", "metric": "KL_reference_to_engine", "value": 0.111, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3 (full 753B, not Flash)", "model_artifact": "bertholomus/GLM-5.3-EXL3-3.0bpw", "model_revision": null, "engine": "TensorFold 0.6.0 + glm-dsa-tp4 branch", "runtime_execution_revision": null, "measured_runtime_short": "3eb35dd", "later_branch_head_short": "ddfad35", "executed_image_digest": null, "system": "four DGX Spark GB10, 128 GB unified memory each, direct dual-port 200G RoCE", "node_count": 4, "parallelism": "TP4", "weight_precision": "EXL3 3.04 bpw whole-model average; exllamav3 1.5.3", "kv_dtype": "served path; panel-specific cache details not retained", "draft": "native MTP-3", "vision": false, "DFlash": false, "experiment_date": null, "temperature": 0, "thinking": null, "input_tokens": null, "output_tokens": null, "host_peak_RAM": null, "repetitions": null, "concurrency": null, "quality_panel": "wikitext-2 test, 32 x 2048 = 65536 positions", "activation_dtype": "BF16", "reference": "BF16 full model", "cache_state": null, "quality_is_task_accuracy": false, "quality_arm": "served engine" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 173, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L173" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "glmfull-speed-method", "parent_record_id": "glm53-full-tp4-exl3", "publisher": "bertholomus", "repository": "bertholomus/glm-5.3-tensorfold-tp4-4xgb10", "revision": "9a5c1100c68cd8ea832803c120ee63313049fc21", "document_path": "bench/tf_bench.py", "pinned_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_bench.py", "document_sha256": "9652bf8a7359414ad451dc05ceee4f4bef6cfcf489e65334e9bec1e7895192b2", "last_commit_at": "2026-10-09T05:31:45Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glmfull-needle-method", "parent_record_id": "glm53-full-tp4-exl3", "publisher": "bertholomus", "repository": "bertholomus/glm-5.3-tensorfold-tp4-4xgb10", "revision": "9a5c1100c68cd8ea832803c120ee63313049fc21", "document_path": "bench/tf_needle.py", "pinned_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_needle.py", "document_sha256": "cdf8ab15fc7ec88073a714696b561f2e7a292ac9a0490571935bbe2cc933a4ab", "last_commit_at": "2026-10-09T05:31:45Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glmfull-greedy-method", "parent_record_id": "glm53-full-tp4-exl3", "publisher": "bertholomus", "repository": "bertholomus/glm-5.3-tensorfold-tp4-4xgb10", "revision": "9a5c1100c68cd8ea832803c120ee63313049fc21", "document_path": "bench/tf_greedy.py", "pinned_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/bench/tf_greedy.py", "document_sha256": "834c9f1e73af9431b585adf00aa3f14254d8713b638a4fa95c36b6a2074dbf37", "last_commit_at": "2026-10-09T05:31:45Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" } ] } ``` ## Paired source quality comparisons ```json { "quality_comparisons": [ { "panel": "32x2048 wikitext-2", "BF16_activations": true, "cache_quantization": false, "forward_KL": 0.109, "reverse_KL": 0.123, "top1_percent": 90.2, "quant_PPL": 2.911, "BF16_PPL": 2.75, "task_accuracy": null } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "shape-derived weights per rank, not peak", "value": 69.86, "unit": "GiB", "is_measured_peak": false, "whole_system_peak": null, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 121, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L121" } ] } ``` ## long context cases ```json { "long_context_cases": [ { "actual_input_tokens": 1039064, "needle_found": true, "independent_repeats": null, "concurrent_full_window_requests_proven": false, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 71, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L71" } ] } ``` ## case reports ```json { "case_reports": [ { "classification": "source_reported_RDMA_completion_collision", "old_error": "ibv_post_recv failed", "fix_short": "cad0d2a", "measured_release_short": "3eb35dd", "later_head_short": "ddfad35", "full_execution_revision": null, "source_document_id": "glmfull-tp4-readme", "source_document_sha256": "088db2d55fbb6aecc2cc6576b3db9a26e70a13a2b5d89b83ad1cb0b54b03a6b4", "source_line": 35, "evidence_url": "https://github.com/bertholomus/glm-5.3-tensorfold-tp4-4xgb10/blob/9a5c1100c68cd8ea832803c120ee63313049fc21/README.md#L35" } ] } ``` ## quality comparisons ```json { "quality_comparisons": [ { "panel": "32x2048 wikitext-2", "BF16_activations": true, "cache_quantization": false, "forward_KL": 0.109, "reverse_KL": 0.123, "top1_percent": 90.2, "quant_PPL": 2.911, "BF16_PPL": 2.75, "task_accuracy": null } ] } ``` --- # GLM Flash Scores: Historical NVFP4 and the EXL3 Retraction — historical NVFP4 Separate a full historical MMLU run and small task subsets from the corrected EXL3 router stack, whose old task scores were withdrawn and await reruns. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: GLM - Model: local-inference-lab/GLM-5.3-Flash-NVFP4 - Original author: NNNtrance - Original project: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark - Pinned document: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md - Source revision: eb11cef295c107c5bd21402106eacea5ebd1fbf3 - Source commit time (not experiment date): 2026-09-09T21:23:55Z - Checked: 2026-10-18 - Experiment date: 2026-09-03 09:08-18:08 UTC - Runtime topics: vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "kv_dtype": "FP8", "context_limit_tokens": 1000000, "experiment_date": "2026-09-03 09:08-18:08 UTC"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Tool-eval eight-trial mean", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L47", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "tool-eval-bench 2.6.1.dev39+gd3352edf5", "scenarios": 88, "trials": 8, "seed": 42, "max_turns": 8, "concurrency": 1, "source_trial_SD": 0.9, "source_CI95": [ 87.1, 88.2 ], "CI_method_independently_verified": false } }, { "label": "Tool-eval Pass^8", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L56", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "tool-eval-bench", "scenarios": 88, "trials": 8, "concurrency": 1 } }, { "label": "GSM8K 200-question subset", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L197", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "GSM8K", "subset_size": 200, "full_test_size": 1319, "correct": 188, "shots": 8, "CoT": true, "concurrency": null } }, { "label": "IFEval prompt strict, custom evaluator", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L155", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "IFEval", "prompts": 541, "correct": 427, "concurrency": null, "evaluator": "custom; not independently cross-checked", "proven_lower_bound": false } }, { "label": "IFEval instruction level, custom evaluator", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L156", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "IFEval", "instructions": 834, "correct": 710, "concurrency": null, "evaluator": "custom; not independently cross-checked", "proven_lower_bound": false } }, { "label": "Needle four sizes x five depths", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L221", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "single-fact NIAH", "correct": 20, "total": 20, "max_actual_input_tokens": 997952, "concurrency": null, "initial_timeout_seconds": 120, "rerun_timeout_seconds": 3600, "identical_content_seed_on_rerun": true, "multihop_validated": false } }, { "label": "Full MMLU exact accuracy", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json#L4", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": null, "thinking": null, "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "MMLU", "subset_size": 14042, "subjects": 57, "shots": 0, "scoring": "loglikelihood, no generation or sampling", "harness": "lm-eval 0.4.9", "concurrency": 8, "timeout_seconds": 36000, "max_retries": 6, "accuracy_stderr": 0.0028276391244761038, "stderr_is_SD": false, "initial_harness_timeout_at_fraction": 0.81, "engine_errors_reported": 0, "rerun_same_content": true } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Selected values are source excerpts, not independent experiment counts. ## Limits - No independent Run AI Here model test. - Read source date, exact execution identity, denominator, memory scope and retries per observation. - No unmatched quality ranking or hardware speed ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite original factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer The September NVFP4 report and October EXL3 correction are different deployment lines. The EXL3 author withdrew its old task scores after finding an unloaded router bias. Current evidence is a same-quant probability check; it does not revalidate old task scores or import the NVFP4 task panel. By Run AI Here editorial · 2026-10-18 Local Spark Qwen draft; Codex source audit and rewrite. No site model benchmark. ## Which scores remain historical NVFP4 observations? The September 3 TP3+EP Marlin NVFP4 t10 campaign uses GMU 0.85, FP8 KV, DFlash2 k7 and a decimal-million configured window. Full MMLU is 14042 zero-shot loglikelihood questions with exact accuracy 0.8594217347956131 and harness SE 0.0028276391244761038. Tool-eval has eight trials over 88 hard scenarios, mean 87.8 and source SD 0.9. GSM8K is only 200 questions, 8-shot,188 correct. IFEval uses a custom unverified evaluator. Sources: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md, https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json, https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json ## Why did the EXL3 author withdraw its old scores? The October 8 notice identifies missing e_score_correction_bias in the old cuda-exl3 plugin. Wrong expert routing invalidates that repository's old task panels. The new TP3 main stack loads the bias and fails boot if missing. TP2 remains unported and untested. This notice is scoped to the EXL3 plugin line, not proof of the same bug in the separate historical NVFP4 line. Sources: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md, https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/tracks/tp3-main/README.md, https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md ## What was rerun on corrected EXL3? The detailed KL table compares the same EXL3 checkpoint to exllamav3 1.5.4 on six author-written texts. Assistant KL is 0.1033 pre-fix and 0.00828 on production a4i. Perplexity 2.222 belongs to bias-fix a4g; production perplexity is not reported in that table. The reference is quantized, not BF16. Task benchmarks have not been rerun, and the prepared text set and per-position logits are not public. Sources: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md, https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md ## How are failed attempts kept? Historical NVFP4 MMLU initially hits a harness queue timeout, and needle tests initially exceed a 120s request cap; same-content reruns use longer timeouts with zero engine errors reported. These are harness retries, not model-accuracy zeros or engine recoveries. A separate model line's completed task scores cannot fill the corrected EXL3 missing scores. Sources: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md, https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md, https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Tool-eval eight-trial mean", "metric": "tool_eval_score", "value": 87.8, "unit": "points", "statistic": "mean across eight trials; source SD 0.9, source CI [87.1,88.2]", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "tool-eval-bench 2.6.1.dev39+gd3352edf5", "scenarios": 88, "trials": 8, "seed": 42, "max_turns": 8, "concurrency": 1, "source_trial_SD": 0.9, "source_CI95": [ 87.1, 88.2 ], "CI_method_independently_verified": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 47, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L47" }, { "label": "Tool-eval Pass^8", "metric": "all_trials_pass_rate", "value": 76.1, "unit": "percent", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "tool-eval-bench", "scenarios": 88, "trials": 8, "concurrency": 1 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 56, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L56" }, { "label": "GSM8K 200-question subset", "metric": "accuracy", "value": 94.0, "unit": "percent", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "GSM8K", "subset_size": 200, "full_test_size": 1319, "correct": 188, "shots": 8, "CoT": true, "concurrency": null }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 197, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L197" }, { "label": "IFEval prompt strict, custom evaluator", "metric": "accuracy", "value": 78.9, "unit": "percent", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "IFEval", "prompts": 541, "correct": 427, "concurrency": null, "evaluator": "custom; not independently cross-checked", "proven_lower_bound": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 155, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L155" }, { "label": "IFEval instruction level, custom evaluator", "metric": "accuracy", "value": 85.1, "unit": "percent", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "IFEval", "instructions": 834, "correct": 710, "concurrency": null, "evaluator": "custom; not independently cross-checked", "proven_lower_bound": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 156, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L156" }, { "label": "Needle four sizes x five depths", "metric": "retrieval_correct_count", "value": 20, "unit": "correct probes", "statistic": "single finite benchmark; no dispersion reported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": 0, "thinking": "on, effort low except MMLU", "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "single-fact NIAH", "correct": 20, "total": 20, "max_actual_input_tokens": 997952, "concurrency": null, "initial_timeout_seconds": 120, "rerun_timeout_seconds": 3600, "identical_content_seed_on_rerun": true, "multihop_validated": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 221, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L221" }, { "label": "Full MMLU exact accuracy", "metric": "accuracy", "value": 0.8594217347956131, "unit": "fraction", "statistic": "single full MMLU, harness standard error 0.0028276391244761038", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4", "model_revision": null, "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10", "runtime_execution_revision": null, "executed_image_digest": null, "system": "three DGX Spark GB10, TP3 + EP", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "NVFP4 storage / Marlin MoE", "kv_dtype": "FP8", "draft": "DFlash2 k7", "context_limit_tokens": 1000000, "memory_utilization": 0.85, "configured_max_requests": 8, "batch_tokens": 2048, "experiment_date": "2026-09-03 09:08-18:08 UTC", "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores", "host_peak_RAM": null, "temperature": null, "thinking": null, "input_tokens": null, "output_tokens": null, "cache_state": null, "benchmark": "MMLU", "subset_size": 14042, "subjects": 57, "shots": 0, "scoring": "loglikelihood, no generation or sampling", "harness": "lm-eval 0.4.9", "concurrency": 8, "timeout_seconds": 36000, "max_retries": 6, "accuracy_stderr": 0.0028276391244761038, "stderr_is_SD": false, "initial_harness_timeout_at_fraction": 0.81, "engine_errors_reported": 0, "rerun_same_content": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-tp3-mmlu", "source_document_sha256": "86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8", "source_line": 4, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json#L4" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "glm-tp3-readme", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "README.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/README.md", "document_sha256": "5912e935333340dbd77ecd01cec8c7e207990bc6f446341f741bb3bd66d6cf4b", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-memory", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "docs/05-memory-ladder.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/05-memory-ladder.md", "document_sha256": "29874450acd65f1e6614bd3fc04fe0815ef8b9036007867b8446e63ebb4092de", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-mmlu", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "results/mmlu/mmlu-full-lm-eval-results.json", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json", "document_sha256": "86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-tool", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "results/tool-eval-bench/tool-eval-bench-8-trials.json", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json", "document_sha256": "7a46e4d7b52956a0fb0662838081b526e7ceeb30e14e553648227bade4dc6df5", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-needle", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "results/needle/needle-in-a-haystack-report.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md", "document_sha256": "cceef7123bbb9e9d34c2c6a44bc00055b81f896cbf51977053766e770c30ae3d", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-results", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "results/README.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/README.md", "document_sha256": "15388126cd782399e618a52b5cbcd2c5af46f7de347f6903e4705d719f84b037", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-exl3-readme", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark", "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f", "document_path": "README.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md", "document_sha256": "a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0", "last_commit_at": "2026-10-16T01:23:58Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-exl3-kl", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark", "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f", "document_path": "results/main-stack/kl-health.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md", "document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1", "last_commit_at": "2026-10-16T01:23:58Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-exl3-gates", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark", "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f", "document_path": "results/main-stack/quality-gates.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md", "document_sha256": "9d0387be075bd6c214ab8e9b2bb70a46577b5052884c06c0db02d1869b295f28", "last_commit_at": "2026-10-16T01:23:58Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-exl3-stack", "parent_record_id": "glm53-flash-nvfp4-tp3-scores", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark", "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f", "document_path": "docs/20-main-stack.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/docs/20-main-stack.md", "document_sha256": "b7ef6f4c82dd1bfa693452cb1fcfb57c8f3270455ca894b667e24048cc20786d", "last_commit_at": "2026-10-16T01:23:58Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" } ] } ``` ## Paired source quality comparisons ```json { "quality_comparisons": [ { "id": "NVFP4-historical-task-panel", "record_id": "glm53-flash-nvfp4-tp3-scores", "task_scores_current_EXL3": false, "task_scores_withdrawn_by_EXL3_notice": false, "source_scope": "separate NVFP4 repository, September 3 measurements", "tool_trial_scores": [ 88, 88, 86, 87, 88, 88, 88, 89 ], "tool_mean": 87.8, "source_SD": 0.9, "source_CI95": [ 87.1, 88.2 ], "MMLU_accuracy": 0.8594217347956131, "MMLU_standard_error": 0.0028276391244761038, "IFEval_evaluator_crosschecked": false, "proven_IFEval_lower_bound": false }, { "id": "EXL3-retraction-and-rerun-gap", "record_id": "glm53-flash-exl3-router-correction", "withdrawn_on": "2026-10-08", "old_task_scores_withdrawn": true, "current_task_accuracy": null, "current_task_benchmarks_rerun": false, "router_bias_loaded_after_fix": true, "production_KL": 0.00828, "production_PPL": null, "bias_fix_PPL": 2.222, "reference_PPL": 2.224, "BF16_reference": false, "public_text_set_available": false, "full_per_position_logits_public": false, "TP2_fixed_or_validated": false } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "historical operational free host memory, not peak", "value_range": [ 6.4, 10 ], "unit": "GB, source approximate", "historical_NVFP4_only": true, "peak_RAM": null, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 28, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L28" } ] } ``` ## case reports ```json { "case_reports": [ { "id": "NVFP4-harness-only-reruns", "classification": "source_reported_harness_timeouts", "MMLU_initial_progress_fraction": 0.81, "MMLU_initial_error": "aiohttp total timeout includes queue", "MMLU_rerun_timeout_seconds": 36000, "needle_initial_timeout_seconds": 120, "needle_rerun_timeout_seconds": 3600, "same_content_and_seed": true, "reported_engine_errors": 0, "initial_partial_not_full_score": true, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 35, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L35" }, { "id": "EXL3-router-bias-missing", "classification": "source_reported_correctness_bug_and_retraction", "plugin_pre_fix_short": "754421f or earlier", "missing_parameter": "e_score_correction_bias", "source_scope": "EXL3 plugin line only; no evidence that separate NVFP4 line shares defect", "old_task_scores_withdrawn": true, "current_task_score": null, "source_document_id": "glm-exl3-readme", "source_document_sha256": "a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0", "source_line": 18, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md#L18" } ] } ``` ## quality comparisons ```json { "quality_comparisons": [ { "id": "NVFP4-historical-task-panel", "record_id": "glm53-flash-nvfp4-tp3-scores", "task_scores_current_EXL3": false, "task_scores_withdrawn_by_EXL3_notice": false, "source_scope": "separate NVFP4 repository, September 3 measurements", "tool_trial_scores": [ 88, 88, 86, 87, 88, 88, 88, 89 ], "tool_mean": 87.8, "source_SD": 0.9, "source_CI95": [ 87.1, 88.2 ], "MMLU_accuracy": 0.8594217347956131, "MMLU_standard_error": 0.0028276391244761038, "IFEval_evaluator_crosschecked": false, "proven_IFEval_lower_bound": false }, { "id": "EXL3-retraction-and-rerun-gap", "record_id": "glm53-flash-exl3-router-correction", "withdrawn_on": "2026-10-08", "old_task_scores_withdrawn": true, "current_task_accuracy": null, "current_task_benchmarks_rerun": false, "router_bias_loaded_after_fix": true, "production_KL": 0.00828, "production_PPL": null, "bias_fix_PPL": 2.222, "reference_PPL": 2.224, "BF16_reference": false, "public_text_set_available": false, "full_per_position_logits_public": false, "TP2_fixed_or_validated": false } ] } ``` --- # GLM Flash Scores: Historical NVFP4 and the EXL3 Retraction — corrected EXL3 Separate a full historical MMLU run and small task subsets from the corrected EXL3 router stack, whose old task scores were withdrawn and await reruns. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: GLM - Model: turboderp/GLM-5.3-Flash-exl3 - Original author: NNNtrance - Original project: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark - Pinned document: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/tracks/tp3-main/README.md - Source revision: 83ed1eccd93a15ae9efe495dbee016cd57823c5f - Source commit time (not experiment date): 2026-10-16T01:23:58Z - Checked: 2026-10-18 - Experiment date: 2026-10-07 to 2026-10-08 - Runtime topics: vLLM - Hardware topics: spark, nvidia - Reported setup: {"system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "kv_dtype": "FP8", "context_limit_tokens": null, "experiment_date": "2026-10-07 to 2026-10-08"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "Pre-fix assistant KL", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L40", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4f", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false } }, { "label": "Production assistant KL", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L42", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4i", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false } }, { "label": "Bias-fix assistant perplexity", "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L41", "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4g", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Selected values are source excerpts, not independent experiment counts. ## Limits - No independent Run AI Here model test. - Read source date, exact execution identity, denominator, memory scope and retries per observation. - No unmatched quality ranking or hardware speed ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite original factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer The September NVFP4 report and October EXL3 correction are different deployment lines. The EXL3 author withdrew its old task scores after finding an unloaded router bias. Current evidence is a same-quant probability check; it does not revalidate old task scores or import the NVFP4 task panel. By Run AI Here editorial · 2026-10-18 Local Spark Qwen draft; Codex source audit and rewrite. No site model benchmark. ## Which scores remain historical NVFP4 observations? The September 3 TP3+EP Marlin NVFP4 t10 campaign uses GMU 0.85, FP8 KV, DFlash2 k7 and a decimal-million configured window. Full MMLU is 14042 zero-shot loglikelihood questions with exact accuracy 0.8594217347956131 and harness SE 0.0028276391244761038. Tool-eval has eight trials over 88 hard scenarios, mean 87.8 and source SD 0.9. GSM8K is only 200 questions, 8-shot,188 correct. IFEval uses a custom unverified evaluator. Sources: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md, https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json, https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json ## Why did the EXL3 author withdraw its old scores? The October 8 notice identifies missing e_score_correction_bias in the old cuda-exl3 plugin. Wrong expert routing invalidates that repository's old task panels. The new TP3 main stack loads the bias and fails boot if missing. TP2 remains unported and untested. This notice is scoped to the EXL3 plugin line, not proof of the same bug in the separate historical NVFP4 line. Sources: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md, https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/tracks/tp3-main/README.md, https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md ## What was rerun on corrected EXL3? The detailed KL table compares the same EXL3 checkpoint to exllamav3 1.5.4 on six author-written texts. Assistant KL is 0.1033 pre-fix and 0.00828 on production a4i. Perplexity 2.222 belongs to bias-fix a4g; production perplexity is not reported in that table. The reference is quantized, not BF16. Task benchmarks have not been rerun, and the prepared text set and per-position logits are not public. Sources: https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md, https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md ## How are failed attempts kept? Historical NVFP4 MMLU initially hits a harness queue timeout, and needle tests initially exceed a 120s request cap; same-content reruns use longer timeouts with zero engine errors reported. These are harness retries, not model-accuracy zeros or engine recoveries. A separate model line's completed task scores cannot fill the corrected EXL3 missing scores. Sources: https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md, https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md, https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "Pre-fix assistant KL", "metric": "KL_reference_to_engine", "value": 0.1033, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4f", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-exl3-kl", "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1", "source_line": 40, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L40" }, { "label": "Production assistant KL", "metric": "KL_reference_to_engine", "value": 0.00828, "unit": "nats", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4i", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-exl3-kl", "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1", "source_line": 42, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L42" }, { "label": "Bias-fix assistant perplexity", "metric": "perplexity", "value": 2.222, "unit": "perplexity", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "GLM-5.3-Flash", "model_artifact": "turboderp/GLM-5.3-Flash-exl3", "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5", "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches", "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7", "runtime_execution_revision": null, "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef", "executed_image_digest": null, "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261", "system": "three DGX Spark GB10", "node_count": 3, "parallelism": "TP3+EP", "weight_precision": "EXL3 4.05 bpw full scope", "kv_dtype": "FP8", "experiment_date": "2026-10-07 to 2026-10-08", "concurrency": null, "input_tokens": 21072, "scored_positions": 21066, "assistant_positions": 6601, "quality_panel": "six author-written texts; text set and per-position logits not published", "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16", "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate", "temperature": null, "thinking": "reference panel has reasoning effort low / clear_thinking", "cache_state": "prefill target log probabilities, before DFlash proposals", "memory_utilization": 0.75, "host_peak_RAM": null, "current_task_benchmark_score": null, "old_EXL3_task_scores_withdrawn": true, "TP2_corrected_or_tested": false, "measured_build": "a4g", "production_perplexity": null, "pre_fix_comparison_is_retracted_task_score": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "glm-exl3-kl", "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1", "source_line": 41, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L41" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "glm-tp3-readme", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "README.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/README.md", "document_sha256": "5912e935333340dbd77ecd01cec8c7e207990bc6f446341f741bb3bd66d6cf4b", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-memory", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "docs/05-memory-ladder.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/05-memory-ladder.md", "document_sha256": "29874450acd65f1e6614bd3fc04fe0815ef8b9036007867b8446e63ebb4092de", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-mmlu", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "results/mmlu/mmlu-full-lm-eval-results.json", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json", "document_sha256": "86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-tool", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "results/tool-eval-bench/tool-eval-bench-8-trials.json", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json", "document_sha256": "7a46e4d7b52956a0fb0662838081b526e7ceeb30e14e553648227bade4dc6df5", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-needle", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "results/needle/needle-in-a-haystack-report.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md", "document_sha256": "cceef7123bbb9e9d34c2c6a44bc00055b81f896cbf51977053766e770c30ae3d", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-tp3-results", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark", "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3", "document_path": "results/README.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/README.md", "document_sha256": "15388126cd782399e618a52b5cbcd2c5af46f7de347f6903e4705d719f84b037", "last_commit_at": "2026-09-09T21:23:55Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-exl3-readme", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark", "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f", "document_path": "README.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md", "document_sha256": "a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0", "last_commit_at": "2026-10-16T01:23:58Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-exl3-kl", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark", "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f", "document_path": "results/main-stack/kl-health.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md", "document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1", "last_commit_at": "2026-10-16T01:23:58Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-exl3-gates", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark", "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f", "document_path": "results/main-stack/quality-gates.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md", "document_sha256": "9d0387be075bd6c214ab8e9b2bb70a46577b5052884c06c0db02d1869b295f28", "last_commit_at": "2026-10-16T01:23:58Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "glm-exl3-stack", "parent_record_id": "glm53-flash-exl3-router-correction", "publisher": "NNNtrance", "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark", "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f", "document_path": "docs/20-main-stack.md", "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/docs/20-main-stack.md", "document_sha256": "b7ef6f4c82dd1bfa693452cb1fcfb57c8f3270455ca894b667e24048cc20786d", "last_commit_at": "2026-10-16T01:23:58Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" } ] } ``` ## Paired source quality comparisons ```json { "quality_comparisons": [ { "id": "NVFP4-historical-task-panel", "record_id": "glm53-flash-nvfp4-tp3-scores", "task_scores_current_EXL3": false, "task_scores_withdrawn_by_EXL3_notice": false, "source_scope": "separate NVFP4 repository, September 3 measurements", "tool_trial_scores": [ 88, 88, 86, 87, 88, 88, 88, 89 ], "tool_mean": 87.8, "source_SD": 0.9, "source_CI95": [ 87.1, 88.2 ], "MMLU_accuracy": 0.8594217347956131, "MMLU_standard_error": 0.0028276391244761038, "IFEval_evaluator_crosschecked": false, "proven_IFEval_lower_bound": false }, { "id": "EXL3-retraction-and-rerun-gap", "record_id": "glm53-flash-exl3-router-correction", "withdrawn_on": "2026-10-08", "old_task_scores_withdrawn": true, "current_task_accuracy": null, "current_task_benchmarks_rerun": false, "router_bias_loaded_after_fix": true, "production_KL": 0.00828, "production_PPL": null, "bias_fix_PPL": 2.222, "reference_PPL": 2.224, "BF16_reference": false, "public_text_set_available": false, "full_per_position_logits_public": false, "TP2_fixed_or_validated": false } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "historical operational free host memory, not peak", "value_range": [ 6.4, 10 ], "unit": "GB, source approximate", "historical_NVFP4_only": true, "peak_RAM": null, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 28, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L28" } ] } ``` ## case reports ```json { "case_reports": [ { "id": "NVFP4-harness-only-reruns", "classification": "source_reported_harness_timeouts", "MMLU_initial_progress_fraction": 0.81, "MMLU_initial_error": "aiohttp total timeout includes queue", "MMLU_rerun_timeout_seconds": 36000, "needle_initial_timeout_seconds": 120, "needle_rerun_timeout_seconds": 3600, "same_content_and_seed": true, "reported_engine_errors": 0, "initial_partial_not_full_score": true, "source_document_id": "glm-tp3-scores", "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495", "source_line": 35, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L35" }, { "id": "EXL3-router-bias-missing", "classification": "source_reported_correctness_bug_and_retraction", "plugin_pre_fix_short": "754421f or earlier", "missing_parameter": "e_score_correction_bias", "source_scope": "EXL3 plugin line only; no evidence that separate NVFP4 line shares defect", "old_task_scores_withdrawn": true, "current_task_score": null, "source_document_id": "glm-exl3-readme", "source_document_sha256": "a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0", "source_line": 18, "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md#L18" } ] } ``` ## quality comparisons ```json { "quality_comparisons": [ { "id": "NVFP4-historical-task-panel", "record_id": "glm53-flash-nvfp4-tp3-scores", "task_scores_current_EXL3": false, "task_scores_withdrawn_by_EXL3_notice": false, "source_scope": "separate NVFP4 repository, September 3 measurements", "tool_trial_scores": [ 88, 88, 86, 87, 88, 88, 88, 89 ], "tool_mean": 87.8, "source_SD": 0.9, "source_CI95": [ 87.1, 88.2 ], "MMLU_accuracy": 0.8594217347956131, "MMLU_standard_error": 0.0028276391244761038, "IFEval_evaluator_crosschecked": false, "proven_IFEval_lower_bound": false }, { "id": "EXL3-retraction-and-rerun-gap", "record_id": "glm53-flash-exl3-router-correction", "withdrawn_on": "2026-10-08", "old_task_scores_withdrawn": true, "current_task_accuracy": null, "current_task_benchmarks_rerun": false, "router_bias_loaded_after_fix": true, "production_KL": 0.00828, "production_PPL": null, "bias_fix_PPL": 2.222, "reference_PPL": 2.224, "BF16_reference": false, "public_text_set_available": false, "full_per_position_logits_public": false, "TP2_fixed_or_validated": false } ] } ``` --- # MiMo Pro on Eight Sparks: Scheduling, Rates and Regression Scores Read a13/a14 cold extraction logs, the queued eighth-request fix, full model/base-engine pins, a GSM8K subset and older-build needle checks. - Type: Performance report - Evidence: source_reported; site reproduced: false - Model family: MiMo - Model: XiaomiMiMo/MiMo-V2.6-Pro-RL - Original author: rhys101 - Original project: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8 - Pinned document: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md - Source revision: 2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10 - Source commit time (not experiment date): 2026-09-24T18:54:01Z - Checked: 2026-10-18 - Experiment date: 2026-09-24 - Runtime topics: SGLang - Hardware topics: spark, nvidia - Reported setup: {"system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-09-24"} - Performance figures: attributed selections with per-point conditions; no cross-source ranking ## Conditions Unknown is null. Recipe options are not measured benchmark settings. ```json { "conditions": { "model_revision": null, "weight_hash": null, "runtime_execution_revision": null, "input_tokens": null, "output_tokens": null, "concurrency": null, "cache_state": null, "quantization": null, "speculative_decoding": null, "temperature": null, "top_p": null }, "test_profiles": [ { "label": "coding C1", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L28", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "metric_denominator": "source per-stream decode" } }, { "label": "coding C8 per stream", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L28", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "metric_denominator": "source per-stream decode" } }, { "label": "eight categories C8 aggregate", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L35", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "metric_denominator": "all eight category output / total wall; not mean per-stream multiplied by 8" } }, { "label": "a13 cold CSV extraction C8", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/long-extraction-c8.txt#L4", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": "three waves, 8 requests per wave", "cache_state": "unique cache salt; prefix cache flushed before each wave", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "input_length_source_label": "four CSV fixtures, duplicated per wave; C8 input counts not retained", "output_tokens_cap": 4096, "actual_output_tokens_group": 43908, "group_wall_seconds": 187.18, "correct": 24, "total": 24, "seed": 20260922, "metric_denominator": "all completion usage tokens / total synchronized wave wall including prefill" } }, { "label": "a14 cold CSV extraction C8", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction-c8.txt#L4", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": "sha256:db156b3c950a0ec9c4a7d629f7574f65c1023150bcf2d0755c45d4f2c3cab661", "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a14", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": "three waves, 8 requests per wave", "cache_state": "unique cache salt; prefix cache flushed before each wave", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "input_length_source_label": "four CSV fixtures, duplicated per wave; C8 input counts not retained", "output_tokens_cap": 4096, "actual_output_tokens_group": 43900, "group_wall_seconds": 143.16, "correct": 24, "total": 24, "seed": 20260922, "metric_denominator": "all completion usage tokens / total synchronized wave wall including prefill" } }, { "label": "a14 cold CSV extraction C1", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction.txt#L13", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a14", "temperature": 0, "thinking": false, "input_tokens": 1498, "output_tokens": null, "repetitions": "four tasks x three repeats", "cache_state": "unique salt and cache flush before every request", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "output_tokens_cap": 4096, "actual_output_tokens_group": 21952, "group_wall_seconds": 209.86, "correct": 12, "total": 12, "seed": 20260922, "metric_denominator": "completion usage / summed client wall including prefill" } }, { "label": "a13 GSM8K first 200", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L83", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": null, "harness_default_concurrency": 8, "actual_harness_concurrency_reported": false, "benchmark": "first 200 GSM8K test questions in fixed order", "correct": 196, "total": 200, "shots": 0, "output_tokens_cap": 1024, "answer_extraction": "Answer: numeric, else last numeric; abs tolerance <1e-6", "wrong_ids": [ 93, 119, 162, 187 ], "a14_GSM8K_rerun": false, "leaderboard_equivalent": false } }, { "label": "a13 5K cold prefill", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L57", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": 3, "cache_state": "unique cold prefix", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "input_length_source_label": "5K", "statistic": "median", "metric_denominator": "TTFT based" } }, { "label": "a13 163K cold prefill", "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L58", "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": 3, "cache_state": "unique cold prefix", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "input_length_source_label": "163K", "statistic": "median", "metric_denominator": "TTFT based" } } ], "recipe_parameters": null, "deployment_options": [] } ``` ## Reading notes - Selected values are source excerpts, not independent experiment counts. ## Limits - No independent Run AI Here model test. - Read source date, exact execution identity, denominator, memory scope and retries per observation. - No unmatched quality ranking or hardware speed ranking. ## License declaration {"declared_spdx": null, "scope": "Attribution and finite original factual interpretation only; models and dependencies retain their own terms."} ## Editorial answer The a14 scheduler change raises cold eight-request extraction aggregate rate from 234.57 to 306.66 tokens/s while all 24 answers pass in each arm. This is long output, not a 128K input test: the single-request fixture log has 1498 input tokens. a13 quality scores and older needle runs keep their original build labels. By Run AI Here editorial · 2026-10-18 Local Spark Qwen draft; Codex source audit and rewrite. No site model benchmark. ## What is the eight-node recipe? The checkpoint is XiaomiMiMo/MiMo-V2.6-Pro-RL at full revision 73875d00b30a89ef8cc353a0b60b0e9f9561952d. Eight GB10 nodes use TP8/EP1, two switched200G RoCEv2 subnets at MTU9000, packed MXFP4 experts on Marlin W4A16, DFlash8 adaptive verify and FA4. The base SGLang SHA and a14 local image ID are pinned separately; a complete patched execution-tree revision and registry digest remain unknown. Sources: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md, https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/versions.lock.json ## What changes in the paired extraction logs? The a13 scheduler counts a finishing prefill request against slots it already holds, leaving the eighth request queued. Across three cold eight-request waves, a13 has43908 completion tokens/187.18s and a14 has 43900/143.16s. Both pass24/24. Arithmetic mean per-stream rate falls 39.46 to 38.4 while complete-wave aggregate rises 234.57 to 306.66, showing why the denominators must stay separate. Sources: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/long-extraction-c8.txt, https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction-c8.txt, https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/long_extraction_c8.py ## How long is long extraction? The C1 source log records 1498 prompt tokens for every fixture, with roughly 1829 output tokens. C8 logs omit prompt counts. Unique salts and prefix-cache flushes make these cold extraction requests, but they are not128K-plus input workloads. Separate a13 cold prefill medians use nominal5K/20K/81K/163K labels. Sources: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction.txt, https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/long_extraction_c1.py, https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md ## Which quality and long-context checks apply to which build? GSM8K first200 fixed-order questions gives 196 correct on a13, greedy/thinking off with output cap1024 and a custom numeric extractor. No a14 GSM8K rerun is reported. Three-fact needle results at 127803 and 244168 tokens belong to a5/a6, not a13/a14. Finite smoke checks include vision; audio/video and peak runtime RAM are unmeasured. Sources: https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/gsm8k200.json, https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/eval/gsm8k_eval.py, https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md ## Selected source-reported figures Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking. ```json [ { "label": "coding C1", "metric": "decode_rate", "value": 91.1, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "metric_denominator": "source per-stream decode" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 28, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L28" }, { "label": "coding C8 per stream", "metric": "decode_rate", "value": 42.3, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "metric_denominator": "source per-stream decode" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 28, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L28" }, { "label": "eight categories C8 aggregate", "metric": "aggregate_output_rate", "value": 190.8, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "metric_denominator": "all eight category output / total wall; not mean per-stream multiplied by 8" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 35, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L35" }, { "label": "a13 cold CSV extraction C8", "metric": "aggregate_output_rate", "value": 234.57, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": "three waves, 8 requests per wave", "cache_state": "unique cache salt; prefix cache flushed before each wave", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "input_length_source_label": "four CSV fixtures, duplicated per wave; C8 input counts not retained", "output_tokens_cap": 4096, "actual_output_tokens_group": 43908, "group_wall_seconds": 187.18, "correct": 24, "total": 24, "seed": 20260922, "metric_denominator": "all completion usage tokens / total synchronized wave wall including prefill" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-a13-long8", "source_document_sha256": "443ce4520c8472347ac496024483ff46ab1b9a6a3851803d64bf382b66a26f37", "source_line": 4, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/long-extraction-c8.txt#L4" }, { "label": "a14 cold CSV extraction C8", "metric": "aggregate_output_rate", "value": 306.66, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": "sha256:db156b3c950a0ec9c4a7d629f7574f65c1023150bcf2d0755c45d4f2c3cab661", "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a14", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": "three waves, 8 requests per wave", "cache_state": "unique cache salt; prefix cache flushed before each wave", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 8, "input_length_source_label": "four CSV fixtures, duplicated per wave; C8 input counts not retained", "output_tokens_cap": 4096, "actual_output_tokens_group": 43900, "group_wall_seconds": 143.16, "correct": 24, "total": 24, "seed": 20260922, "metric_denominator": "all completion usage tokens / total synchronized wave wall including prefill" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-a14-long8", "source_document_sha256": "36b64993a9c32e4b08be0ee600aa743e3fe4c62c917a6ae3ca05f9560c42233a", "source_line": 4, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction-c8.txt#L4" }, { "label": "a14 cold CSV extraction C1", "metric": "aggregate_output_rate", "value": 104.6, "unit": "tokens/s", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a14", "temperature": 0, "thinking": false, "input_tokens": 1498, "output_tokens": null, "repetitions": "four tasks x three repeats", "cache_state": "unique salt and cache flush before every request", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "output_tokens_cap": 4096, "actual_output_tokens_group": 21952, "group_wall_seconds": 209.86, "correct": 12, "total": 12, "seed": 20260922, "metric_denominator": "completion usage / summed client wall including prefill" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-a14-long1", "source_document_sha256": "aa85e6233aaf60ab708d6580efada76287b2d560b4cc65f76f72f1cd3f5ab0d3", "source_line": 13, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction.txt#L13" }, { "label": "a13 GSM8K first 200", "metric": "accuracy", "value": 98.0, "unit": "percent", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": null, "cache_state": null, "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": null, "harness_default_concurrency": 8, "actual_harness_concurrency_reported": false, "benchmark": "first 200 GSM8K test questions in fixed order", "correct": 196, "total": 200, "shots": 0, "output_tokens_cap": 1024, "answer_extraction": "Answer: numeric, else last numeric; abs tolerance <1e-6", "wrong_ids": [ 93, 119, 162, 187 ], "a14_GSM8K_rerun": false, "leaderboard_equivalent": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 83, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L83" }, { "label": "a13 5K cold prefill", "metric": "prefill_rate", "value": 1880, "unit": "tokens/s", "statistic": "median of three source runs; no SD", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": 3, "cache_state": "unique cold prefix", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "input_length_source_label": "5K", "statistic": "median", "metric_denominator": "TTFT based" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 57, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L57" }, { "label": "a13 163K cold prefill", "metric": "prefill_rate", "value": 1499, "unit": "tokens/s", "statistic": "median of three source runs; no SD", "uncertainty": null, "conditions": { "model": "MiMo-V2.6-Pro-RL", "model_artifact": "XiaomiMiMo/MiMo-V2.6-Pro-RL", "model_revision": "73875d00b30a89ef8cc353a0b60b0e9f9561952d", "engine": "patched SGLang with Marlin W4A16 MXFP4 experts, DFlash8 adaptive verify and FA4", "base_engine_revision": "06008c170dae333e22caceb51f36224071f9b5b3", "runtime_execution_revision": null, "executed_image_digest": null, "executed_image_id": null, "system": "eight DGX Spark GB10, two switched 200G RoCEv2 subnets, MTU9000", "node_count": 8, "parallelism": "TP8/EP1", "weight_precision": "MXFP4 expert storage; Marlin W4A16; optional FP8 o_proj and draft", "kv_dtype": null, "experiment_date": "2026-09-24", "measured_build": "a13", "temperature": 0, "thinking": false, "input_tokens": null, "output_tokens": null, "repetitions": 3, "cache_state": "unique cold prefix", "peak_runtime_RAM": null, "context_limit_tokens": null, "drafter_revision": null, "concurrency": 1, "input_length_source_label": "163K", "statistic": "median", "metric_denominator": "TTFT based" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 58, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L58" } ] ``` ## Supplemental source identities ```json { "supplemental_sources": [ { "document_id": "mimopro-versions", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "versions.lock.json", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/versions.lock.json", "document_sha256": "e67a69bc019e7dc5e1a9c5184190901ca147b5f585ac4d1605ca23fde2b1b20c", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "mimopro-gsm8k", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "results/a13/gsm8k200.json", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/gsm8k200.json", "document_sha256": "4a281c460a7fa3a07b41cf3856faeba767c229a6d43b874ee7ff5d6729865cff", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "mimopro-a13-long8", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "results/a13/long-extraction-c8.txt", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a13/long-extraction-c8.txt", "document_sha256": "443ce4520c8472347ac496024483ff46ab1b9a6a3851803d64bf382b66a26f37", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "mimopro-a14-long8", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "results/a14/long-extraction-c8.txt", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction-c8.txt", "document_sha256": "36b64993a9c32e4b08be0ee600aa743e3fe4c62c917a6ae3ca05f9560c42233a", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "mimopro-a14-long1", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "results/a14/long-extraction.txt", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/results/a14/long-extraction.txt", "document_sha256": "aa85e6233aaf60ab708d6580efada76287b2d560b4cc65f76f72f1cd3f5ab0d3", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "mimopro-score-method", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "eval/gsm8k_eval.py", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/eval/gsm8k_eval.py", "document_sha256": "207a2f4d38a1870a876bd62f8b9070baaf2a07faccbf25ac9ffe194c64b53353", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "mimopro-speed-method", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "bench/mimobench.py", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/mimobench.py", "document_sha256": "d0425c39d1e5ac0f768ea8a451e39a3ad552e8808c2a5c269061faea3bb0b918", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "mimopro-long1-method", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "bench/long_extraction_c1.py", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/long_extraction_c1.py", "document_sha256": "3012886b036a75e049296c0d9a02aff20898003c806821fd39e1b8405eb6aabd", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "mimopro-long8-method", "parent_record_id": "mimopro-tp8-sglang-scheduler", "publisher": "rhys101", "repository": "rhys101/MiMo-V2.6-Pro-RL-Spark-8", "revision": "2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10", "document_path": "bench/long_extraction_c8.py", "pinned_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/bench/long_extraction_c8.py", "document_sha256": "3c27bb057d58d7afef9fd137bb7a67bb9341a05c85e0f29947e12362b37c685a", "last_commit_at": "2026-09-24T18:54:01Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" } ] } ``` ## Paired source quality comparisons ```json { "quality_comparisons": [ { "benchmark": "GSM8K first 200, build a13", "correct": 196, "total": 200, "wrong_ids": [ 93, 119, 162, 187 ], "shots": 0, "thinking": false, "a14_rerun": false, "leaderboard_score": false, "full_test": false } ] } ``` ## memory observations ```json { "memory_observations": [ { "kind": "weight storage / rank fit description, not peak RAM", "weight_files_GB": 535, "weights_per_rank_GB": [ 66, 69 ], "peak_runtime_RAM": null, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 118, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L118" } ] } ``` ## long context cases ```json { "long_context_cases": [ { "actual_input_tokens": 127803, "builds": [ "a5", "a6" ], "three_facts": true, "current_a13_a14_rerun": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 84, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L84" }, { "actual_input_tokens": 244168, "builds": [ "a5" ], "three_facts": true, "current_a13_a14_rerun": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 84, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L84" } ] } ``` ## case reports ```json { "case_reports": [ { "id": "a13-eighth-stream-queued", "classification": "source_reported_scheduler_slot_accounting_bug", "initial_build": "a13", "fixed_build": "a14", "initial_C8_aggregate": 234.57, "fixed_C8_aggregate": 306.66, "initial_tokens": 43908, "fixed_tokens": 43900, "initial_wall_seconds": 187.18, "fixed_wall_seconds": 143.16, "initial_correct": 24, "fixed_correct": 24, "total": 24, "initial_mean_per_stream": 39.46, "fixed_mean_per_stream": 38.4, "initial_max_TTFT_seconds": 46.39, "fixed_max_TTFT_seconds": 8.17, "C1_fixture_input_tokens": 1498, "C8_input_counts_retained": false, "universal_long_context_fix": false, "source_document_id": "mimopro-tp8-readme", "source_document_sha256": "97c24b080dd443c8d0c8479f21c09dfb66ffad3d7d4042e1a3acbb399a9c1bef", "source_line": 49, "evidence_url": "https://github.com/rhys101/MiMo-V2.6-Pro-RL-Spark-8/blob/2f3e5fd804d341ff55a7a7cfdef1d9b782c80c10/README.md#L49" } ] } ``` ## quality comparisons ```json { "quality_comparisons": [ { "benchmark": "GSM8K first 200, build a13", "correct": 196, "total": 200, "wrong_ids": [ 93, 119, 162, 187 ], "shots": 0, "thinking": false, "a14_rerun": false, "leaderboard_score": false, "full_test": false } ] } ``` ---