{ "id": "A19", "path": "/guides/qwen-flash-next-paired-memory-oom-recovery/", "title": "Qwen Long Concurrency: Paired Memory Peaks and OOM Recovery", "summary": "Read two 250K-request memory controls, a 500K single-request peak, a production OOM and invalidated/repeated recovery tests without mixing hardware or cohorts.", "answer": "A bounded gather lowers the isolated transient and the 500K single-request device peak, but the paired two-250K host-KV cell reports the same +1248 MiB non-pool peak on both trees. Production failure, development recovery and the smaller ship-tree rerun have different conditions.", "published_on": "2026-10-18", "modified_on": "2026-10-18", "source_checked_on": "2026-10-18", "sources": [ { "publisher": "nanogenomic", "repository": "nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000", "platform": "github", "revision": "11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2", "document_path": "BENCHMARKS.md", "pinned_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md", "document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "last_commit_at": "2026-10-01T04:27:43Z", "url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000", "report_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md", "checked_on": "2026-10-18", "metadata_status": "checked", "record_id": "qwen-flash-next-nanogen-memory-recovery" } ], "supplemental_sources": [ { "document_id": "qwen-nanogen-readme", "parent_record_id": "qwen-flash-next-nanogen-memory-recovery", "publisher": "nanogenomic", "repository": "nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000", "revision": "11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2", "document_path": "README.md", "pinned_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/README.md", "document_sha256": "bcfd94276d2bebef681e8a7737444ecec9814d6c73dce9ecf95a0b9aa437a3ab", "last_commit_at": "2026-10-01T04:27:43Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" }, { "document_id": "qwen-nanogen-commits", "parent_record_id": "qwen-flash-next-nanogen-memory-recovery", "publisher": "nanogenomic", "repository": "nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000", "revision": "11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2", "document_path": "COMMIT-MAP.tsv", "pinned_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/COMMIT-MAP.tsv", "document_sha256": "7a6b188d895f6d99fe46ca650832e7d1d8f182aa0b2c59e1442bd7794042fa92", "last_commit_at": "2026-10-01T04:27:43Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields" } ], "facts": [], "reported_results": [ { "label": "on-GPU control minimum free VRAM", "metric": "available_GPU_memory", "value": 3414, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 870, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "on-GPU control non-pool peak over idle", "metric": "non_pool_memory_over_idle", "value": 1090, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 871, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "host KV minimum free VRAM", "metric": "available_GPU_memory", "value": 6962, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 870, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "host KV non-pool peak over idle", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 871, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "524288 FP8 old gather isolated transient", "metric": "kernel_transient_memory", "value": 1024.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "old gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 954, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "524288 FP8 bounded gather isolated transient", "metric": "kernel_transient_memory", "value": 32.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "bounded gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 954, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "500K old gather device peak", "metric": "peak_device_used_memory", "value": 97237, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "old gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 972, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "500K bounded gather device peak", "metric": "peak_device_used_memory", "value": 96093, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "bounded gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 972, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "2x250K host-KV old gather non-pool peak", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "old gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 981, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "2x250K host-KV bounded gather non-pool peak", "metric": "non_pool_memory_over_idle", "value": 1248, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "bounded gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 981, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "Production OOM failed_allocation", "metric": "failed_allocation", "value": 560.0, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 639, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L639", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "Production OOM available_GPU_memory", "metric": "available_GPU_memory", "value": 280.94, "unit": "MiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 640, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L640", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" }, { "label": "Production OOM process_device_memory", "metric": "process_device_memory", "value": 93.92, "unit": "GiB", "statistic": "source-reported observation; dispersion unreported", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 642, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L642", "source_record_id": "qwen-flash-next-nanogen-memory-recovery" } ], "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null }, "generation": { "text_draft": "local Spark Qwen, then Codex factual review and rewrite; actual receipt in editorial-batch8 report", "review": "Run AI Here with Codex", "image": "deterministic matplotlib data figures and evidence maps", "local_first": true }, "image": { "url": "/assets/batch8-qwen-memory-hero.webp", "alt": "Qwen: the concurrent peak did not fall. Both trees report +1,248 MiB. This is a paired concurrent delta, not total GPU or host RAM.", "width": 1600, "height": 900, "sha256": "a81ca1581ada8f1ba340bc46f1d4d2c624cccc50866af8d4ae763ebd2b712719", "caption": "Both trees report +1,248 MiB. This is a paired concurrent delta, not total GPU or host RAM." }, "case_reports": [ { "id": "deep-queue-production-OOM", "classification": "source_reported_real_allocator_failure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 633, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L633", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "downtime_seconds": 774, "error": "torch.OutOfMemoryError in eager prefill; scheduler SIGQUIT and process-tree termination", "monitoring_alert_before_crash_seconds": 53, "admission_consumed_alert": false }, { "id": "recovery-branch-versus-ship", "classification": "source_reported_fault_injection_and_real_pressure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 999, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L999", "branch": "oom-retract, host KV off", "branch_pool_tokens": [ 451712, 451840 ], "runs": [ { "forced": 39, "real": 6, "recoveries": 45 }, { "forced": 39, "real": 3, "recoveries": 42 } ], "engine_alive": true, "scheduler_suite": { "initial_harness_errors": 9, "initial_error": "runner IndexError", "rerun_passed": 100 }, "ship_tree": { "initial_valid": false, "initial_error": "BatchEncoding not JSON serializable", "initial_injects": 0, "initial_recoveries": 0, "initial_verdict": false, "rerun_injections": 2, "rerun_recoveries": 2, "rerun_all_needles_correct": true, "engine_alive": true, "host_KV_pool_tokens": 1310720 }, "recovery_predicates": [ "overlap enabled", "EXTEND", "TP1", "PP1", "no MLP sync", "no disaggregation", "HiSparse off", "PDMux off", "no diffusion config", "device-wide drain completes" ], "max_retractions": 3, "exhaustion_status": 503, "next_request_status": 200, "decode_OOM_recovered": false, "driver_error_recovered": false, "peer_outputs_bit_identical": false }, { "id": "host-UVA-regression", "classification": "unresolved_nonreproducing_microbenchmark", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 992, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L992", "first_new_ms": [ 341, 376 ], "later_new_ms": [ 4.99, 5.68 ], "later_new_peak_MiB": [ 159, 189 ], "first_touch_explanation_proven": false, "kernel_time_improvement_claim": false }, { "id": "composite-gate-missing-same-bundle-NIAH", "classification": "harness_composition_failure", "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 873, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L873", "decode_path_gate": true, "standalone_needle_correct": 9, "standalone_needle_total": 9, "same_bundle_NIAH_cells": 0, "long_context_gate": false, "overall_gate": false, "model_quality_failure_established": false } ], "memory_observations": [ { "label": "on-GPU control minimum free VRAM", "value": 3414, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870" }, { "label": "on-GPU control non-pool peak over idle", "value": 1090, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "on-GPU control", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871" }, { "label": "host KV minimum free VRAM", "value": 6962, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "minimum_free_device_memory" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L870" }, { "label": "host KV non-pool peak over idle", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.15b", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "host KV", "memory_scope": "during cold 2x250K prefill", "metric_kind": "peak_device_nonpool_delta_over_idle" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L871" }, { "label": "524288 FP8 old gather isolated transient", "value": 1024.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "old gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954" }, { "label": "524288 FP8 bounded gather isolated transient", "value": 32.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": null, "input_tokens": 524288, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 microbenchmark", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": 2, "arm": "bounded gather", "memory_scope": "gather allocation only, excluding output", "bit_exact_all_12_cases_both_runs": true }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L954" }, { "label": "500K old gather device peak", "value": 97237, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "old gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972" }, { "label": "500K bounded gather device peak", "value": 96093, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": 500000, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 in-engine A/B", "cache_state": "same-pool in-engine NIAH", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": "three depths at each of 100000/300000/500000", "arm": "bounded gather", "memory_scope": "in-engine device used peak across three needle depths", "KV_pool_tokens": 689728, "idle_device_MiB": 94661, "needle_passes": 9, "needle_trials": 9 }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L972" }, { "label": "2x250K host-KV old gather non-pool peak", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "old gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981" }, { "label": "2x250K host-KV bounded gather non-pool peak", "value": 1248, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 2, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported", "host_peak_RAM": null, "source_section": "3.16 concurrent A/B", "cache_state": "cold prefill, two approximately 250K requests", "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "arm": "bounded gather", "memory_scope": "paired concurrent host-KV cold-prefill peak over idle", "measurable_peak_improvement": false }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L981" }, { "label": "Production OOM failed_allocation", "value": 560.0, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L639" }, { "label": "Production OOM available_GPU_memory", "value": 280.94, "unit": "MiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L640" }, { "label": "Production OOM process_device_memory", "value": 93.92, "unit": "GiB", "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "community NVFP4; exact upstream artifact not identified in the selected report", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived patched QSA host-KV / bounded-gather branches", "engine_base_short": "6fa3fe69e2", "system": "RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02", "node_count": 1, "parallelism": "TP1", "weight_precision": "NVFP4", "kv_dtype": "FP8", "PLE": "host pinned RAM", "concurrency": 1, "input_tokens": null, "output_tokens": null, "temperature": null, "thinking": null, "experiment_date": "2026-09-30", "host_peak_RAM": null, "source_section": "3.14 production failure", "cache_state": null, "control_pool_tokens": 451840, "host_KV_pool_tokens": 1310720, "controlled_pool_sizes_equal": false, "repetitions": null, "queued_requests": 14, "pending_tokens": 1485849, "full_token_usage": 0.73, "mamba_usage": 0.06, "KV_pool_tokens": 522880, "memory_fraction_static": 0.98, "other_process_device_MiB": 676, "exact_failed_request_input_tokens": null, "memory_scope": "allocator failure snapshot; not a sampled campaign peak" }, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L642" } ], "long_context_cases": [ { "id": "host-KV-two-mains-turn3", "input_length_source_labels": [ "250K", "200K" ], "additional_subagents": 2, "host_KV_pool_tokens": 1310720, "control_pool_tokens": 451840, "host_KV_TTFT_seconds": [ 1.2, 1.0 ], "control_TTFT_seconds": [ 22.9, 41.1 ], "host_KV_cache_hit_rate": 0.9949, "control_cache_hit_rate": 0, "control_requests_completed": true, "host_peak_RAM": null, "source_document_id": "qwen-nanogen-bench", "source_document_sha256": "7015f023635f4d1503e7e6df2ab373ebf87aafbce2cdd127c29f6ff2a8b77076", "source_line": 833, "evidence_url": "https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md#L833" } ], "schema_version": "1.0.0", "record_type": "source_backed_explainer", "evidence_state": "source_reported", "site_reproduced": false, "metrics": null, "performance_ranking_allowed": false, "body_markdown": "# Qwen Long Concurrency: Paired Memory Peaks and OOM Recovery\n\n![Qwen: the concurrent peak did not fall. Both trees report +1,248 MiB. This is a paired concurrent delta, not total GPU or host RAM.](/assets/batch8-qwen-memory-hero.webp)\n\n*Both trees report +1,248 MiB. This is a paired concurrent delta, not total GPU or host RAM.*\n\nThe most useful memory result in this Qwen3.8-Flash-Next report is a negative one. A change that cuts an isolated allocation and a 500K single-request peak does **not** reduce the measured peak in its two-request, approximately 250K-per-request production-like cell. Keeping that result prevents a deployment guide from turning a kernel improvement into an unsupported concurrency-capacity promise. Run AI Here has read the original measurements; it has not run these model experiments.\n\n## First separate the two physical cards\n\nThe [maintainer README](https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/README.md) distinguishes a production/reference RTX PRO 6000 Blackwell **Max-Q Workstation**, 300 W, PCIe Gen4 x16 and driver 610.43.02, from a rented **Server Edition**, 600 W, PCIe Gen5 x16 and driver 580.95.05 used for later validation. Both report 97,887 MiB device capacity. The host-KV, bounded-gather and recovery validation belongs to the Server Edition. The September 30 production crash belongs to the Max-Q lane. A shared product-family name does not make their latency or failure margins interchangeable, and no controlled cross-SKU scaling result is established.\n\nThe model is a community NVFP4 Qwen3.8-Flash-Next artifact with FP8 KV and host-resident components. The selected report does not give a complete upstream weight revision for these cells. Its patched SGLang branches and a base prefix are documented, but the exact full executed tree and image digest for every arm are missing. The pinned document revision identifies what was read, rather than silently supplying those execution identities.\n\n## The concurrent memory controls answer two questions\n\nIn [section 3.15b of the original benchmark ledger](https://github.com/nanogenomic/qwen-3-8-flash-next-RTX-PRO-6000/blob/11a09ff7fbf2cdb60dbccb2e345fdfb9aa43f2f2/BENCHMARKS.md), a cold two-by-250K prefill compares on-GPU KV against host KV on the same validation card. The control auto-sizes to a 451,840-token pool; host KV uses 1,310,720. The control pool was not intentionally pinned to match the production lane. This matters because placement and capacity change together.\n\n| During the concurrent cold prefill | On-GPU control | Host KV |\n|---|---:|---:|\n| Minimum free device memory, MiB | 3,414 | 6,962 |\n| Peak non-pool allocation above idle, MiB | 1,090 | 1,248 |\n\nThe host-KV arm has more minimum free VRAM while its non-pool increment is slightly larger. Those observations are compatible because they describe different accounting quantities. Neither column supplies peak whole-host RAM. Subtracting a free-memory observation from an unrelated device capacity or combining different idle baselines would create a result the author did not measure.\n\n![Host KV: concurrent prefill headroom. Cold 2 x 250K prefill: minimum free VRAM, not peak host RAM. Source section 3.15b.](/assets/batch8-qwen-headroom.webp)\n\n*Cold 2 x 250K prefill: minimum free VRAM, not peak host RAM. Source section 3.15b.*\n\nA separate old-versus-bounded-gather A/B keeps the host-KV setup and uses two cold approximately 250K requests. Both trees report **+1,248 MiB non-pool peak over idle**. The author explicitly declines an end-to-end peak improvement for this cell. The likely explanation that another allocation dominates is left as a hypothesis; the report does not isolate that allocation. A planner should therefore retain the equal peak when estimating this workload's headroom.\n\n## Why the 500K result can still improve\n\nSection 3.16 traces full-context prefill gather temporaries that grow with the entire request, even when prefill is chunked. In an isolated 524,288-token FP8 kernel microbenchmark, its own transient allocation falls from 1,024.0 to 32.0 MiB, excluding the output. Twelve cases in each of two runs are bit-exact with zero differing values. That is a kernel-level check, not proof that complete generated answers are deterministic or that every inference path is equivalent.\n\nThe in-engine A/B instead holds a 689,728-token pool and 94,661 MiB idle device usage. At 500,000 input tokens, old and new device-used peaks are 97,237 and 96,093 MiB. All nine needle cases across three lengths and three depths pass in both arms. This supports a lower peak for that single-request sweep, while leaving the paired concurrent no-improvement result intact. The smallest kernel case and the wide-int64 case also show why “always 32 MiB” would overstate the implementation's behavior.\n\n![500K: a different cell does improve. In-engine device peaks at 500,000 tokens. Both arms pass 9/9 needles over the full three-length sweep.](/assets/batch8-qwen-500k-peak.webp)\n\n*In-engine device peaks at 500,000 tokens. Both arms pass 9/9 needles over the full three-length sweep.*\n\nHost-UVA timing has an unresolved reproduction problem. The first new-path run takes 341–376 ms, whereas later runs take 4.99–5.68 ms and report 159–189 MiB peaks. A first-touch explanation is suggested but not proven. The source excludes a host-UVA kernel-time improvement claim, and so does this guide.\n\n## Prefix loss can hurt without an HTTP error\n\nThe same-card two-main-agent test sends histories labelled 250K and 200K, then adds two subagents on turn three. Host KV retains prefixes of 249,984 and 200,000 tokens; the control evicts both. Host-KV first-token times are 1.2 and 1.0 seconds, against 22.9 and 41.1 seconds in the control. Every control request still completes. This is evidence of a cache-capacity failure mode rather than an HTTP failure or a general throughput comparison.\n\nThe source's 95.56 percent multi-turn cached fraction belongs to six conversations growing from about 60K to 80K, not to the 250K–500K main agents. Its two-520K host-KV case runs, but the on-GPU control cell does not execute. Keep that missing comparison instead of inventing a speedup.\n\n## The real production outage happened outside a full pool\n\nOn September 30 at 16:44:17, the Max-Q production lane fails an eager-prefill allocation of 560 MiB with 280.94 MiB free. The process reports 93.92 GiB device usage including non-PyTorch memory; another process uses 676 MiB. The configured static fraction is 0.98 and the pool has 522,880 tokens. The final log has one running request, fourteen queued, 1,485,849 pending tokens, token usage 0.73 and Mamba usage 0.06. Scheduler termination interrupts service for 12 minutes 54 seconds.\n\nThat snapshot shows why a token-pool percentage alone cannot bound every external allocation. It does not prove that power limit or PCIe generation caused the crash, nor establish a universal safe static fraction. The queue alert arrives 53 seconds before failure but does not feed admission. A monitoring signal and an enforced dispatch constraint are separate operational facts.\n\n## Recovery counts must keep their branch and invalid attempts\n\nThe development `oom-retract` battery runs with host KV **off**: one run has 39 injected and six real-pressure OOMs, with 45 recoveries; the second has 39 plus three, with 42. A scheduler suite initially has nine runner errors, then passes 100 tests after a harness correction. These are not forty-five ship-tree recoveries.\n\n![Recovery evidence must keep its sample size. The large branch battery is not a 45-event ship-tree validation. Decode and driver errors remain fatal.](/assets/batch8-qwen-recovery.webp)\n\n*The large branch battery is not a 45-event ship-tree validation. Decode and driver errors remain fatal.*\n\nThe first host-KV ship-tree injection attempt is invalid because request serialization fails: zero injections and zero recoveries occur. The valid rerun has two injections, two recoveries, correct needles and a live engine. Recovery requires the source's narrow overlap/EXTEND/TP1/PP1 conditions, the other exclusion predicates and a completed drain. After three retractions an unfit request gets 503; the next request succeeds. Decode OOM and driver failures stay fatal. Peer outputs are not bit-identical, including ordinary OOM-free repeats.\n\nFinally, a composite long-context gate has no NIAH cells in its own bundle and therefore remains false, despite nine separately passing needles. That is a harness-composition gap, not demonstrated model-quality failure. Useful next evidence would combine the real serving workload, repeated per-cell GPU and host peaks, exact executed artifacts and the assembled quality verdict. Those missing fields remain unknown in the JSON export.\n" }