THE SOURCE IN CONTEXT
A bounded gather lowers the isolated transient and the 500K single-request device peak, but the paired two-250K host-KV cell reports the same +1248 MiB non-pool peak on both trees. Production failure, development recovery and the smaller ship-tree rerun have different conditions.
Which memory observations are directly paired?
The original report compares cold 2x250K prefill on the same Server Edition 600 W card. Minimum free device memory is 3414 MiB on the on-GPU control versus 6962 MiB with host KV; non-pool peaks over idle are 1090 versus 1248 MiB. Pools differ, so this is an operational configuration comparison. Another same-host-KV old/new gather comparison reports +1248 MiB in both arms, with no measurable peak benefit in that concurrent cell.
Source: nanogenomic · pinned document nanogenomic · pinned document
Where does bounded gather reduce the peak?
The isolated 524288-token FP8 kernel reports 1024 to 32 MiB transient allocation, excluding output, with bit-exact checks in two runs. The in-engine single-request 500000-token sweep reports 97237 to 96093 MiB device used peak at the same 689728-token pool and 94661 MiB idle. These are different measurement scopes and cannot replace the concurrent cell. Host-UVA timing fails to reproduce consistently.
Source: nanogenomic · pinned document
What failed in production?
The September 30 Max-Q Workstation 300 W lane fails an eager-prefill 560 MiB allocation with 280.94 MiB free and 93.92 GiB process device usage. At failure one request runs, fourteen queue, token pool usage is 0.73 and Mamba usage 0.06. The outage is 12m54s. Monitoring existed but was not consumed by admission. No total host peak or exact failing request input is published.
Source: nanogenomic · pinned document
How much recovery was actually retested?
The host-KV-off development branch has 45 and 42 recoveries across two injection/real-pressure batteries. The ship tree first has an invalid harness run with zero injections; its corrected rerun has two injections and two recoveries. Decode OOM and driver errors remain fatal. The composite gate lacks NIAH cells in its own bundle even though a separate invocation passes nine needles.
Source: nanogenomic · pinned document
Reported figures and their conditions
Selected facts reported by the original author. Each row has its own configuration and measurement boundary. Run AI Here has not reproduced these tests.
| Configuration / workload | Metric | Reported value | Evidence |
|---|---|---|---|
| on-GPU control minimum free VRAM | available GPU memory | 3414 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| on-GPU control non-pool peak over idle | non pool memory over idle | 1090 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| host KV minimum free VRAM | available GPU memory | 6962 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| host KV non-pool peak over idle | non pool memory over idle | 1248 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| 524288 FP8 old gather isolated transient | kernel transient memory | 1024.0 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| 524288 FP8 bounded gather isolated transient | kernel transient memory | 32.0 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| 500K old gather device peak | peak device used memory | 97237 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| 500K bounded gather device peak | peak device used memory | 96093 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| 2x250K host-KV old gather non-pool peak | non pool memory over idle | 1248 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| 2x250K host-KV bounded gather non-pool peak | non pool memory over idle | 1248 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| Production OOM failed_allocation | failed allocation | 560.0 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| Production OOM available_GPU_memory | available GPU memory | 280.94 MiBsource-reported observation; dispersion unreported | Original row ↗ |
| Production OOM process_device_memory | process device memory | 93.92 GiBsource-reported observation; dispersion unreported | Original row ↗ |
on-GPU control minimum free VRAM · available GPU memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 2 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.15b |
| cache_state | cold prefill, two approximately 250K requests |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| arm | on-GPU control |
| memory_scope | during cold 2x250K prefill |
| metric_kind | minimum_free_device_memory |
on-GPU control non-pool peak over idle · non pool memory over idle · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 2 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.15b |
| cache_state | cold prefill, two approximately 250K requests |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| arm | on-GPU control |
| memory_scope | during cold 2x250K prefill |
| metric_kind | peak_device_nonpool_delta_over_idle |
host KV minimum free VRAM · available GPU memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 2 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.15b |
| cache_state | cold prefill, two approximately 250K requests |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| arm | host KV |
| memory_scope | during cold 2x250K prefill |
| metric_kind | minimum_free_device_memory |
host KV non-pool peak over idle · non pool memory over idle · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 2 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.15b |
| cache_state | cold prefill, two approximately 250K requests |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| arm | host KV |
| memory_scope | during cold 2x250K prefill |
| metric_kind | peak_device_nonpool_delta_over_idle |
524288 FP8 old gather isolated transient · kernel transient memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | Not specified |
| input_tokens | 524288 |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.16 microbenchmark |
| cache_state | Not specified |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | 2 |
| arm | old gather |
| memory_scope | gather allocation only, excluding output |
| bit_exact_all_12_cases_both_runs | True |
524288 FP8 bounded gather isolated transient · kernel transient memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | Not specified |
| input_tokens | 524288 |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.16 microbenchmark |
| cache_state | Not specified |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | 2 |
| arm | bounded gather |
| memory_scope | gather allocation only, excluding output |
| bit_exact_all_12_cases_both_runs | True |
500K old gather device peak · peak device used memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 1 |
| input_tokens | 500000 |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.16 in-engine A/B |
| cache_state | same-pool in-engine NIAH |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | three depths at each of 100000/300000/500000 |
| arm | old gather |
| memory_scope | in-engine device used peak across three needle depths |
| KV_pool_tokens | 689728 |
| idle_device_MiB | 94661 |
| needle_passes | 9 |
| needle_trials | 9 |
500K bounded gather device peak · peak device used memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 1 |
| input_tokens | 500000 |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.16 in-engine A/B |
| cache_state | same-pool in-engine NIAH |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | three depths at each of 100000/300000/500000 |
| arm | bounded gather |
| memory_scope | in-engine device used peak across three needle depths |
| KV_pool_tokens | 689728 |
| idle_device_MiB | 94661 |
| needle_passes | 9 |
| needle_trials | 9 |
2x250K host-KV old gather non-pool peak · non pool memory over idle · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 2 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.16 concurrent A/B |
| cache_state | cold prefill, two approximately 250K requests |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| arm | old gather |
| memory_scope | paired concurrent host-KV cold-prefill peak over idle |
| measurable_peak_improvement | False |
2x250K host-KV bounded gather non-pool peak · non pool memory over idle · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | one RTX PRO 6000 Blackwell Server Edition, 600 W, PCIe Gen5 x16, driver 580.95.05; 97887 MiB |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 2 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 to 2026-10-01 source campaign; exact validation-cell timestamps unreported |
| host_peak_RAM | Not specified |
| source_section | 3.16 concurrent A/B |
| cache_state | cold prefill, two approximately 250K requests |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| arm | bounded gather |
| memory_scope | paired concurrent host-KV cold-prefill peak over idle |
| measurable_peak_improvement | False |
Production OOM failed_allocation · failed allocation · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02 |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 1 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 |
| host_peak_RAM | Not specified |
| source_section | 3.14 production failure |
| cache_state | Not specified |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| queued_requests | 14 |
| pending_tokens | 1485849 |
| full_token_usage | 0.73 |
| mamba_usage | 0.06 |
| KV_pool_tokens | 522880 |
| memory_fraction_static | 0.98 |
| other_process_device_MiB | 676 |
| exact_failed_request_input_tokens | Not specified |
| memory_scope | allocator failure snapshot; not a sampled campaign peak |
Production OOM available_GPU_memory · available GPU memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02 |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 1 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 |
| host_peak_RAM | Not specified |
| source_section | 3.14 production failure |
| cache_state | Not specified |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| queued_requests | 14 |
| pending_tokens | 1485849 |
| full_token_usage | 0.73 |
| mamba_usage | 0.06 |
| KV_pool_tokens | 522880 |
| memory_fraction_static | 0.98 |
| other_process_device_MiB | 676 |
| exact_failed_request_input_tokens | Not specified |
| memory_scope | allocator failure snapshot; not a sampled campaign peak |
Production OOM process_device_memory · process device memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | community NVFP4; exact upstream artifact not identified in the selected report |
| model_revision | Not specified |
| runtime_execution_revision | Not specified |
| executed_image_digest | Not specified |
| engine | SGLang-derived patched QSA host-KV / bounded-gather branches |
| engine_base_short | 6fa3fe69e2 |
| system | RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16, driver 610.43.02 |
| node_count | 1 |
| parallelism | TP1 |
| weight_precision | NVFP4 |
| kv_dtype | FP8 |
| PLE | host pinned RAM |
| concurrency | 1 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| temperature | Not specified |
| thinking | Not specified |
| experiment_date | 2026-09-30 |
| host_peak_RAM | Not specified |
| source_section | 3.14 production failure |
| cache_state | Not specified |
| control_pool_tokens | 451840 |
| host_KV_pool_tokens | 1310720 |
| controlled_pool_sizes_equal | False |
| repetitions | Not specified |
| queued_requests | 14 |
| pending_tokens | 1485849 |
| full_token_usage | 0.73 |
| mamba_usage | 0.06 |
| KV_pool_tokens | 522880 |
| memory_fraction_static | 0.98 |
| other_process_device_MiB | 676 |
| exact_failed_request_input_tokens | Not specified |
| memory_scope | allocator failure snapshot; not a sampled campaign peak |
Local Spark Qwen draft; Codex source audit and rewrite. No site model benchmark.