Qwen Long Concurrency: Paired Memory Peaks and OOM Recovery

Both trees report +1,248 MiB. This is a paired concurrent delta, not total GPU or host RAM.

The most useful memory result in this Qwen3.8-Flash-Next report is a negative one. A change that cuts an isolated allocation and a 500K single-request peak does not reduce the measured peak in its two-request, approximately 250K-per-request production-like cell. Keeping that result prevents a deployment guide from turning a kernel improvement into an unsupported concurrency-capacity promise. Run AI Here has read the original measurements; it has not run these model experiments.

First separate the two physical cards

The maintainer README distinguishes a production/reference RTX PRO 6000 Blackwell Max-Q Workstation, 300 W, PCIe Gen4 x16 and driver 610.43.02, from a rented Server Edition, 600 W, PCIe Gen5 x16 and driver 580.95.05 used for later validation. Both report 97,887 MiB device capacity. The host-KV, bounded-gather and recovery validation belongs to the Server Edition. The September 30 production crash belongs to the Max-Q lane. A shared product-family name does not make their latency or failure margins interchangeable, and no controlled cross-SKU scaling result is established.

The model is a community NVFP4 Qwen3.8-Flash-Next artifact with FP8 KV and host-resident components. The selected report does not give a complete upstream weight revision for these cells. Its patched SGLang branches and a base prefix are documented, but the exact full executed tree and image digest for every arm are missing. The pinned document revision identifies what was read, rather than silently supplying those execution identities.

The concurrent memory controls answer two questions

In section 3.15b of the original benchmark ledger, a cold two-by-250K prefill compares on-GPU KV against host KV on the same validation card. The control auto-sizes to a 451,840-token pool; host KV uses 1,310,720. The control pool was not intentionally pinned to match the production lane. This matters because placement and capacity change together.

The host-KV arm has more minimum free VRAM while its non-pool increment is slightly larger. Those observations are compatible because they describe different accounting quantities. Neither column supplies peak whole-host RAM. Subtracting a free-memory observation from an unrelated device capacity or combining different idle baselines would create a result the author did not measure.

Cold 2 x 250K prefill: minimum free VRAM, not peak host RAM. Source section 3.15b.

A separate old-versus-bounded-gather A/B keeps the host-KV setup and uses two cold approximately 250K requests. Both trees report +1,248 MiB non-pool peak over idle. The author explicitly declines an end-to-end peak improvement for this cell. The likely explanation that another allocation dominates is left as a hypothesis; the report does not isolate that allocation. A planner should therefore retain the equal peak when estimating this workload's headroom.

Why the 500K result can still improve

Section 3.16 traces full-context prefill gather temporaries that grow with the entire request, even when prefill is chunked. In an isolated 524,288-token FP8 kernel microbenchmark, its own transient allocation falls from 1,024.0 to 32.0 MiB, excluding the output. Twelve cases in each of two runs are bit-exact with zero differing values. That is a kernel-level check, not proof that complete generated answers are deterministic or that every inference path is equivalent.

The in-engine A/B instead holds a 689,728-token pool and 94,661 MiB idle device usage. At 500,000 input tokens, old and new device-used peaks are 97,237 and 96,093 MiB. All nine needle cases across three lengths and three depths pass in both arms. This supports a lower peak for that single-request sweep, while leaving the paired concurrent no-improvement result intact. The smallest kernel case and the wide-int64 case also show why “always 32 MiB” would overstate the implementation's behavior.

In-engine device peaks at 500,000 tokens. Both arms pass 9/9 needles over the full three-length sweep.

Host-UVA timing has an unresolved reproduction problem. The first new-path run takes 341–376 ms, whereas later runs take 4.99–5.68 ms and report 159–189 MiB peaks. A first-touch explanation is suggested but not proven. The source excludes a host-UVA kernel-time improvement claim, and so does this guide.

Prefix loss can hurt without an HTTP error

The same-card two-main-agent test sends histories labelled 250K and 200K, then adds two subagents on turn three. Host KV retains prefixes of 249,984 and 200,000 tokens; the control evicts both. Host-KV first-token times are 1.2 and 1.0 seconds, against 22.9 and 41.1 seconds in the control. Every control request still completes. This is evidence of a cache-capacity failure mode rather than an HTTP failure or a general throughput comparison.

The source's 95.56 percent multi-turn cached fraction belongs to six conversations growing from about 60K to 80K, not to the 250K–500K main agents. Its two-520K host-KV case runs, but the on-GPU control cell does not execute. Keep that missing comparison instead of inventing a speedup.

The real production outage happened outside a full pool

On September 30 at 16:44:17, the Max-Q production lane fails an eager-prefill allocation of 560 MiB with 280.94 MiB free. The process reports 93.92 GiB device usage including non-PyTorch memory; another process uses 676 MiB. The configured static fraction is 0.98 and the pool has 522,880 tokens. The final log has one running request, fourteen queued, 1,485,849 pending tokens, token usage 0.73 and Mamba usage 0.06. Scheduler termination interrupts service for 12 minutes 54 seconds.

That snapshot shows why a token-pool percentage alone cannot bound every external allocation. It does not prove that power limit or PCIe generation caused the crash, nor establish a universal safe static fraction. The queue alert arrives 53 seconds before failure but does not feed admission. A monitoring signal and an enforced dispatch constraint are separate operational facts.

Recovery counts must keep their branch and invalid attempts

The development oom-retract battery runs with host KV off: one run has 39 injected and six real-pressure OOMs, with 45 recoveries; the second has 39 plus three, with 42. A scheduler suite initially has nine runner errors, then passes 100 tests after a harness correction. These are not forty-five ship-tree recoveries.

The large branch battery is not a 45-event ship-tree validation. Decode and driver errors remain fatal.

The first host-KV ship-tree injection attempt is invalid because request serialization fails: zero injections and zero recoveries occur. The valid rerun has two injections, two recoveries, correct needles and a live engine. Recovery requires the source's narrow overlap/EXTEND/TP1/PP1 conditions, the other exclusion predicates and a completed drain. After three retractions an unfit request gets 503; the next request succeeds. Decode OOM and driver failures stay fatal. Peer outputs are not bit-identical, including ordinary OOM-free repeats.

Finally, a composite long-context gate has no NIAH cells in its own bundle and therefore remains false, despite nine separately passing needles. That is a harness-composition gap, not demonstrated model-quality failure. Useful next evidence would combine the real serving workload, repeated per-cell GPU and host peaks, exact executed artifacts and the assembled quality verdict. Those missing fields remain unknown in the JSON export.