Qwen Long Context: Concurrency, Memory and Failure Evidence

Original reports only. No new Run AI Here model execution.

Long context changes the question a deployment report needs to answer. A server can accept a large window, submit several requests and report a high token rate without keeping every long document resident at once. Useful evidence records the actual request shape, overlapping execution, queue, prefix cache, completion length and memory sampling window together. This note reads original Qwen3.8-Flash-Next reports with those distinctions intact. Run AI Here has not repeated their model tests.

One-card long batches: what actually overlapped

The Penny Royal original results contain September 11 v2.5.0 observations on one RTX PRO 6000 Blackwell, TP1. The relevant configuration uses RadixArk ModelOpt NVFP4, RAM PLE, FP8 KV, FR-Spec/native NEXTN, a 524,288-token request limit, an 824,384-token pool, 24 Mamba slots and 32 GiB HiCache/NIXL. “Online FP8” changes selected projections; it does not mean every model component changes precision.

Each value is one batch observation. Aggregate means all output tokens divided by synchronized group makespan, including first-token waiting. Requests end at natural stop or a 4,096-token cap, so the arms produce different totals. These are contextual batch rates, with no repeated-run dispersion. The online arm records four-way overlap for 30.152 seconds at 128K and 24.850 seconds at 490K. The source does not establish four fully distinct, full-length resident prefix copies. Exact per-request input counts and cache sharing are missing in this summary.

One batch per cell. These aggregate rates include synchronized group wall time and different completion totals.

The matching online-FP8 C1 observations use exactly 1,024 output tokens: 195.634 tokens/s after the first token at 128K, and 172.644 at 490K. TTFT is 12.906 and 63.296 seconds respectively. Those long observations are single samples. The source explicitly says cold TTFT did not improve against its option-off samples. A higher generation rate therefore need not shorten the complete request by the same proportion. Several release-state changes also accompany the precision option, preventing an isolated FP8 causal claim.

The newer release is a separate experiment

October 10 Penny Royal 3.0 release metadata identifies another checkpoint, adaptive MTP and a 1,067,072-token pool. Its cold inputs are exactly 63,921 and 489,915 tokens; reported native mean prefill is 17,235.90 and 8,839.75 tokens/s, with TTFT 4.71 and 60.37 seconds. Three-key recall passes in the longer input. Completion length for those cold cases is not supplied here.

The 3.0 regular screens instead use 128 input and 1,024 output tokens, three repeats, temperature 0.8, top-p 0.95, top-k 40, min-p 0.05 and xhigh reasoning. Native means are 218.63 tokens/s at C1 and 489.35 aggregate at C4. They weight complete active reporting intervals, excluding idle/prefill-crossing intervals and partial tails. They do not measure complete agent-task wall time or replace the older long C4 cells. The accepted public core and the local checkout with private diagnostics have separate full identities. A document commit, public release image and actual native execution are separate provenance fields.

A four-card matrix provides a real campaign peak

The CS Technology Services original report uses four RTX PRO 6000 Max-Q cards, TP4, PCIe Gen4 without NVLink, vLLM 0.1.dev20073+g8e685d198 and harness v0.4.29. At nominal 128K, C1/C8/C16 aggregate rates are 128.5/505.0/729.9 tokens/s. Each cell lasts 30 seconds with ignore_eos=true and an 8,192-token output cap. Effective concurrency was checked. C32 is omitted for capacity, not an observed crash.

Its 374,976 MiB sampled peak is summed used VRAM across the whole matrix, over 501 samples in 1,206 seconds. Per-cell peak and host peak RAM are absent. Matrix TTFT is warm-prefix/queue time; cold C1 takes 24.25 seconds at 128,021 actual input tokens. Per-user rates in that report are derived by division. Image digest, weight revision and full launch command were not retained. This is a different deployment, without a controlled comparison to Penny Royal.

GiB and MiB are retained. Free, used, KV occupancy and host RAM are different quantities.

Penny Royal's 7.52 GiB free after graph capture is startup headroom. Its 1,897 MiB sampled minimum free belongs to ten GPU-media checks. Neither is the long C4 peak used memory. Keep free versus used, one versus four cards, startup versus workload and GPU versus host measurements separate. Pool occupancy is another counter again; percentages cannot be converted into whole-process bytes without an established mapping.

Sustained submissions can still queue

The credited community follow-up describes a separate operator's v2.5.0 configuration with a 1,114,304-token pool. Ten rounds submit four approximately 240K requests with 8,192 reported output tokens each. The author reports 93–107 seconds per round, 40/40 HTTP 200 and no OOMs, retractions or aborts. Queuing is observed and pool occupancy peaks at 57 percent. This supports sustained operation for that workload; it does not establish four running throughout, near-full-pool stress, exact retrieval correctness, peak RAM or a changed runtime default. The approximate input label remains approximate.

A state-cache failure below the request ceiling

An original SGLang incident comment records approximately 140K cold input, a 262K-class window, FP8 E4M3 KV, no speculation, 4,096-token prefill chunks, 51 Mamba slots and a 12-request ceiling. HiCache uses write-through. Only two requests are running when Mamba usage jumps from 0.18 to 1.00 while full-token usage is 0.46. The scheduler raises Can not alloc mamba cache; its child exits and the container reload interrupts service. The reporter mentions one queued request, while the cited failure log shows zero at that instant. Both observations are retained.

Source-reported outage. The proposed fallback and CPU tests do not prove every GPU workload fixed.

The related PR37619 remains open and unmerged at this check. Its author reports two CPU tests and no rerun of the GPU reproducer. The proposed fallback skips an optional unfinished checkpoint when slots cannot be allocated. That scope does not establish a universal fix or a new speed result. Request slots, KV capacity and held Mamba states are separate constraints.

For deployment selection, first match hardware, exact checkpoint, runtime, cache policy and requested output. Then ask whether the source observes simultaneous execution and measures memory over the same interval. Preserve a null where that evidence is missing. This produces a more useful capacity decision than combining the fastest token rate, largest configured window and smallest memory number from different runs.