Qwen Flash-Next: Concurrency Has a Latency Cost

Qwen3.8-Flash-Next can serve more requests while each user waits longer. The useful question is which request mix was actually tested. A single-Spark SGLang study publishes a ShareGPT concurrency sweep; a separate two-Spark vLLM/b12x study publishes task throughput and cold-prefix latency. Their different checkpoints, cache policies and hardware prevent a controlled engine ranking. A third operator's memory failure explains why a configured window and a scheduler limit are insufficient capacity evidence.

Single Spark: read latency beside aggregate throughput

The nabe2030 recipe uses senfu/Qwen3.8-Flash-Next-NVFP4 at 5d37b3b3711d8406174b96ff950c0aa16324b266, SGLang commit 4fb9b5b5bad41ac7de5586ec812c421fe3aa2b8b, and NVMe file-backed PLE. Experts and the LM head use NVFP4; the dense path uses FP8. Those details distinguish it from the other Flash-Next conversions on this site.

Its aggregated result table uses vllm bench serve with 200 ShareGPT prompts, thinking off, MTP off and radix caching disabled. One warmup request is discarded. The C1 summary is the median of three trials; the higher-concurrency points are single trials. All selected senfu cases completed 200 requests without errors. Individual prompt/output distributions, temperature and per-request cache history are not supplied.

For a shared endpoint, more total output does not preserve the interactive experience. Keep first-token wait and token spacing beside aggregate throughput when deciding how many users to admit. Do not estimate an individual stream's speed by dividing aggregate tokens/s by concurrency: the timing definitions and request completion patterns need not match. Single higher-load trials also provide no run-to-run error bar. These results are useful measurements of this workload, rather than a latency guarantee for a different team.

Two Sparks: preserve the older concurrency cohort

The TP2 repository contains several generations of measurements. The selected k76 concurrency series belongs to recipe v2.0.0 on October 10, using two directly connected Sparks and llama-benchy 0.4.1.dev4+g0d4de4271. It requests 2,048 input and 512 output tokens, temperature 1.0, top-p 0.95, top-k 20 and thinking on, with prefix caching enabled. It reports three runs within one boot.

The version ledger binds v2.0.0 to GDN-MSE checkpoint 16c9bd54788d12390838a65ce4a4ecda97fa5f1d. The later v2.1.0/v2.2.0 checkpoint ends in ...85406 and includes a different MTP drafter. Keeping the old checkpoint identity prevents newer recipe defaults from silently rewriting historical results. The ledger's release-image digest is retained separately from the unreported digest of the exact executed test container.

Both columns use tokens/s. The machine-readable source CSV retains the exact values and original run paths. Aggregate throughput divides all output by elapsed wall time including prompt processing. Per-request decode describes one stream after its first token. Their denominators differ; neither column can be reconstructed by simply multiplying or dividing the other by the number of users.

The larger C8 aggregate SD remains visible. It describes dispersion in those three runs; it does not identify a cause, quantify between-boot variation or establish a confidence interval. Reporting mean and SD does not itself require a normally distributed population. A scheduling explanation would need more evidence than the summary cells provide.

Long-context latency comes from a separate release

The v2.2.0 cold-prefix arm uses the later checkpoint, C1, temperature zero and one requested output token. It averages four boots with three cold prompts per boot. SD is between boot means. The method and build notes separate this prefill work from the older concurrency campaign.

A cold prefix means there was no matching prefix-cache hit. It does not establish cold weight pages, cold compilation caches or a cold operating-system page cache. One requested output token measures the route to the first answer token, rather than long-answer generation. These points cannot prove that sixteen simultaneous 128K prompts fit or retain the same latency. Read the earlier memory/context explanation for the distinction between a shared KV pool and a request ceiling.

Memory failure is useful evidence too

In a separate first-person dual-GB10 operator report, jcagle describes vLLM 0.30.0 with MiaAI overlays, FP8 KV, three MTP drafts, 16 sequence slots and capped GPU clocks. This is a third configuration, rather than another repetition of k76. The report is post version 3, updated October 13; its public URL is mutable, so a dated snapshot hash is retained in the case export instead of claiming an immutable commit URL.

The operator reports an eight-request deep-prefill burst that reached 2.4 GiB head-node MemAvailable at GMU 0.80 before earlyoom killed the engine. With GMU 0.78, the reported eight-by-64K-class load completed at 2,510 combined prefill tokens/s and 117 seconds TTFT, with 3.8 GiB available on the head. Exact individual token counts, output lengths, sampling and repeated-run dispersion are missing. These observations demonstrate the operator's outcome; they do not make 0.78 a universal safe setting or isolate GMU as the only causal variable.

The single-Spark SGLang recipe separately lists a 49,152-token per-request window, 393,216 total tokens and eight slots for one configuration. That configuration is not evidence that every slot was simultaneously filled. Its 3.04 GB KV figure at 128K is a source-labelled allocation, with byte convention unverified, rather than whole-model peak RAM. The KV and speculation guide explains why those distinctions matter.

A practical reading order

Choose the source matching your checkpoint, runtime and hardware. Next, compare the tested prompt/output mix with your actual workload. Then inspect aggregate output, per-request decode, first-token delay and host memory together. Finally, mark configured capacities that the source did not actually fill. This is an editorial method for choosing what to verify, not a request to reproduce these benchmarks automatically.

Does C32 throughput prove C32 long-context serving?

No. The single-Spark C32 ShareGPT mix does not publish a simultaneous full-window fill. Its prompt distribution is also absent.

Can the two Spark articles rank SGLang against vLLM?

No controlled engine comparison is available here. Checkpoints, nodes, cache behavior, MTP and request shapes differ.

Which missing fields should an AI retain?

Keep null execution digests, unknown request lengths, trial counts and memory scope. Use the For AI filters to retrieve complete reports and their conditions rather than a speed-only fragment.