Qwen Flash-Next: Quantization, Memory and Long Context

A smaller quantized file, a lower RSS reading and a longer configured window describe different capabilities. For Qwen3.8-Flash-Next, the useful evidence here comes from two separate deployments: a one-Spark NVFP4/MXFP8 recipe with measured cold-prefix latency, and an RTX PRO 6000 experiment that changes how the n-gram PLE table is stored. Read their hardware, allocations and request shapes before estimating whether a setup fits.

The left panel shows original means and between-boot SD. The right panel retains source-labelled GB and distinguishes reclaimable mapped pages. The panels describe different machines; neither is whole-model peak RAM.

Start with the complete single-Spark artifact

The one-Spark recipe serves ursuciprian/Qwen3.8-Flash-Next-NVFP4-GDN-MSE at 03f4a0570496bbe741189f4a2a7d611679e85406. It also reads an MXFP8 prefill shard from local-inference-lab/Qwen3.8-Flash-Next-NVFP4 at 7c4f1bc1a2d6847e0cbc01ac6b823f00251de8dd. Record both identities. “NVFP4” alone omits the dense, attention and prefill paths, as well as the refitted four-draft MTP head.

The selected experiment is the v2.4.0 promotion arm on one 128 GB unified-memory DGX Spark, using vLLM and b12x. Its saved experimental recipe fixes the checkpoint and image tag. The build report describes the subsequent release image with an added compiled-cache layer. That release digest is useful provenance, but it is not silently substituted for an unreported digest of the exact cold-test image. Engine commit prefixes likewise remain prefixes when the complete executed revision is unavailable.

Read the actual 16K–128K cold-prefix workload

The source capability CSV selects C1, temperature zero and one requested output token. Each value averages four boots with three cold prompts per boot. The uncertainty is SD between boot means, not a confidence interval or the spread of every individual request.

These observations describe the cost of reading a fresh prefix. They do not measure a long answer after that prefix. “Cold prefix” excludes a matching prompt-cache hit; it does not say that weights, filesystem pages or compiled kernels were cold. The selected fields do not supply every request's actual completion count or all sampling parameters, so those gaps stay visible in JSON.

The author reports a separate retrieval gate through 128K and explicitly leaves larger one-Spark depths unmeasured in this snapshot. Keep a retrieval gate, this latency curve and the configured window as separate evidence. None proves general answer quality over the whole configured window.

Keep the KV pool separate from the request ceiling

The experimental recipe sets FP8 KV and an explicit 15,032,385,536-byte pool, or 14 GiB. The build report records an allocated capacity of 993,754 tokens. The per-request ceiling is 262,144 tokens and the scheduler accepts up to eight request slots.

Those three values are not a claim that eight full windows fit at once. Admission also depends on shared pool use, state and request scheduling. A token-capacity figure is not measured peak process memory. The recipe's gpu_memory_utilization=0.01 is a startup guard in conjunction with explicit KV bytes, as its own comment explains; it does not mean the model consumes one percent of unified memory. The PLE table is paged from NVMe, introducing another distinct working set.

Compare PLE memory on its own hardware

The PLE sidecar report uses one 96 GB RTX PRO 6000 Blackwell, 176 GB host RAM and local NVMe. Its main table uses the mixed NVFP4/FP8 checkpoint, 8K input, 512 requested output tokens and no prefix-cache hits. The reported launch window is 32,768 with 36 request slots. These conditions are not the Spark experiment's conditions.

Two selected memory rows make the distinction concrete. The BF16 PLE table is labelled 95.4 GB, with approximately 95 GB host RSS. The INT4 group-16 sidecar is labelled 32 GB, with 32.9 GB host RSS. Its format metadata specifies group-16 INT4 with FP16 scales.

Mapped RSS contains resident file pages that the operating system can reclaim. It is not a permanent anonymous allocation, and table size is not whole-model memory use. The source's GB labels are preserved; the site does not infer their exact byte convention or silently equate them with GiB. The full checkpoint revision, executed image digest and timing of the RSS sample remain unreported.

Separate first-request residence from warm throughput

An optional experiment drops page cache before each arm and compares INT4 mmap with an anonymous resident copy. The source reports 19.0 tokens/s on the first mmap request and 74.3 on the first resident request without MTP. These are first-request observations from a different cohort, whose exact request shape is not given; the main table's 8K/512 shape is not copied into it.

The resident sidecar adds 29.8 GiB of anonymous memory above roughly 8 GiB already held by the worker. This is another allocation, not the INT4 mmap RSS restated in different units. Anonymous memory also has different reclaim and swap behaviour; it should not be described as universally unreclaimable.

Within this optional cohort, both residence modes retain the same reported KV allocation for each MTP setting: 426,639 tokens without MTP and 134,192 with three drafts. The observation does not isolate a universal memory formula. The source also reports much closer warm rates and limited quality/sanity checks, so neither a cold-request advantage nor a smaller sidecar establishes general quality equivalence.

What should accompany a deployment decision?

Keep the exact main checkpoint, extra prefill weights, PLE format, drafter and executed runtime together. Name the memory measurement: file bytes, mapped RSS, anonymous allocation, KV capacity or minimum free memory. Record actual input/output length, concurrency, cache state and whether variation is within one boot or between boots.

For automated reading, the two complete source records are available through qwen-flash-next-memory; the adjacent article JSON retains the same conditions. Missing values remain null. Use the runtime choice guide and KV/speculation explanation to frame a follow-up test. Run AI Here has not executed either deployment or independently measured these models.