Qwen on RTX 5090: What the 128K Recipe Tested

A Qwen3.8-Flash-Next server can advertise a 131,072-token window while its longest published request contains 104,805 tokens. That is useful long-context evidence with a precise boundary. The original RTX 5090 WSL2 recipe also publishes memory telemetry from a shorter 32K gate, rejects MTP promotion and leaves ngram speculation unrun. Preserving these separate outcomes makes the report more useful than a single “fits on 32 GB” headline.

Start with the CPU offload and the exact artifact

The original kutaelee repository describes one RTX 5090 reporting 32,607 MiB, a 16-core/32-thread desktop CPU, a 128 GiB-class host and Ubuntu under WSL2 with a 96 GiB VM ceiling. The selected model is jamesrogers/Qwen3.8-Flash-Next-MTP-MXFP4-GGUF at 4a768bf4458a83e1c80667b4d818ec721080546e. Its file is 126,846,153,856 bytes, with a publisher-provided SHA-256 retained in the JSON conditions. Run AI Here inspected the public documents; it did not download or execute this weight file.

The tested engine is ik_llama.cpp merge commit d854dfd36a7262fc46cb7542716b04a9f6d4b47d, including the pinned MTP patch. A repository documentation revision and a runtime execution revision identify different things, so both are retained. Later changes at a repository's moving main branch are not automatically part of this test.

Most importantly, the baseline configuration keeps 38 expert layers on the CPU, defers experts, enables mmap and uses Q8_0 KV. It allows one request and one server slot. This is partial CPU expert offload, not a claim that the whole model resides in the GPU's memory. A Q8 n-gram embedding table is part of the model artifact; ngram speculative decoding is a different feature. Keeping speculation off does not mean the model's embedding table is absent or unused.

Read the actual request lengths

The three reported baseline gates use different prompts. Throughput comes from server timing fields, according to the methodology. It is not a client-side elapsed-time estimate.

The 61-token request confirms that a server configured for 128K completed a short smoke test. It does not test a 128K input. The longer gate is more informative about retrieval at depth, while still falling short of a completely filled 131,072-token input. Actual completion lengths, needle depth, per-gate sampling and repeat counts are not supplied in the selected table. Do not silently insert the runtime's current defaults into those empty fields.

These three rows are not a controlled context-scaling experiment. Request content, startup state and task differ, and no uncertainty distribution accompanies these gate values. It would be misleading to fit a latency law or predict a maximum memory requirement from the table. A request passing also does not establish that several such requests can run together: every published row uses C1.

Keep the 32K memory telemetry in its own cohort

The source reports 25.15 seconds of startup and the following observations during the completed 32K monitoring gate:

There is no paired peak-RAM, swap or peak-GPU series for the 104,805-token needle in the selected documents. Those longer-test fields remain null. Do not relabel the shorter gate's peak as a “128K memory requirement,” and do not treat absent long-context swap data as zero swap.

The reported MemAvailable scope deserves particular care. The source names a 96 GiB WSL ceiling but does not establish that its available-memory sample and that ceiling describe the same boundary. The site therefore preserves the original number without calculating a supposed spare VM budget from it. Host RSS, mapped file pages, GPU allocations and a VM limit should not simply be added together. Units remain KiB or MiB rather than silently becoming decimal GB.

mmap success addresses a specific allocation problem

The source explains that the default CUDA-host override attempted a 98.61 GiB anonymous pinned allocation during validation. The accepted baseline sets GGML_CUDA_NO_PINNED=1 and retains mmap. That observation explains the author's configuration choice; it does not guarantee identical memory behavior on another operating system, cache state or runtime build.

The limitations document identifies cold expert-page faults and CPU expert work as important constraints for this workstation. Those are source interpretations with a clear configuration attached. They do not establish that every 5090 deployment is CPU limited, nor that every alternative conversion has the same behavior. The Spark memory article describes different hardware and PLE backing experiments; those measurements cannot substitute for missing WSL telemetry.

Retain the failed and unrun speculation paths

The repository reports that MTP n=4 preserved eight correctness checks, but two completed comparable prose samples reached only 2.52 and 3.14 tokens/s. Canonical MTP n=1 also preserved tested correctness and was slower across its completed short gates. Its peak GPU memory reached 31,775 MiB. The memory point is a separate MTP short-gate observation, not the baseline's long-context peak.

The author consequently rejected MTP promotion for this exact MXFP4 conversion and 38-layer CPU-expert split. Ngram speculative decoding was not run because the protocol required a beneficial MTP lane first. “Not run” provides no measured ngram speed, memory improvement or failure rate. The n-gram embedding table remains a separate model component throughout this decision.

This is a useful operational result: correctness, speed and headroom can disagree. Passing a small correctness gate does not compel deployment of a slower speculative path. It also does not prove general quality equivalence. Read the broader KV/speculation explanation before transferring a favorable MTP result from a different checkpoint or engine.

What a useful follow-up report would add

For this configuration, the most valuable missing evidence would pair actual prompt and output counts with telemetry from the same long request. It would identify whether MemAvailable was sampled inside the VM or on the host, retain the cache state, and repeat an unchanged workload before summarizing variation. Any concurrency claim would need completed multi-request runs with their scheduling and memory outcomes. These are editorial suggestions for future evidence, not measurements performed by this site.

Has the full 128K input been validated here?

The selected source validates an exact needle in 104,805 input tokens under a 131,072-token configured ceiling. It does not publish a fully filled 131,072-token input gate.

Does the 104K pass establish repository-analysis quality?

No. Exact retrieval of one embedded answer is narrower than multi-document reasoning, trustworthy code changes or long output generation.

Is ngram missing because the embedding table was disabled?

No. The skipped experiment is speculative decoding. The Q8 n-gram embedding table belongs to the model artifact and is a separate mechanism. The For AI reference keeps these states distinct in the exported conditions.