THE SOURCE IN CONTEXT
The paired experiment runs on eight 32 GB RTX 5090 D GPUs, two Xeon Gold 6530 sockets and 503 GiB host memory over PCIe. Both arms already use custom vLLM patches and CUDA graphs. The placed arm moves the same offload budget onto rarely selected expert pages and enables exact staging; it is not a stock-versus-optimized upstream comparison.
What is being compared?
The paired experiment runs on eight 32 GB RTX 5090 D GPUs, two Xeon Gold 6530 sockets and 503 GiB host memory over PCIe. Both arms already use custom vLLM patches and CUDA graphs. The placed arm moves the same offload budget onto rarely selected expert pages and enables exact staging; it is not a stock-versus-optimized upstream comparison.
Source: devin-lai · pinned document
What improves, and what regresses?
Across three trials, interactive C1 output rises from 75.37 to 97.23 tokens/s, while C8 rises from 166.55 to 247.77. The same placement reduces the random-token C1 result from 67.46 to 63.00. The negative result is central: calibration-dependent expert placement is useful when traffic resembles its routing profile and can be harmful when it does not.
Source: devin-lai · pinned document
Why is the largest number not interactive speed?
The 368.55-token/s held-out-text result is aggregate output at 128 concurrent requests, with 1,000-token inputs and 256-token outputs in a single trial. The corresponding latency and queueing behavior differ from C1. The synthetic 337.45-token/s result at 256 users is another workload and another memory budget. Do not collapse them into one headline for a single person.
Source: devin-lai · pinned document
How strong is the evidence?
The paired interactive comparison has three trials, while placement-only screens and wide-batch campaigns have one. The corpus and prompt shape differ across sections. The report measures the launch checkpoint, not the 1017 refresh, and the collection does not fill in missing prompt lengths or weight hashes from a different source.
Source: devin-lai · pinned document
Reported figures and their conditions
Selected facts reported by the original author. Each row has its own configuration and measurement boundary. Run AI Here has not reproduced these tests.
| Configuration / workload | Metric | Reported value | Evidence |
|---|---|---|---|
| Interactive C1, patched baseline | output throughput | 75.37 tokens/sreported value | Original row ↗ |
| Interactive C1, expert placement + staging | output throughput | 97.23 tokens/sreported value | Original row ↗ |
| Interactive C8, expert placement + staging | output throughput | 247.77 tokens/sreported value | Original row ↗ |
| Random-token C1, baseline | output throughput | 67.46 tokens/sreported value | Original row ↗ |
| Random-token C1, placed | output throughput | 63.0 tokens/sreported value | Original row ↗ |
| Held-out text C128, 128-sequence preset | aggregate output | 368.55 tokens/sreported value | Original row ↗ |
Interactive C1, patched baseline · output throughput · conditions
| system | 8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only |
|---|---|
| node_count | 1 |
| parallelism | TP8 + expert parallelism |
| kv_dtype | Not specified |
| context_limit_tokens | Not specified |
| experiment_date | 2026-09-16 |
| model | DeepSeek V4.1 Flash launch checkpoint, not 1017 |
| quantization | MXFP4 experts / FP8 dense / BF16 expert activations |
| runtime | already patched graph-enabled vLLM |
| calibration | shipped mixed real-text profile |
| concurrency | 1 |
| repetitions | 3 |
| placement | layer offload |
Interactive C1, expert placement + staging · output throughput · conditions
| system | 8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only |
|---|---|
| node_count | 1 |
| parallelism | TP8 + expert parallelism |
| kv_dtype | Not specified |
| context_limit_tokens | Not specified |
| experiment_date | 2026-09-16 |
| model | DeepSeek V4.1 Flash launch checkpoint, not 1017 |
| quantization | MXFP4 experts / FP8 dense / BF16 expert activations |
| runtime | already patched graph-enabled vLLM |
| calibration | shipped mixed real-text profile |
| concurrency | 1 |
| repetitions | 3 |
| placement | cold expert pages + exact staging |
Interactive C8, expert placement + staging · output throughput · conditions
| system | 8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only |
|---|---|
| node_count | 1 |
| parallelism | TP8 + expert parallelism |
| kv_dtype | Not specified |
| context_limit_tokens | Not specified |
| experiment_date | 2026-09-16 |
| model | DeepSeek V4.1 Flash launch checkpoint, not 1017 |
| quantization | MXFP4 experts / FP8 dense / BF16 expert activations |
| runtime | already patched graph-enabled vLLM |
| calibration | shipped mixed real-text profile |
| concurrency | 8 |
| repetitions | 3 |
| placement | cold expert pages + exact staging |
Random-token C1, baseline · output throughput · conditions
| system | 8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only |
|---|---|
| node_count | 1 |
| parallelism | TP8 + expert parallelism |
| kv_dtype | Not specified |
| context_limit_tokens | Not specified |
| experiment_date | 2026-09-16 |
| model | DeepSeek V4.1 Flash launch checkpoint, not 1017 |
| quantization | MXFP4 experts / FP8 dense / BF16 expert activations |
| runtime | already patched graph-enabled vLLM |
| calibration | shipped mixed real-text profile |
| concurrency | 1 |
| repetitions | 3 |
| workload | uniform random token IDs |
Random-token C1, placed · output throughput · conditions
| system | 8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only |
|---|---|
| node_count | 1 |
| parallelism | TP8 + expert parallelism |
| kv_dtype | Not specified |
| context_limit_tokens | Not specified |
| experiment_date | 2026-09-16 |
| model | DeepSeek V4.1 Flash launch checkpoint, not 1017 |
| quantization | MXFP4 experts / FP8 dense / BF16 expert activations |
| runtime | already patched graph-enabled vLLM |
| calibration | shipped mixed real-text profile |
| concurrency | 1 |
| repetitions | 3 |
| workload | uniform random token IDs |
| placement | cold expert pages + staging |
Held-out text C128, 128-sequence preset · aggregate output · conditions
| system | 8 x NVIDIA RTX 5090 D (32 GB each); 503 GiB host RAM; PCIe only |
|---|---|
| node_count | 1 |
| parallelism | TP8 + expert parallelism |
| kv_dtype | Not specified |
| context_limit_tokens | Not specified |
| experiment_date | 2026-09-16 |
| model | DeepSeek V4.1 Flash launch checkpoint, not 1017 |
| quantization | MXFP4 experts / FP8 dense / BF16 expert activations |
| runtime | already patched graph-enabled vLLM |
| calibration | shipped mixed real-text profile |
| concurrency | 128 |
| input_tokens | 1000 |
| output_tokens | 256 |
| repetitions | 1 |
| max_num_seqs | 128 |
AI-assisted source reading and original editorial synthesis; no independent model execution.