THE SOURCE IN CONTEXT
This experiment changes the n-gram PLE table backing, rather than quantizing every model component. The source compares a BF16 table held in RAM with file-mapped sidecars on one 96 GB RTX PRO 6000 and a 176 GB host. Main-table throughput uses 8K input, 512 requested output tokens and no prefix-cache hits. Full checkpoint revision and executed image digest are not supplied.
Which allocation is reduced?
This experiment changes the n-gram PLE table backing, rather than quantizing every model component. The source compares a BF16 table held in RAM with file-mapped sidecars on one 96 GB RTX PRO 6000 and a 176 GB host. Main-table throughput uses 8K input, 512 requested output tokens and no prefix-cache hits. Full checkpoint revision and executed image digest are not supplied.
Source: primitive-ai · pinned document primitive-ai · pinned document
Why is RSS different from anonymous memory?
The selected BF16 and INT4 rows retain artifact size separately from host RSS. File-mapped RSS includes reclaimable page-cache pages; it is not a permanent anonymous allocation. The optional INT4 resident copy instead adds a reported 29.8 GiB of anonymous sidecar memory, above the worker allocation. Source-labelled GB and GiB are not silently equated.
Source: primitive-ai · pinned document
What remains outside the evidence?
The optional cold-residence cohort drops page cache before each arm and has a different, incompletely reported request shape. Its first-request speed is not a warm throughput average. MTP changes the allocated KV token pool; mmap versus resident does not change that pool within each reported MTP setting. Neither the table-size reduction nor suite-level sanity proves general quality equivalence.
Source: primitive-ai · pinned document
Does the native window fit every launch configuration?
The source reports a native-window startup failure with MTP when 7.57 GiB of KV was required but 5.95 GiB remained. Its selected main table used a shorter 32768-token launch window, while a separate field report used 135168. These are configuration-specific observations, not a universal maximum or evidence that every format fits the native window.
Source: primitive-ai · pinned document
Reported figures and their conditions
Selected facts reported by the original author. Each row has its own configuration and measurement boundary. Run AI Here has not reproduced these tests.
| Configuration / workload | Metric | Reported value | Evidence |
|---|---|---|---|
| BF16 in RAM table size | artifact size | 95.4 GBsource-reported PLE table size, not whole model or peak RAM | Original row ↗ |
| BF16 in RAM host RSS | host rss | 95 GBapproximate source host RSS | Original row ↗ |
| INT4 group-16 mmap table size | artifact size | 32 GBsource-reported PLE table size, not whole model or peak RAM | Original row ↗ |
| INT4 group-16 mmap host RSS | host rss | 32.9 GBsource host RSS including reclaimable file pages | Original row ↗ |
| mmap cold first request | decode rate | 19.0 tokens/sfirst-request observation; warm asymptote is a separate quantity | Original row ↗ |
| resident cold first request | decode rate | 74.3 tokens/sfirst-request observation; warm asymptote is a separate quantity | Original row ↗ |
| INT4 resident sidecar anonymous allocation | anonymous memory | 29.8 GiBsidecar allocation only, on top of roughly 8 GiB held by worker | Original row ↗ |
| optional residence cohort KV pool, MTP 0 | kv pool tokens | 426639 tokensreported identical mmap/resident allocation for this MTP setting | Original row ↗ |
| optional residence cohort KV pool, MTP 3 | kv pool tokens | 134192 tokensreported identical mmap/resident allocation for this MTP setting | Original row ↗ |
BF16 in RAM table size · artifact size · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | 32768 |
| configured_max_requests | 36 |
| input_tokens | 8192 |
| output_tokens | 512 |
| actual_output_tokens | Not specified |
| concurrency | 1 |
| cache_state | prefix-cache-free; mmap page-cache residency not independently logged |
| thinking | True |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | Not specified |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | two seeds, a/b; not independent boots |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | BF16 in RAM |
| memory_kind | PLE_artifact_size |
| memory_backing | in RAM |
BF16 in RAM host RSS · host rss · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | 32768 |
| configured_max_requests | 36 |
| input_tokens | 8192 |
| output_tokens | 512 |
| actual_output_tokens | Not specified |
| concurrency | 1 |
| cache_state | prefix-cache-free; mmap page-cache residency not independently logged |
| thinking | True |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | Not specified |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | two seeds, a/b; not independent boots |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | BF16 in RAM |
| memory_kind | host_RSS_not_necessarily_anonymous |
| memory_backing | in RAM |
{'approximate': True}
INT4 group-16 mmap table size · artifact size · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | 32768 |
| configured_max_requests | 36 |
| input_tokens | 8192 |
| output_tokens | 512 |
| actual_output_tokens | Not specified |
| concurrency | 1 |
| cache_state | prefix-cache-free; mmap page-cache residency not independently logged |
| thinking | True |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | Not specified |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | two seeds, a/b; not independent boots |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | INT4 group-16 mmap |
| memory_kind | PLE_artifact_size |
| memory_backing | reclaimable mapped file pages |
INT4 group-16 mmap host RSS · host rss · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | 32768 |
| configured_max_requests | 36 |
| input_tokens | 8192 |
| output_tokens | 512 |
| actual_output_tokens | Not specified |
| concurrency | 1 |
| cache_state | prefix-cache-free; mmap page-cache residency not independently logged |
| thinking | True |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | Not specified |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | two seeds, a/b; not independent boots |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | INT4 group-16 mmap |
| memory_kind | host_RSS_not_necessarily_anonymous |
| memory_backing | reclaimable mapped file pages |
mmap cold first request · decode rate · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | Not specified |
| configured_max_requests | Not specified |
| input_tokens | Not specified |
| output_tokens | Not specified |
| actual_output_tokens | Not specified |
| concurrency | 1 |
| cache_state | page cache dropped before each arm |
| thinking | Not specified |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | 0 |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | Not specified |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | INT4 group-16 |
| workload | first request, exact request shape unreported; optional resident-vs-mmap cohort, not main 8K/512 table |
| memory_backing | NVMe mmap |
resident cold first request · decode rate · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | Not specified |
| configured_max_requests | Not specified |
| input_tokens | Not specified |
| output_tokens | Not specified |
| actual_output_tokens | Not specified |
| concurrency | 1 |
| cache_state | page cache dropped before each arm |
| thinking | Not specified |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | 0 |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | Not specified |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | INT4 group-16 |
| workload | first request, exact request shape unreported; optional resident-vs-mmap cohort, not main 8K/512 table |
| memory_backing | anonymous resident copy |
INT4 resident sidecar anonymous allocation · anonymous memory · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | Not specified |
| configured_max_requests | 36 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| actual_output_tokens | Not specified |
| concurrency | Not specified |
| cache_state | prefix-cache-free; mmap page-cache residency not independently logged |
| thinking | Not specified |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | Not specified |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | two seeds, a/b; not independent boots |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | INT4 group-16 |
| memory_kind | anonymous_resident_sidecar_not_total_RSS |
optional residence cohort KV pool, MTP 0 · kv pool tokens · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | Not specified |
| configured_max_requests | 36 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| actual_output_tokens | Not specified |
| concurrency | Not specified |
| cache_state | prefix-cache-free; mmap page-cache residency not independently logged |
| thinking | Not specified |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | 0 |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | two seeds, a/b; not independent boots |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | INT4 group-16 |
| memory_kind | allocated_KV_token_pool_not_measured_RAM |
optional residence cohort KV pool, MTP 3 · kv pool tokens · conditions
| model | Qwen3.8-Flash-Next |
|---|---|
| model_artifact | primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 |
| model_revision | Not specified |
| PLE_artifact | primitive-ai/Qwen3.8-Flash-Next-PLE-quant |
| PLE_artifact_revision | a0fa93f2b9ed5fcab0cf01b60e88ff1ce13a4763 |
| system | one RTX PRO 6000 Blackwell 96 GB, 176 GB host RAM, local NVMe |
| node_count | 1 |
| engine | vLLM with exact-image PLE overlay |
| engine_version | Not specified |
| runtime_execution_revision | Not specified |
| image_tag | vllm/vllm-openai:qwen38-flash-next |
| executed_image_digest | Not specified |
| experiment_date | Not specified |
| source_update_boundary | 2026-09-06 corrections present; individual table trial dates not stated |
| model_weight_precision | mixed NVFP4/FP8 checkpoint; PLE format varies |
| context_limit_tokens | Not specified |
| configured_max_requests | 36 |
| input_tokens | Not specified |
| output_tokens | Not specified |
| actual_output_tokens | Not specified |
| concurrency | Not specified |
| cache_state | prefix-cache-free; mmap page-cache residency not independently logged |
| thinking | Not specified |
| temperature | Not specified |
| top_p | Not specified |
| draft_tokens | 3 |
| MTP_main_table_execution | Not specified |
| kv_dtype | Not specified |
| throughput_repetitions | two seeds, a/b; not independent boots |
| memory_sampling_instant | Not specified |
| reported_size_unit | GB as labelled by source, byte convention not independently specified |
| PLE_format | INT4 group-16 |
| memory_kind | allocated_KV_token_pool_not_measured_RAM |
Original source-grounded editorial synthesis with local Spark Qwen assistance and Codex review; no site model benchmark.