{ "id": "A15", "path": "/guides/qwen-flash-next-long-context-concurrency/", "title": "Qwen Long Context: Concurrency, Memory and Failure Evidence", "summary": "Read 128K and 490K Qwen batches, a four-GPU 128K matrix, sampled peak VRAM and a Mamba crash with original versions and measurement limits.", "answer": "Four submitted requests do not guarantee four running requests. The sources contain completed long-context batches, but their cache states, timing denominators and memory windows differ. Keep the four-GPU campaign peak separate from per-cell latency, and retain the original scheduler failure.", "published_on": "2026-10-18", "modified_on": "2026-10-18", "source_checked_on": "2026-10-18", "sources": [ { "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "platform": "github", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "RESULTS.md", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md", "document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "last_commit_at": "2026-10-17T16:41:23Z", "url": "https://github.com/jpezzulli/sglang-rtxpro6000", "report_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md", "checked_on": "2026-10-18", "metadata_status": "checked", "record_id": "qwen38-flash-next-rtxpro-long-concurrency" } ], "supplemental_sources": [ { "document_id": "penny-readme", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "README.md", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/README.md", "document_sha256": "7645e7297eb1326393e8e4106fb73334980dfe44540b45e51655bc1dadd295ad", "last_commit_at": "2026-10-17T16:41:23Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "penny-community-results", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "COMMUNITY-RESULTS.md", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md", "document_sha256": "2bbe9428f0701dc597dd0417193c12e72157dff2aca3ee6481b3104755a1301c", "last_commit_at": "2026-10-17T16:41:23Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "penny-release-3.0", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "release-3.0.json", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.json", "document_sha256": "1799eca8496bf143fc48bbdc83a4f6ccc34b8773c43083b47e29721b1ceb3dcf", "last_commit_at": "2026-10-17T16:41:23Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "penny-release-3.0.2", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "jpezzulli", "repository": "jpezzulli/sglang-rtxpro6000", "revision": "ca9be12f9be920e916d02954a41702a981b18823", "document_path": "release-3.0.2.json", "pinned_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.2.json", "document_sha256": "4843d621fa7f735a4c95d6bc79c9582d465cfaecef6b473689158d8c190920e2", "last_commit_at": "2026-10-17T16:41:23Z", "checked_on": "2026-10-18", "public_URL_is_immutable": true, "revision_kind": "Git document revision; not necessarily the executed runtime or weights" }, { "document_id": "qwen-cstech-4gpu", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "CS Technology Services", "repository": null, "revision": "snapshot-2026-10-18", "document_path": "qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell", "pinned_url": "https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell", "document_sha256": "b81878ba45f2f6b7a6de06b0bf930bd1dfa3cb497202ed3fcd368efe66065930", "last_commit_at": "2026-08-26", "checked_on": "2026-10-18", "public_URL_is_immutable": false, "revision_kind": "Mutable original report; retrieval snapshot date and SHA256, no Git revision" }, { "document_id": "qwen-mamba-incident", "parent_record_id": "qwen38-flash-next-rtxpro-long-concurrency", "publisher": "leehom0123", "repository": "sgl-project/sglang", "revision": "comment-5606493113", "document_path": "/issues/37619/comments/5606493113", "pinned_url": "https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113", "document_sha256": "d36cc21fa7b9577474ba20c9fe26becbb3c61bab9b9c4a5c421447b9d905c3f4", "last_commit_at": "2026-09-09T18:05:35Z", "checked_on": "2026-10-18", "public_URL_is_immutable": false, "revision_kind": "GitHub issue comment ID; mutable body with saved SHA256" } ], "facts": [], "reported_results": [ { "label": "128K C4 option off", "metric": "aggregate_output_rate", "value": 367.127, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K four-request three-key retrieval", "input_length_source_label": "128K", "concurrency": 4, "precision_option": "option off", "actual_output_tokens_group": 15725, "group_wall_seconds": 42.833, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": null, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 231, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L231", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "128K C4 online FP8", "metric": "aggregate_output_rate", "value": 422.037, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K four-request three-key retrieval", "input_length_source_label": "128K", "concurrency": 4, "precision_option": "online FP8", "actual_output_tokens_group": 15898, "group_wall_seconds": 37.67, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": 30.152, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 231, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L231", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "490K C4 option off", "metric": "aggregate_output_rate", "value": 236.678, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K four-request three-key retrieval", "input_length_source_label": "490K", "concurrency": 4, "precision_option": "option off", "actual_output_tokens_group": 15455, "group_wall_seconds": 65.3, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": null, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 232, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L232", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "490K C4 online FP8", "metric": "aggregate_output_rate", "value": 329.01, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K four-request three-key retrieval", "input_length_source_label": "490K", "concurrency": 4, "precision_option": "online FP8", "actual_output_tokens_group": 15127, "group_wall_seconds": 45.977, "requested_output_cap_per_request": 4096, "finish_reasons": [ "stop", "length" ], "all_four_overlap_seconds": 24.85, "metric_denominator": "all group output tokens / synchronized group makespan including TTFT", "all_responses_three_keys_exact": true, "all_four_full_distinct_prefixes_proven": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 232, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L232", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "128K online FP8 C1 decode_rate", "metric": "decode_rate", "value": 195.634, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "128K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 229, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L229", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "128K online FP8 C1 ttft", "metric": "ttft", "value": 12.906, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "128K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "128K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 229, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L229", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "490K online FP8 C1 decode_rate", "metric": "decode_rate", "value": 172.644, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "490K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 230, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L230", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "490K online FP8 C1 ttft", "metric": "ttft", "value": 63.296, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "workload": "490K C1 three-key retrieval", "concurrency": 1, "precision_option": "online FP8", "input_length_source_label": "490K", "metric_denominator": "client generation after first token, excluding TTFT" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 230, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L230", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "Post-graph free VRAM", "metric": "available_GPU_memory", "value": 7.52, "unit": "GiB", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "concurrency": null, "memory_kind": "post_graph_available_not_peak_used", "memory_cohort": "startup", "paired_long_text_batch_peak": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 243, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L243", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "Media-window minimum free VRAM", "metric": "available_GPU_memory", "value": 1897, "unit": "MiB", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "runtime_execution_revision": null, "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 824384, "mamba_slots": 24, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-09-11", "recipe_release": "v2.5.0", "input_tokens": null, "output_tokens": null, "actual_output_tokens": null, "cache_state": "post-warmup context tests; exact C4 cache sharing unreported", "concurrency": null, "memory_kind": "sampled_minimum_free_not_per_long_text_batch_peak", "memory_cohort": "ten selected GPU-media checks; ordinary/large JPEG, concurrent images and video history", "paired_long_text_batch_peak": false }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 263, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L263", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 cold 64K prefill_rate", "metric": "prefill_rate", "value": 17235.9, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 63921, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "native mean input rate" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 44, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L44", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 cold 64K ttft", "metric": "ttft", "value": 4.71, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 63921, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "client first-token latency" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 44, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L44", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 cold 490K prefill_rate", "metric": "prefill_rate", "value": 8839.75, "unit": "tokens/s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 489915, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "native mean input rate" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 45, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L45", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 cold 490K ttft", "metric": "ttft", "value": 60.37, "unit": "s", "statistic": "one reported observation; no dispersion", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": null, "top_p": null, "thinking": null, "repetitions": 1, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 489915, "output_tokens": null, "actual_output_tokens": null, "cache_state": "100% uncached prompt", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "metric_denominator": "client first-token latency" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 45, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L45", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 short C1 native mean", "metric": "native_decode_rate", "value": 218.63, "unit": "tokens/s", "statistic": "native mean; three regular repeats, no SD", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": 0.8, "top_p": 0.95, "thinking": "xhigh", "repetitions": 3, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 128, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "regular warmed screens", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 1, "top_k": 40, "min_p": 0.05, "metric_denominator": "elapsed-weighted complete active native log intervals; excludes idle/prefill crossing and partial tail; aggregate if C4" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 26, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L26", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" }, { "label": "3.0 short C4 native mean", "metric": "native_decode_rate", "value": 489.35, "unit": "tokens/s", "statistic": "native mean; three regular repeats, no SD", "uncertainty": null, "conditions": { "model": "Qwen3.8-Flash-Next", "model_artifact": "OrcaRouter ModelOpt NVFP4 checkpoint; exact revision unreported here", "model_revision": null, "runtime_execution_revision": "90b2b412e2bfd06f0ef3be1b41fbfbdee45bc14f", "executed_image_digest": null, "engine": "SGLang-derived Penny Royal", "system": "one RTX PRO 6000 Blackwell SM120, 96 GB; TP1", "node_count": 1, "parallelism": "TP1", "weight_precision": "ModelOpt NVFP4 experts; optional online FP8 projections", "kv_dtype": "FP8", "PLE": "host RAM", "context_limit_tokens": 524288, "KV_pool_tokens": 1067072, "mamba_slots": 36, "HiCache_NIXL_GiB": 32, "FR_Spec": true, "temperature": 0.8, "top_p": 0.95, "thinking": "xhigh", "repetitions": 3, "experiment_date": "2026-10-10", "recipe_release": "3.0", "input_tokens": 128, "output_tokens": 1024, "actual_output_tokens": 1024, "cache_state": "regular warmed screens", "accepted_public_core": "b48b70f3f80494d78a778fd4d26958ede343075f", "local_runtime_has_private_diagnostics": true, "media_preprocessing": "A4000 sidecar; not an inference TP rank", "concurrency": 4, "top_k": 40, "min_p": 0.05, "metric_denominator": "elapsed-weighted complete active native log intervals; excludes idle/prefill crossing and partial tail; aggregate if C4" }, "evidence_state": "source_reported", "site_reproduced": false, "source_document_id": "penny-results", "source_document_sha256": "51376fac57c0f5a277ada7d2a9359d365a7f861b424544c98bd49959b2b18e8d", "source_line": 27, "evidence_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md#L27", "source_record_id": "qwen38-flash-next-rtxpro-long-concurrency" } ], "conditions": { "independent_site_runs": 0, "controlled_cross_version_speedup": false, "per_cell_peak_RAM_known": false }, "generation": { "text_draft": "local Spark Qwen, then Codex factual review and rewrite; actual receipt in editorial-batch7 report", "review": "Run AI Here with Codex", "image": "deterministic matplotlib data figures and evidence maps", "local_first": true }, "image": { "url": "/assets/batch7-qwen-long-hero.webp", "alt": "Penny Royal original tests, a four-GPU matrix, and a dated scheduler incident.", "width": 1600, "height": 900, "sha256": "c9c9a6a8d6a514b46d6c76d9fdb80b723964002b6f08613991356ea4aacb83d9", "caption": "Original reports only. No new Run AI Here model execution." }, "case_reports": [ { "id": "qwen-cstech-128k-matrix", "classification": "source_reported_duration_matrix", "source_url": "https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell", "source_snapshot_sha256": "b81878ba45f2f6b7a6de06b0bf930bd1dfa3cb497202ed3fcd368efe66065930", "public_URL_is_immutable": false, "source_published_on": "2026-08-26", "model": "Qwen/Qwen3.8-Flash-Next-FP8", "model_revision": null, "engine": "vLLM 0.1.dev20073+g8e685d198", "executed_image_digest": null, "runtime_execution_revision": null, "harness": "llm-inference-bench v0.4.29", "system": "4 RTX PRO 6000 Blackwell Max-Q, 300 W/card, Threadripper, PCIe Gen4, no NVLink", "parallelism": "TP4", "context_limit_tokens": 262144, "KV_pool_tokens": 2662752, "configured_max_requests": 256, "draft_tokens": 3, "temperature": null, "top_p": null, "thinking": null, "input_length_source_label": "128k nominal matrix context; exact per-cell prompt token counts unreported", "duration_seconds_per_cell": 30, "output_tokens_cap": 8192, "ignore_eos": true, "effective_concurrency_validated_by_source": true, "cache_state": "duration-mode integrated scout warms shared prefix; matrix-start TTFT includes cache hit and queueing", "cells": [ { "concurrency": 1, "aggregate_output_tokens_per_second": 128.5, "TTFT_seconds": 1.41 }, { "concurrency": 8, "aggregate_output_tokens_per_second": 505.0, "TTFT_seconds": 2.02 }, { "concurrency": 16, "aggregate_output_tokens_per_second": 729.9, "TTFT_seconds": 3.42 } ], "omitted_cell": { "concurrency": 32, "status": "not_run_capacity", "aggregate_output_tokens_per_second": null, "observed_runtime_crash": false }, "memory": { "kind": "sampled_peak_summed_GPU_used_over_entire_matrix", "value": 374976, "unit": "MiB", "sample_count": 501, "campaign_seconds": 1206, "paired_per_cell_peak": null, "host_peak_RAM": null }, "cold_C1": { "actual_input_tokens": 128021, "TTFT_seconds": 24.25, "prefill_tokens_per_second": 5279 }, "per_user_table_is_derived_aggregate_divided_by_streams": true }, { "id": "qwen-mamba-long-prefill-crash", "classification": "source_reported_scheduler_assertion", "source_url": "https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113", "source_snapshot_sha256": "d36cc21fa7b9577474ba20c9fe26becbb3c61bab9b9c4a5c421447b9d905c3f4", "source_updated_at": "2026-09-09T18:05:35Z", "public_URL_is_immutable": false, "model": "Qwen3.8-Flash-Next-NVFP4", "model_revision": null, "engine": "SGLang 0.0.0.dev1+g4ccff141d", "runtime_execution_revision": null, "executed_image_digest": null, "weight_precision": "modelopt_mixed", "kv_dtype": "fp8_e4m3", "context_length_source_label": "262k", "actual_prompt_tokens": null, "prompt_length_source_label": "approximately 140k", "chunked_prefill_size": 4096, "mamba_track_interval": 64, "mamba_slots": 51, "configured_max_requests": 12, "observed_running_requests": 2, "observed_queued_requests_reporter": 1, "queued_requests_at_cited_log_line": 0, "mamba_usage_before": 0.18, "mamba_usage_at_failure": 1.0, "full_token_usage_at_failure": 0.46, "HiCache": true, "hicache_ratio": 2.0, "write_policy": "write_through", "speculation": false, "error": "AssertionError: Can not alloc mamba cache", "outcome": "scheduler child exited; container auto-restarted, service outage during reload", "fix_reference": "https://github.com/sgl-project/sglang/pull/37619", "fix_PR_state": "open", "fix_PR_merged": false, "fix_PR_head_revision": "cd530ac217ac0a67b913f7083679c0890670d228", "author_GPU_reproducer_rerun": false, "universal_fix_verified": false, "peak_RAM": null } ], "memory_observations": [ { "kind": "sampled_peak_summed_GPU_used_over_entire_matrix", "value": 374976, "unit": "MiB", "sample_count": 501, "campaign_seconds": 1206, "paired_per_cell_peak": null, "host_peak_RAM": null }, { "kind": "v2.5.0_GPU_media_window_minimum_free", "value": 1897, "unit": "MiB", "long_text_C4_peak": null, "source_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md" } ], "long_context_cases": [ { "classification": "source_reported_four_submitted_stress", "recipe_release": "v2.5.0", "contributor": "BESTIA / Prometeus field report; separate contributor from runtime author", "source_url": "https://www.reddit.com/r/BlackwellPerformance/comments/1weaxdz/comment/p9i76fb/", "source_summary_url": "https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md", "context_limit_tokens": 524288, "KV_pool_tokens": 1114304, "input_tokens": null, "input_length_source_label": "approximately 240K", "submitted_concurrency": 4, "simultaneous_running_concurrency": null, "rounds": 10, "total_requests": 40, "output_tokens_reported": 8192, "round_seconds_range": [ 93, 107 ], "HTTP_200": 40, "OOMs_reported": 0, "retractions_reported": 0, "aborts_reported": 0, "peak_pool_occupancy_percent": 57, "peak_memory_bytes": null, "queuing_observed": true, "retrieval_correctness": null } ], "schema_version": "1.0.0", "record_type": "source_backed_explainer", "evidence_state": "source_reported", "site_reproduced": false, "metrics": null, "performance_ranking_allowed": false, "body_markdown": "# Qwen Long Context: Concurrency, Memory and Failure Evidence\n\n![Penny Royal original tests, a four-GPU matrix, and a dated scheduler incident.](/assets/batch7-qwen-long-hero.webp)\n\n*Original reports only. No new Run AI Here model execution.*\n\nLong context changes the question a deployment report needs to answer. A server can accept a large window, submit several requests and report a high token rate without keeping every long document resident at once. Useful evidence records the actual request shape, overlapping execution, queue, prefix cache, completion length and memory sampling window together. This note reads original Qwen3.8-Flash-Next reports with those distinctions intact. Run AI Here has not repeated their model tests.\n\n## One-card long batches: what actually overlapped\n\nThe [Penny Royal original results](https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/RESULTS.md) contain September 11 v2.5.0 observations on one RTX PRO 6000 Blackwell, TP1. The relevant configuration uses RadixArk ModelOpt NVFP4, RAM PLE, FP8 KV, FR-Spec/native NEXTN, a 524,288-token request limit, an 824,384-token pool, 24 Mamba slots and 32 GiB HiCache/NIXL. “Online FP8” changes selected projections; it does not mean every model component changes precision.\n\n| Source-labelled workload | Option off, aggregate tokens/s | Online FP8, aggregate tokens/s | Online-arm output tokens / group seconds |\n|---|---:|---:|---|\n| 128K, four requests | 367.127 | 422.037 | 15,898 / 37.670 |\n| 490K, four requests | 236.678 | 329.010 | 15,127 / 45.977 |\n\nEach value is one batch observation. Aggregate means all output tokens divided by synchronized group makespan, including first-token waiting. Requests end at natural stop or a 4,096-token cap, so the arms produce different totals. These are contextual batch rates, with no repeated-run dispersion. The online arm records four-way overlap for 30.152 seconds at 128K and 24.850 seconds at 490K. The source does not establish four fully distinct, full-length resident prefix copies. Exact per-request input counts and cache sharing are missing in this summary.\n\n![Penny Royal v2.5.0 aggregate rates at 128K and 490K, with distinct off and online-FP8 arms.](/assets/batch7-qwen-long-rates.webp)\n\n*One batch per cell. These aggregate rates include synchronized group wall time and different completion totals.*\n\nThe matching online-FP8 C1 observations use exactly 1,024 output tokens: 195.634 tokens/s after the first token at 128K, and 172.644 at 490K. TTFT is 12.906 and 63.296 seconds respectively. Those long observations are single samples. The source explicitly says cold TTFT did not improve against its option-off samples. A higher generation rate therefore need not shorten the complete request by the same proportion. Several release-state changes also accompany the precision option, preventing an isolated FP8 causal claim.\n\n## The newer release is a separate experiment\n\nOctober 10 [Penny Royal 3.0 release metadata](https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/release-3.0.json) identifies another checkpoint, adaptive MTP and a 1,067,072-token pool. Its cold inputs are exactly 63,921 and 489,915 tokens; reported native mean prefill is 17,235.90 and 8,839.75 tokens/s, with TTFT 4.71 and 60.37 seconds. Three-key recall passes in the longer input. Completion length for those cold cases is not supplied here.\n\nThe 3.0 regular screens instead use 128 input and 1,024 output tokens, three repeats, temperature 0.8, top-p 0.95, top-k 40, min-p 0.05 and xhigh reasoning. Native means are 218.63 tokens/s at C1 and 489.35 aggregate at C4. They weight complete active reporting intervals, excluding idle/prefill-crossing intervals and partial tails. They do not measure complete agent-task wall time or replace the older long C4 cells. The accepted public core and the local checkout with private diagnostics have separate full identities. A document commit, public release image and actual native execution are separate provenance fields.\n\n## A four-card matrix provides a real campaign peak\n\nThe [CS Technology Services original report](https://www.cstech.dev/articles/qwen3-8-flash-next-fp8-on-4x-rtx-pro-6000-blackwell) uses four RTX PRO 6000 Max-Q cards, TP4, PCIe Gen4 without NVLink, vLLM `0.1.dev20073+g8e685d198` and harness v0.4.29. At nominal 128K, C1/C8/C16 aggregate rates are 128.5/505.0/729.9 tokens/s. Each cell lasts 30 seconds with `ignore_eos=true` and an 8,192-token output cap. Effective concurrency was checked. C32 is omitted for capacity, not an observed crash.\n\nIts 374,976 MiB sampled peak is summed used VRAM across the **whole matrix**, over 501 samples in 1,206 seconds. Per-cell peak and host peak RAM are absent. Matrix TTFT is warm-prefix/queue time; cold C1 takes 24.25 seconds at 128,021 actual input tokens. Per-user rates in that report are derived by division. Image digest, weight revision and full launch command were not retained. This is a different deployment, without a controlled comparison to Penny Royal.\n\n![Do not attach a peak from one campaign to an unrelated latency cell.](/assets/batch7-qwen-memory-scope.webp)\n\n*GiB and MiB are retained. Free, used, KV occupancy and host RAM are different quantities.*\n\nPenny Royal's 7.52 GiB free after graph capture is startup headroom. Its 1,897 MiB sampled minimum free belongs to ten GPU-media checks. Neither is the long C4 peak used memory. Keep free versus used, one versus four cards, startup versus workload and GPU versus host measurements separate. Pool occupancy is another counter again; percentages cannot be converted into whole-process bytes without an established mapping.\n\n## Sustained submissions can still queue\n\nThe [credited community follow-up](https://github.com/jpezzulli/sglang-rtxpro6000/blob/ca9be12f9be920e916d02954a41702a981b18823/COMMUNITY-RESULTS.md) describes a separate operator's v2.5.0 configuration with a 1,114,304-token pool. Ten rounds submit four approximately 240K requests with 8,192 reported output tokens each. The author reports 93–107 seconds per round, 40/40 HTTP 200 and no OOMs, retractions or aborts. Queuing is observed and pool occupancy peaks at 57 percent. This supports sustained operation for that workload; it does not establish four running throughout, near-full-pool stress, exact retrieval correctness, peak RAM or a changed runtime default. The approximate input label remains approximate.\n\n## A state-cache failure below the request ceiling\n\nAn [original SGLang incident comment](https://github.com/sgl-project/sglang/pull/37619#issuecomment-5606493113) records approximately 140K cold input, a 262K-class window, FP8 E4M3 KV, no speculation, 4,096-token prefill chunks, 51 Mamba slots and a 12-request ceiling. HiCache uses write-through. Only two requests are running when Mamba usage jumps from 0.18 to 1.00 while full-token usage is 0.46. The scheduler raises `Can not alloc mamba cache`; its child exits and the container reload interrupts service. The reporter mentions one queued request, while the cited failure log shows zero at that instant. Both observations are retained.\n\n![Original SGLang incident: approximately 140K cold prompt, HiCache write-through.](/assets/batch7-qwen-mamba-failure.webp)\n\n*Source-reported outage. The proposed fallback and CPU tests do not prove every GPU workload fixed.*\n\nThe related [PR37619](https://github.com/sgl-project/sglang/pull/37619) remains open and unmerged at this check. Its author reports two CPU tests and no rerun of the GPU reproducer. The proposed fallback skips an optional unfinished checkpoint when slots cannot be allocated. That scope does not establish a universal fix or a new speed result. Request slots, KV capacity and held Mamba states are separate constraints.\n\nFor deployment selection, first match hardware, exact checkpoint, runtime, cache policy and requested output. Then ask whether the source observes simultaneous execution and measures memory over the same interval. Preserve a null where that evidence is missing. This produces a more useful capacity decision than combining the fastest token rate, largest configured window and smallest memory number from different runs.\n" }