{
  "id": "A05",
  "path": "/guides/two-spark-deployment-conditions/",
  "title": "Two DGX Sparks: Read the Deployment Conditions",
  "summary": "Read three community launch profiles with artifact revisions, KV pools, speculation, context limits, actual concurrency and transport checks.",
  "answer": "Two Sparks establish a hardware boundary, not a reproduction. Preserve the artifact, runtime, allocated KV, speculation, transport and actual request shape; keep historical failures and missing execution identifiers explicit.",
  "published_on": "2026-10-18",
  "modified_on": "2026-10-18",
  "source_checked_on": "2026-10-18",
  "sources": [
    {
      "publisher": "WeZZard",
      "url": "https://github.com/WeZZard/dgx-spark-bench",
      "repository": "WeZZard/dgx-spark-bench",
      "platform": "github",
      "revision": "25e8dbf1b6113dd0b3f12d27a670fb284944d260",
      "document_path": "README.md",
      "pinned_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md",
      "report_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md",
      "document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469",
      "last_commit_at": "2026-09-06T00:35:29Z",
      "checked_on": "2026-10-18",
      "metadata_status": "checked",
      "record_id": "spark-pair-moe-served-rate"
    }
  ],
  "facts": [
    {
      "label": "Physical link versus actual transport",
      "value": "200 GbE wire did not establish container RDMA; verbs devices and active NCCL transport had to be checked",
      "classification": "source_reported_configuration_incident",
      "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md#L9",
      "source_document_sha256": "9e7904db6d48fc44691c9f170c03f402c77480dcbf54f66bc107ef57ef0ea94f",
      "source_document_id": "spark-docs-interconnect",
      "source_line": 9
    },
    {
      "label": "Earlier speculation campaign pool",
      "value": {
        "date": "2026-09-04",
        "no_speculation_tokens": 600000,
        "with_speculation_tokens": 145024,
        "not_the_September_5_pool": true
      },
      "classification": "source_reported_configuration_not_general_capacity",
      "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L44",
      "source_document_sha256": "46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f",
      "source_document_id": "spark-docs-speculation",
      "source_line": 44
    },
    {
      "label": "Qwen image version observation",
      "value": {
        "observed_on": "2026-09-04",
        "image_tag": "sglang-qwen38fn:sm121-ours",
        "reported_engine_version": "0.0.0.dev1+gd91c3682b",
        "campaign_immutable_digest": null
      },
      "classification": "source_reported_image_observation_not_immutable_campaign_version",
      "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/engines.md#L14",
      "source_document_sha256": "421632af313b5cfe7fca8b65ddabe0afb7b13e34229aca03e2af00c6a6f5452e",
      "source_document_id": "spark-docs-engines",
      "source_line": 14
    },
    {
      "label": "Qwen long-prompt FP8 KV failure",
      "value": "A single 30,301-token prompt also failed; the source identifies chunked prefill with a nonzero prefix, not concurrency itself, as the trigger in that image",
      "classification": "source_reported_failure_bound_to_build",
      "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L100",
      "source_document_sha256": "46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f",
      "source_document_id": "spark-docs-speculation",
      "source_line": 100
    }
  ],
  "reported_results": [],
  "conditions": {
    "hardware": "2 x DGX Spark GB10 sm_121",
    "node_count": 2,
    "memory_per_node_gib": 121,
    "physical_interconnect_gbps": 200,
    "campaign_date": "2026-09-05",
    "prefix_cache": false,
    "output_tokens_per_request": 512,
    "ignore_eos": true,
    "warmup": "every workload/concurrency shape, full generation; each node has its own JIT cache",
    "repetitions": 3,
    "summary_statistic": "median of three campaigns",
    "served_rate_denominator": "all output tokens / whole request wall clock, including TTFT",
    "runtime_execution_immutable_digest": null,
    "shared_memory_pool_claim": false,
    "cross_model_ranking_allowed": false,
    "current_1017_reproduction_claim": false
  },
  "generation": {
    "text_draft": "local Spark Qwen",
    "review": "Run AI Here with Codex",
    "image": "Codex image_gen fallback; local Qwen Image HTTP 502",
    "local_first": true
  },
  "image": {
    "url": "/assets/dual-spark-conditions-concept.webp",
    "alt": "Conceptual two-node deployment with separate local-memory spaces, a network link and configuration symbols.",
    "width": 1600,
    "height": 900,
    "sha256": "e07ed0d8c93e0a963c4010ec79e7d739f4511f40ab28a018ba3bbe7a122a2cf5",
    "caption": "AI-generated conceptual illustration; not measured topology or hardware photography."
  },
  "supplemental_sources": [
    {
      "document_id": "spark-REPORT",
      "parent_record_id": "spark-pair-moe-served-rate",
      "publisher": "WeZZard",
      "repository": "WeZZard/dgx-spark-bench",
      "revision": "25e8dbf1b6113dd0b3f12d27a670fb284944d260",
      "document_path": "REPORT.md",
      "pinned_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md",
      "document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e",
      "checked_on": "2026-10-18"
    },
    {
      "document_id": "spark-docs-interconnect",
      "parent_record_id": "spark-pair-moe-served-rate",
      "publisher": "WeZZard",
      "repository": "WeZZard/dgx-spark-bench",
      "revision": "25e8dbf1b6113dd0b3f12d27a670fb284944d260",
      "document_path": "docs/interconnect.md",
      "pinned_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md",
      "document_sha256": "9e7904db6d48fc44691c9f170c03f402c77480dcbf54f66bc107ef57ef0ea94f",
      "checked_on": "2026-10-18"
    },
    {
      "document_id": "spark-docs-speculation",
      "parent_record_id": "spark-pair-moe-served-rate",
      "publisher": "WeZZard",
      "repository": "WeZZard/dgx-spark-bench",
      "revision": "25e8dbf1b6113dd0b3f12d27a670fb284944d260",
      "document_path": "docs/speculation.md",
      "pinned_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md",
      "document_sha256": "46c64755b468e6e996e84fd1d19727160a9130bb3565731e7b5f26185c1dfe7f",
      "checked_on": "2026-10-18"
    },
    {
      "document_id": "spark-docs-engines",
      "parent_record_id": "spark-pair-moe-served-rate",
      "publisher": "WeZZard",
      "repository": "WeZZard/dgx-spark-bench",
      "revision": "25e8dbf1b6113dd0b3f12d27a670fb284944d260",
      "document_path": "docs/engines.md",
      "pinned_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/engines.md",
      "document_sha256": "421632af313b5cfe7fca8b65ddabe0afb7b13e34229aca03e2af00c6a6f5452e",
      "checked_on": "2026-10-18"
    }
  ],
  "deployment_profiles": [
    {
      "id": "ds-v3",
      "model_artifact": "deepseek-ai/DeepSeek-V4-Flash-0731",
      "checkpoint_revision": "7872f01b1d1fe23eabc4c98b48bffcef5a386062",
      "weight_precision": "MXFP4 experts (block 32; source weight audit)",
      "engine": "vLLM",
      "image_tag": "vllm-dsv4:ray",
      "executed_engine_version": null,
      "image_digest": null,
      "parallelism": "TP2; Ray across two nodes",
      "context_limit_tokens": 262144,
      "configured_max_requests": 12,
      "measured_concurrency": [
        1,
        6
      ],
      "kv_dtype": "fp8_ds_mla",
      "kv_pool_tokens": 517484,
      "speculation": "DSpark k=5, probabilistic",
      "prefix_cache": false,
      "configuration_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L928",
        "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e",
        "source_document_id": "spark-REPORT",
        "source_line": 928
      },
      "artifact_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L591",
        "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e",
        "source_document_id": "spark-REPORT",
        "source_line": 591
      },
      "pool_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L97",
        "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469",
        "source_document_id": "spark-pair-moe-served-rate",
        "source_line": 97
      }
    },
    {
      "id": "qwen-v3",
      "model_artifact": "RadixArk/Qwen3.8-Flash-Next-NVFP4",
      "checkpoint_revision": "7b719225242aacd3dbd3f9407468c2ee9a9d2594",
      "weight_precision": "NVFP4",
      "engine": "SGLang",
      "image_tag": "sglang-qwen38fn:sm121-ours",
      "executed_engine_version": null,
      "image_digest": null,
      "parallelism": "TP2; two nodes",
      "context_limit_tokens": 120000,
      "configured_max_requests": null,
      "measured_concurrency": [
        1,
        6
      ],
      "kv_dtype": "bfloat16",
      "kv_pool_tokens": 161728,
      "speculation": "NEXTN; one step, top-k 1, draft tokens 2",
      "prefix_cache": false,
      "configuration_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L562",
        "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e",
        "source_document_id": "spark-REPORT",
        "source_line": 562
      },
      "artifact_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L303",
        "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e",
        "source_document_id": "spark-REPORT",
        "source_line": 303
      },
      "pool_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L103",
        "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469",
        "source_document_id": "spark-pair-moe-served-rate",
        "source_line": 103
      }
    },
    {
      "id": "glm-v3",
      "model_artifact": "LibertAIDAI/GLM-5.3-Flash-NVFP4",
      "checkpoint_revision": "aa28e1f54130286c95fee10d0705c74ce8743734",
      "weight_precision": "NVFP4 experts (block 16; source weight audit)",
      "engine": "SGLang",
      "image_tag": "sglang-glm53:gb10-tilelang",
      "executed_engine_version": null,
      "image_digest": null,
      "parallelism": "TP2 / EP2; two nodes",
      "context_limit_tokens": 131072,
      "configured_max_requests": 8,
      "measured_concurrency": [
        1,
        8
      ],
      "kv_dtype": "bfloat16",
      "kv_pool_tokens": 162688,
      "speculation": "DFlash2; draft tokens 8",
      "prefix_cache": false,
      "mem_fraction_static": 0.9,
      "draft_artifact_revision": null,
      "configuration_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L270",
        "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e",
        "source_document_id": "spark-REPORT",
        "source_line": 270
      },
      "artifact_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L18",
        "source_document_sha256": "d08b9489656537f7b4a5d11cf4f07ed9f58c9b86ba6c7badd20ca52260a8224e",
        "source_document_id": "spark-REPORT",
        "source_line": 18
      },
      "pool_evidence": {
        "evidence_url": "https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L106",
        "source_document_sha256": "b815d32807147f9638b1abbeadc02af257c55bd1f7651a123e5a8090bebe7469",
        "source_document_id": "spark-pair-moe-served-rate",
        "source_line": 106
      }
    }
  ],
  "schema_version": "1.0.0",
  "record_type": "source_backed_explainer",
  "evidence_state": "source_reported",
  "site_reproduced": false,
  "metrics": null,
  "performance_ranking_allowed": false,
  "body_markdown": "# Two DGX Sparks: Read the Deployment Conditions\n\nA two-node model deployment is a combination of weights, runtime, attention cache, speculation and transport. A model name and a cable speed leave too much unspecified to reproduce it.\n\nThis note reads [WeZZard’s pinned two-Spark campaign](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md), remeasured on 5 September 2026, alongside its detailed configuration ledger. It explains what to record before trying a similar deployment. These are community observations; Run AI Here has not run the models.\n\n![Conceptual two-node deployment with separate local-memory spaces, a network link and configuration symbols.](/assets/dual-spark-conditions-concept.webp)\n\nAI-generated conceptual illustration. It is not a measured topology, a combined-memory claim or a photograph of the author’s cluster.\n\n## Start with the campaign boundary\n\nThe source reports two GB10 DGX Sparks, 121 GiB per node and a back-to-back 200 GbE link. Its corrected campaign has prefix caching off, 512 output tokens per request with `--ignore-eos`, full-length warmup for each workload and user shape, and median summaries from three campaigns. Served rate includes first-token waiting time in the request wall clock. [Original conditions](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L15).\n\nEach node retains its own memory and runtime state. Do not translate the two memory specifications into a promise of one freely allocatable memory pool. A launch-time context cap, an allocated KV pool and measured simultaneous users describe different constraints.\n\n## Preserve the actual artifact and launch profile\n\nThe artifact names and checkpoint revisions below come from the [detailed REPORT ledger](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md). Short revision prefixes are displayed here; the full revisions and separate source-document hashes are retained in JSON.\n\n| Campaign profile | Artifact / revision prefix | Weight format | Runtime image tag / parallelism |\n|---|---|---|---|\n| `ds-v3` | `deepseek-ai/DeepSeek-V4-Flash-0731` / `7872f01b` | MXFP4 experts | `vllm-dsv4:ray` / TP2, Ray |\n| `qwen-v3` | `RadixArk/Qwen3.8-Flash-Next-NVFP4` / `7b719225` | NVFP4 | `sglang-qwen38fn:sm121-ours` / TP2 |\n| `glm-v3` | `LibertAIDAI/GLM-5.3-Flash-NVFP4` / `aa28e1f5` | NVFP4 experts | `sglang-glm53:gb10-tilelang` / TP2, EP2 |\n\nThe local image tags identify the reported environments, but a tag is not an immutable image digest. They are not a general installation command or an endorsement of an unverified image with the same name.\n\n## Keep context, KV pool and concurrency separate\n\n| Profile | Allocated KV dtype / pool tokens | Speculation | Launch context cap | Measured users / launch maximum |\n|---|---|---|---:|---|\n| `ds-v3` | `fp8_ds_mla` / 517,484 | DSpark k=5, probabilistic | 262,144 | 1 and 6 / 12 |\n| `qwen-v3` | `bfloat16` / 161,728 | NEXTN, 2 draft tokens | 120,000 | 1 and 6 / unspecified |\n| `glm-v3` | `bfloat16` / 162,688 | DFlash2, 8 draft tokens | 131,072 | 1 and 8 / 8 |\n\nSources: [corrected pools](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/README.md#L90) and launch profiles [DeepSeek](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L928), [Qwen](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L562), [GLM](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/REPORT.md#L270). GLM’s static memory fraction is 0.90.\n\nThe configured maximum is not a successful measurement at that load. Nor does a pool value establish a usable per-request context by division: engine accounting, shared state and scheduling need inspection. Preserve the input-token count for each measured cell; a workload label is insufficient.\n\n## Verify the transport inside the container\n\nThe source’s [interconnect investigation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/interconnect.md) found that host networking worked while container verbs devices were absent and NCCL used sockets. A visible HCA listing alone did not prove active RDMA.\n\nFor a comparable setup, inspect the devices exposed to the running containers and the transport actually used during a request. Retain observations from both nodes. A successful host bandwidth test and a server-ready message answer different questions from a completed distributed inference request.\n\n## Separate historical tuning incidents from the final campaign\n\nThe earlier [4 September speculation investigation](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/speculation.md#L44) recorded a Qwen pool change from 600,000 to 145,024 tokens after enabling MTP. That earlier value is not the corrected campaign’s 161,728-token pool. It illustrates why the allocated pool must be re-read after a configuration change, rather than supplying a universal MTP memory formula.\n\nThat investigation also corrected an FP8-KV failure explanation: a single long prompt could trigger the chunked-prefill path. “Short prompt succeeded” therefore did not establish that the tested image supported long-context FP8 KV. The successful profile in the table uses bfloat16 KV. Keep failure records attached to their own build and settings.\n\n## What to capture before calling a reproduction successful\n\nRecord the exact artifact revision, conversion, image digest and executed runtime version. Read the allocated KV dtype and pool from startup evidence. Preserve speculation settings, context cap, prefix-cache state, actual input/output lengths, measured users and request failures. Warm the shapes you will measure on both nodes and retain clock and power observations under load.\n\nThe [engine inventory](https://github.com/WeZZard/dgx-spark-bench/blob/25e8dbf1b6113dd0b3f12d27a670fb284944d260/docs/engines.md#L14) separately records a Qwen-image version observation on 4 September. It does not supply an immutable image digest binding all three corrected campaign launches, so those execution fields remain unknown here. GLM’s exact draft-model revision also remains unknown in this extraction.\n\nThese profiles do not establish a cross-model ranking, current best settings or DeepSeek 1017 compatibility. Use the separate [1017 evidence note](/guides/deepseek-v41-flash-1017/) for checkpoint identity and the [Spark preflight checklist](/hardware/dgx-spark/deployment-checklist/) to prepare a run record.\n"
}