Full GLM-5.3 on Four Sparks: Cache Layout, Speed and Fidelity

Source-reported HTTP rates include prompt time; exact concurrent token accounting is not retained.

Full GLM-5.3 is a different deployment problem from GLM-5.3-Flash. The BertholomusAI four-GB10 recipe targets the 753B non-Flash model, all 79 layers plus its MTP layer, using EXL3 and TensorFold. Its value is a documented route to fitting and serving that full model, with explicit cache layouts and a fidelity panel. It does not provide a general model leaderboard, a production peak-memory trace or a claim that four long jobs each occupy the entire configured window. Run AI Here has not reproduced its GPU measurements.

The hardware and quantization contract

The report uses four DGX Spark/GB10 nodes with 128 GB unified memory each, one tensor-parallel rank per node, direct RoCE links and both 200G ports. Its runtime is TensorFold 0.6.0 plus the author's glm-dsa-tp4 branch. The source credits TensorFold's original author separately from the model-specific recipe. Substituting a switched fabric, different rank layout or another engine would be another deployment requiring its own evidence.

The quantization uses exllamav3 1.5.3: routed experts are 3-bit, attention/shared experts/dense layers are promoted, the MTP experts use 4 bits and the output head uses 6. The reported whole-model average is 3.04 bits per weight, across 39 shards totalling 292.7 GB. A “3.0 bpw” artifact name is therefore not a statement that every tensor has identical precision. The exact selected checkpoint revision is not retained in the recipe, so the artifact name and null revision remain separate fields.

The source derives about 69.86 GiB weights per rank from its real configuration. That is a fit calculation, not a sampled peak. It excludes the complete inference working set, graph captures, cache buffers and operating-system demands. File GB, weight GiB and peak unified-memory usage should not be treated as interchangeable capacity numbers. Peak whole-system RAM and a time-series sampling method are absent.

Keep the release tree distinct from its later fixes

October 4 release measurements are labelled engine prefix 3eb35dd. The document's later branch head is ddfad35, with an RDMA completion-matching correction and graph prewarm work on top. Only short executed prefixes are provided for this table; the document's full Git revision does not fill that gap. Both identities are retained so a reader can identify the measurement generation before trying a newer recipe.

The release speed table remains attributed to 3eb35dd, not silently relabelled as the later branch head.

The RDMA incident involves matching queue-pair numbers across two devices, where a collision can produce ibv_post_recv failed. The later fix scopes matching to the device. Another change captures rounds and draft-step graphs during warm-up, removing work from the first concurrent burst. The source reports finite greedy/concurrent equality checks after those changes. It explicitly leaves the original release table unchanged; this guide does not relabel all old numbers as a fresh run of the new head.

Two cache layouts make different promises

Configuration A uses replicated Q5 cache, a 65,536-token limit per stream and four streams. It reports 82 aggregate code tokens/s, with 0.59–0.60-second first-token times, plus 71–73 aggregate chat tokens/s. These are source HTTP observations, not per-user rates. The prompts, cache arrangement and completion behavior matter to any comparison.

Configuration B uses Q5 cache with decode context parallelism over four ranks and a shared 1,048,576-token pool. Four code streams report approximately 64 aggregate tokens/s; chat reports 56. Windows are extents in one pool. This trades a different cache arrangement for a larger configured context. It does not grant four independent million-token allocations. Without raw input and output counts for every concurrent cell, the report supports its own conditional observations rather than a normalized throughput claim.

The source uses GLM's native MTP layer with three draft steps, and no DFlash. Vision is not implemented in this release. Those deployment choices stay attached to the speeds. A separate Flash model with another drafter, parser or visual path would not be a matched performance control.

Audit the token-rate denominator before comparing engines

The README describes its release numbers as HTTP measurements with prompt time included. The available serial streaming probe counts nonempty SSE chunks and derives a post-first-chunk rate. Chunks are not guaranteed to contain one tokenizer token, and that script does not independently validate them against usage counts. The exact concurrent harness and raw usage underlying the release table are not retained in this repository. It is not established that the concurrent release table was computed by this particular serial helper.

These limitations should travel with the values, rather than becoming either an invented corrected speed or an accusation that every reported rate uses the helper's denominator. The separate greedy-output script does read completion usage and wall time for its own requests. That does not retrospectively establish how every other table was calculated. A future reproducible comparison should retain synchronized group wall time, tokenizer or usage output totals, warm-up policy and both TTFT and decode definitions.

The quality panel measures fidelity to BF16

The source's quantization panel uses wikitext-2 test text, 32 segments of 2,048 tokens, BF16 activations and no cache quantization. Forward KL from BF16 to quantized probabilities is 0.109, reverse KL is 0.123 and top-1 agreement is 90.2 percent. Perplexity is 2.911 for the quantized model versus 2.750 for BF16. These are specific probability/text-panel measurements, not percentages of correctly answered tasks.

Panel perplexity is not MMLU, GSM8K or a general task score; no benchmark ranking is derived.

A served-engine prompt-path check reports KL 0.111 against stored BF16 logits on the same panel. Another longer text panel exercises sparse attention and has a different KL; a separate cache-quantization comparison answers yet another question. Combining weight quantization, cache quantization and served-engine differences into one “accuracy loss” would erase the controls. The selected JSON preserves each comparison's reference and scope. No MMLU, GSM8K or coding-task score is inferred from a small KL or a finite greedy equality check.

The long-window evidence is useful and finite

The report finds a needle at 1,039,064 actual input tokens. It also reports 402 prefill tokens/s at a source-labelled 32K input and approximately 76 seconds to first token near 31K. Those rounded input labels are not exact token counts. A planner interested in long documents should budget prefill latency separately from the better short-request decode rate.

A per-rank weight estimate excludes cache, graphs, working memory and system services.

The million-token result establishes a completed retrieval case in this setup. It does not establish multi-hop accuracy, repeated long-context stability or simultaneous full-window requests. The repository's needle helper and finite output-equality probes give useful inspection points, but missing repetitions, model revision, exact concurrent accounting and peak memory remain missing. This recipe is most useful when those boundaries guide the next experiment: hold the artifact and runtime fixed, record the fabric and cache layout, then capture latency, completion correctness and memory over the same workload interval.