{
  "id": "A21",
  "path": "/guides/glm53-flash-scores-router-bias-retraction/",
  "title": "GLM Flash Scores: Historical NVFP4 and the EXL3 Retraction",
  "summary": "Separate a full historical MMLU run and small task subsets from the corrected EXL3 router stack, whose old task scores were withdrawn and await reruns.",
  "answer": "The September NVFP4 report and October EXL3 correction are different deployment lines. The EXL3 author withdrew its old task scores after finding an unloaded router bias. Current evidence is a same-quant probability check; it does not revalidate old task scores or import the NVFP4 task panel.",
  "published_on": "2026-10-18",
  "modified_on": "2026-10-18",
  "source_checked_on": "2026-10-18",
  "sources": [
    {
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "platform": "github",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "docs/06-benchmarks.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md",
      "document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "report_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md",
      "checked_on": "2026-10-18",
      "metadata_status": "checked",
      "record_id": "glm53-flash-nvfp4-tp3-scores"
    },
    {
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "platform": "github",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "tracks/tp3-main/README.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/tracks/tp3-main/README.md",
      "document_sha256": "6cb79207b37211c1542b98dfe26408e7d0b741133be67286db7838c48976223f",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "report_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/tracks/tp3-main/README.md",
      "checked_on": "2026-10-18",
      "metadata_status": "checked",
      "record_id": "glm53-flash-exl3-router-correction"
    }
  ],
  "supplemental_sources": [
    {
      "document_id": "glm-tp3-readme",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "README.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/README.md",
      "document_sha256": "5912e935333340dbd77ecd01cec8c7e207990bc6f446341f741bb3bd66d6cf4b",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-memory",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "docs/05-memory-ladder.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/05-memory-ladder.md",
      "document_sha256": "29874450acd65f1e6614bd3fc04fe0815ef8b9036007867b8446e63ebb4092de",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-mmlu",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "results/mmlu/mmlu-full-lm-eval-results.json",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json",
      "document_sha256": "86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-tool",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "results/tool-eval-bench/tool-eval-bench-8-trials.json",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json",
      "document_sha256": "7a46e4d7b52956a0fb0662838081b526e7ceeb30e14e553648227bade4dc6df5",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-needle",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "results/needle/needle-in-a-haystack-report.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md",
      "document_sha256": "cceef7123bbb9e9d34c2c6a44bc00055b81f896cbf51977053766e770c30ae3d",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-results",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "results/README.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/README.md",
      "document_sha256": "15388126cd782399e618a52b5cbcd2c5af46f7de347f6903e4705d719f84b037",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-exl3-readme",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "README.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md",
      "document_sha256": "a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-exl3-kl",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "results/main-stack/kl-health.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md",
      "document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-exl3-gates",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "results/main-stack/quality-gates.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md",
      "document_sha256": "9d0387be075bd6c214ab8e9b2bb70a46577b5052884c06c0db02d1869b295f28",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-exl3-stack",
      "parent_record_id": "glm53-flash-nvfp4-tp3-scores",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "docs/20-main-stack.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/docs/20-main-stack.md",
      "document_sha256": "b7ef6f4c82dd1bfa693452cb1fcfb57c8f3270455ca894b667e24048cc20786d",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-readme",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "README.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/README.md",
      "document_sha256": "5912e935333340dbd77ecd01cec8c7e207990bc6f446341f741bb3bd66d6cf4b",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-memory",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "docs/05-memory-ladder.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/05-memory-ladder.md",
      "document_sha256": "29874450acd65f1e6614bd3fc04fe0815ef8b9036007867b8446e63ebb4092de",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-mmlu",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "results/mmlu/mmlu-full-lm-eval-results.json",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json",
      "document_sha256": "86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-tool",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "results/tool-eval-bench/tool-eval-bench-8-trials.json",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json",
      "document_sha256": "7a46e4d7b52956a0fb0662838081b526e7ceeb30e14e553648227bade4dc6df5",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-needle",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "results/needle/needle-in-a-haystack-report.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md",
      "document_sha256": "cceef7123bbb9e9d34c2c6a44bc00055b81f896cbf51977053766e770c30ae3d",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-tp3-results",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark",
      "revision": "eb11cef295c107c5bd21402106eacea5ebd1fbf3",
      "document_path": "results/README.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/README.md",
      "document_sha256": "15388126cd782399e618a52b5cbcd2c5af46f7de347f6903e4705d719f84b037",
      "last_commit_at": "2026-09-09T21:23:55Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-exl3-readme",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "README.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md",
      "document_sha256": "a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-exl3-kl",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "results/main-stack/kl-health.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md",
      "document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-exl3-gates",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "results/main-stack/quality-gates.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md",
      "document_sha256": "9d0387be075bd6c214ab8e9b2bb70a46577b5052884c06c0db02d1869b295f28",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    },
    {
      "document_id": "glm-exl3-stack",
      "parent_record_id": "glm53-flash-exl3-router-correction",
      "publisher": "NNNtrance",
      "repository": "NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark",
      "revision": "83ed1eccd93a15ae9efe495dbee016cd57823c5f",
      "document_path": "docs/20-main-stack.md",
      "pinned_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/docs/20-main-stack.md",
      "document_sha256": "b7ef6f4c82dd1bfa693452cb1fcfb57c8f3270455ca894b667e24048cc20786d",
      "last_commit_at": "2026-10-16T01:23:58Z",
      "checked_on": "2026-10-18",
      "public_URL_is_immutable": true,
      "revision_kind": "Git source document revision; executed engine and weight revisions are separate fields"
    }
  ],
  "facts": [],
  "reported_results": [
    {
      "label": "Tool-eval eight-trial mean",
      "metric": "tool_eval_score",
      "value": 87.8,
      "unit": "points",
      "statistic": "mean across eight trials; source SD 0.9, source CI [87.1,88.2]",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4",
        "model_revision": null,
        "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10",
        "runtime_execution_revision": null,
        "executed_image_digest": null,
        "system": "three DGX Spark GB10, TP3 + EP",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "NVFP4 storage / Marlin MoE",
        "kv_dtype": "FP8",
        "draft": "DFlash2 k7",
        "context_limit_tokens": 1000000,
        "memory_utilization": 0.85,
        "configured_max_requests": 8,
        "batch_tokens": 2048,
        "experiment_date": "2026-09-03 09:08-18:08 UTC",
        "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores",
        "host_peak_RAM": null,
        "temperature": 0,
        "thinking": "on, effort low except MMLU",
        "input_tokens": null,
        "output_tokens": null,
        "cache_state": null,
        "benchmark": "tool-eval-bench 2.6.1.dev39+gd3352edf5",
        "scenarios": 88,
        "trials": 8,
        "seed": 42,
        "max_turns": 8,
        "concurrency": 1,
        "source_trial_SD": 0.9,
        "source_CI95": [
          87.1,
          88.2
        ],
        "CI_method_independently_verified": false
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-tp3-scores",
      "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "source_line": 47,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L47",
      "source_record_id": "glm53-flash-nvfp4-tp3-scores"
    },
    {
      "label": "Tool-eval Pass^8",
      "metric": "all_trials_pass_rate",
      "value": 76.1,
      "unit": "percent",
      "statistic": "single finite benchmark; no dispersion reported",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4",
        "model_revision": null,
        "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10",
        "runtime_execution_revision": null,
        "executed_image_digest": null,
        "system": "three DGX Spark GB10, TP3 + EP",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "NVFP4 storage / Marlin MoE",
        "kv_dtype": "FP8",
        "draft": "DFlash2 k7",
        "context_limit_tokens": 1000000,
        "memory_utilization": 0.85,
        "configured_max_requests": 8,
        "batch_tokens": 2048,
        "experiment_date": "2026-09-03 09:08-18:08 UTC",
        "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores",
        "host_peak_RAM": null,
        "temperature": 0,
        "thinking": "on, effort low except MMLU",
        "input_tokens": null,
        "output_tokens": null,
        "cache_state": null,
        "benchmark": "tool-eval-bench",
        "scenarios": 88,
        "trials": 8,
        "concurrency": 1
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-tp3-scores",
      "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "source_line": 56,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L56",
      "source_record_id": "glm53-flash-nvfp4-tp3-scores"
    },
    {
      "label": "GSM8K 200-question subset",
      "metric": "accuracy",
      "value": 94.0,
      "unit": "percent",
      "statistic": "single finite benchmark; no dispersion reported",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4",
        "model_revision": null,
        "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10",
        "runtime_execution_revision": null,
        "executed_image_digest": null,
        "system": "three DGX Spark GB10, TP3 + EP",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "NVFP4 storage / Marlin MoE",
        "kv_dtype": "FP8",
        "draft": "DFlash2 k7",
        "context_limit_tokens": 1000000,
        "memory_utilization": 0.85,
        "configured_max_requests": 8,
        "batch_tokens": 2048,
        "experiment_date": "2026-09-03 09:08-18:08 UTC",
        "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores",
        "host_peak_RAM": null,
        "temperature": 0,
        "thinking": "on, effort low except MMLU",
        "input_tokens": null,
        "output_tokens": null,
        "cache_state": null,
        "benchmark": "GSM8K",
        "subset_size": 200,
        "full_test_size": 1319,
        "correct": 188,
        "shots": 8,
        "CoT": true,
        "concurrency": null
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-tp3-scores",
      "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "source_line": 197,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L197",
      "source_record_id": "glm53-flash-nvfp4-tp3-scores"
    },
    {
      "label": "IFEval prompt strict, custom evaluator",
      "metric": "accuracy",
      "value": 78.9,
      "unit": "percent",
      "statistic": "single finite benchmark; no dispersion reported",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4",
        "model_revision": null,
        "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10",
        "runtime_execution_revision": null,
        "executed_image_digest": null,
        "system": "three DGX Spark GB10, TP3 + EP",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "NVFP4 storage / Marlin MoE",
        "kv_dtype": "FP8",
        "draft": "DFlash2 k7",
        "context_limit_tokens": 1000000,
        "memory_utilization": 0.85,
        "configured_max_requests": 8,
        "batch_tokens": 2048,
        "experiment_date": "2026-09-03 09:08-18:08 UTC",
        "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores",
        "host_peak_RAM": null,
        "temperature": 0,
        "thinking": "on, effort low except MMLU",
        "input_tokens": null,
        "output_tokens": null,
        "cache_state": null,
        "benchmark": "IFEval",
        "prompts": 541,
        "correct": 427,
        "concurrency": null,
        "evaluator": "custom; not independently cross-checked",
        "proven_lower_bound": false
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-tp3-scores",
      "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "source_line": 155,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L155",
      "source_record_id": "glm53-flash-nvfp4-tp3-scores"
    },
    {
      "label": "IFEval instruction level, custom evaluator",
      "metric": "accuracy",
      "value": 85.1,
      "unit": "percent",
      "statistic": "single finite benchmark; no dispersion reported",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4",
        "model_revision": null,
        "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10",
        "runtime_execution_revision": null,
        "executed_image_digest": null,
        "system": "three DGX Spark GB10, TP3 + EP",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "NVFP4 storage / Marlin MoE",
        "kv_dtype": "FP8",
        "draft": "DFlash2 k7",
        "context_limit_tokens": 1000000,
        "memory_utilization": 0.85,
        "configured_max_requests": 8,
        "batch_tokens": 2048,
        "experiment_date": "2026-09-03 09:08-18:08 UTC",
        "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores",
        "host_peak_RAM": null,
        "temperature": 0,
        "thinking": "on, effort low except MMLU",
        "input_tokens": null,
        "output_tokens": null,
        "cache_state": null,
        "benchmark": "IFEval",
        "instructions": 834,
        "correct": 710,
        "concurrency": null,
        "evaluator": "custom; not independently cross-checked",
        "proven_lower_bound": false
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-tp3-scores",
      "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "source_line": 156,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L156",
      "source_record_id": "glm53-flash-nvfp4-tp3-scores"
    },
    {
      "label": "Needle four sizes x five depths",
      "metric": "retrieval_correct_count",
      "value": 20,
      "unit": "correct probes",
      "statistic": "single finite benchmark; no dispersion reported",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4",
        "model_revision": null,
        "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10",
        "runtime_execution_revision": null,
        "executed_image_digest": null,
        "system": "three DGX Spark GB10, TP3 + EP",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "NVFP4 storage / Marlin MoE",
        "kv_dtype": "FP8",
        "draft": "DFlash2 k7",
        "context_limit_tokens": 1000000,
        "memory_utilization": 0.85,
        "configured_max_requests": 8,
        "batch_tokens": 2048,
        "experiment_date": "2026-09-03 09:08-18:08 UTC",
        "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores",
        "host_peak_RAM": null,
        "temperature": 0,
        "thinking": "on, effort low except MMLU",
        "input_tokens": null,
        "output_tokens": null,
        "cache_state": null,
        "benchmark": "single-fact NIAH",
        "correct": 20,
        "total": 20,
        "max_actual_input_tokens": 997952,
        "concurrency": null,
        "initial_timeout_seconds": 120,
        "rerun_timeout_seconds": 3600,
        "identical_content_seed_on_rerun": true,
        "multihop_validated": false
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-tp3-scores",
      "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "source_line": 221,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L221",
      "source_record_id": "glm53-flash-nvfp4-tp3-scores"
    },
    {
      "label": "Full MMLU exact accuracy",
      "metric": "accuracy",
      "value": 0.8594217347956131,
      "unit": "fraction",
      "statistic": "single full MMLU, harness standard error 0.0028276391244761038",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4",
        "model_revision": null,
        "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10",
        "runtime_execution_revision": null,
        "executed_image_digest": null,
        "system": "three DGX Spark GB10, TP3 + EP",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "NVFP4 storage / Marlin MoE",
        "kv_dtype": "FP8",
        "draft": "DFlash2 k7",
        "context_limit_tokens": 1000000,
        "memory_utilization": 0.85,
        "configured_max_requests": 8,
        "batch_tokens": 2048,
        "experiment_date": "2026-09-03 09:08-18:08 UTC",
        "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores",
        "host_peak_RAM": null,
        "temperature": null,
        "thinking": null,
        "input_tokens": null,
        "output_tokens": null,
        "cache_state": null,
        "benchmark": "MMLU",
        "subset_size": 14042,
        "subjects": 57,
        "shots": 0,
        "scoring": "loglikelihood, no generation or sampling",
        "harness": "lm-eval 0.4.9",
        "concurrency": 8,
        "timeout_seconds": 36000,
        "max_retries": 6,
        "accuracy_stderr": 0.0028276391244761038,
        "stderr_is_SD": false,
        "initial_harness_timeout_at_fraction": 0.81,
        "engine_errors_reported": 0,
        "rerun_same_content": true
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-tp3-mmlu",
      "source_document_sha256": "86ed1c5586cfe259e7ff438fff2baf1df2dd2ebc069b3db5bbce0763c9ef03c8",
      "source_line": 4,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json#L4",
      "source_record_id": "glm53-flash-nvfp4-tp3-scores"
    },
    {
      "label": "Pre-fix assistant KL",
      "metric": "KL_reference_to_engine",
      "value": 0.1033,
      "unit": "nats",
      "statistic": "source-reported observation; dispersion unreported",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "turboderp/GLM-5.3-Flash-exl3",
        "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5",
        "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches",
        "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7",
        "runtime_execution_revision": null,
        "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef",
        "executed_image_digest": null,
        "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261",
        "system": "three DGX Spark GB10",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "EXL3 4.05 bpw full scope",
        "kv_dtype": "FP8",
        "experiment_date": "2026-10-07 to 2026-10-08",
        "concurrency": null,
        "input_tokens": 21072,
        "scored_positions": 21066,
        "assistant_positions": 6601,
        "quality_panel": "six author-written texts; text set and per-position logits not published",
        "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16",
        "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate",
        "temperature": null,
        "thinking": "reference panel has reasoning effort low / clear_thinking",
        "cache_state": "prefill target log probabilities, before DFlash proposals",
        "memory_utilization": 0.75,
        "host_peak_RAM": null,
        "current_task_benchmark_score": null,
        "old_EXL3_task_scores_withdrawn": true,
        "TP2_corrected_or_tested": false,
        "measured_build": "a4f",
        "production_perplexity": null,
        "pre_fix_comparison_is_retracted_task_score": false
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-exl3-kl",
      "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1",
      "source_line": 40,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L40",
      "source_record_id": "glm53-flash-exl3-router-correction"
    },
    {
      "label": "Production assistant KL",
      "metric": "KL_reference_to_engine",
      "value": 0.00828,
      "unit": "nats",
      "statistic": "source-reported observation; dispersion unreported",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "turboderp/GLM-5.3-Flash-exl3",
        "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5",
        "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches",
        "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7",
        "runtime_execution_revision": null,
        "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef",
        "executed_image_digest": null,
        "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261",
        "system": "three DGX Spark GB10",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "EXL3 4.05 bpw full scope",
        "kv_dtype": "FP8",
        "experiment_date": "2026-10-07 to 2026-10-08",
        "concurrency": null,
        "input_tokens": 21072,
        "scored_positions": 21066,
        "assistant_positions": 6601,
        "quality_panel": "six author-written texts; text set and per-position logits not published",
        "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16",
        "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate",
        "temperature": null,
        "thinking": "reference panel has reasoning effort low / clear_thinking",
        "cache_state": "prefill target log probabilities, before DFlash proposals",
        "memory_utilization": 0.75,
        "host_peak_RAM": null,
        "current_task_benchmark_score": null,
        "old_EXL3_task_scores_withdrawn": true,
        "TP2_corrected_or_tested": false,
        "measured_build": "a4i",
        "production_perplexity": null,
        "pre_fix_comparison_is_retracted_task_score": false
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-exl3-kl",
      "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1",
      "source_line": 42,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L42",
      "source_record_id": "glm53-flash-exl3-router-correction"
    },
    {
      "label": "Bias-fix assistant perplexity",
      "metric": "perplexity",
      "value": 2.222,
      "unit": "perplexity",
      "statistic": "source-reported observation; dispersion unreported",
      "uncertainty": null,
      "conditions": {
        "model": "GLM-5.3-Flash",
        "model_artifact": "turboderp/GLM-5.3-Flash-exl3",
        "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5",
        "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches",
        "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7",
        "runtime_execution_revision": null,
        "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef",
        "executed_image_digest": null,
        "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261",
        "system": "three DGX Spark GB10",
        "node_count": 3,
        "parallelism": "TP3+EP",
        "weight_precision": "EXL3 4.05 bpw full scope",
        "kv_dtype": "FP8",
        "experiment_date": "2026-10-07 to 2026-10-08",
        "concurrency": null,
        "input_tokens": 21072,
        "scored_positions": 21066,
        "assistant_positions": 6601,
        "quality_panel": "six author-written texts; text set and per-position logits not published",
        "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16",
        "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate",
        "temperature": null,
        "thinking": "reference panel has reasoning effort low / clear_thinking",
        "cache_state": "prefill target log probabilities, before DFlash proposals",
        "memory_utilization": 0.75,
        "host_peak_RAM": null,
        "current_task_benchmark_score": null,
        "old_EXL3_task_scores_withdrawn": true,
        "TP2_corrected_or_tested": false,
        "measured_build": "a4g",
        "production_perplexity": null,
        "pre_fix_comparison_is_retracted_task_score": false
      },
      "evidence_state": "source_reported",
      "site_reproduced": false,
      "source_document_id": "glm-exl3-kl",
      "source_document_sha256": "9fcb41cb68098fe0e982f727f6c57b86656748fd41bbd6a0f64dfc794dbb2ba1",
      "source_line": 41,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md#L41",
      "source_record_id": "glm53-flash-exl3-router-correction"
    }
  ],
  "conditions": {
    "historical_NVFP4": {
      "model": "GLM-5.3-Flash",
      "model_artifact": "local-inference-lab/GLM-5.3-Flash-NVFP4",
      "model_revision": null,
      "engine": "LIL vLLM 0.1.dev0+lil.jovian.9c4dd0548, harem/glm53-lil:t10",
      "runtime_execution_revision": null,
      "executed_image_digest": null,
      "system": "three DGX Spark GB10, TP3 + EP",
      "node_count": 3,
      "parallelism": "TP3+EP",
      "weight_precision": "NVFP4 storage / Marlin MoE",
      "kv_dtype": "FP8",
      "draft": "DFlash2 k7",
      "context_limit_tokens": 1000000,
      "memory_utilization": 0.85,
      "configured_max_requests": 8,
      "batch_tokens": 2048,
      "experiment_date": "2026-09-03 09:08-18:08 UTC",
      "status": "historical NVFP4 line, maintenance stopped September 6; not corrected EXL3 scores",
      "host_peak_RAM": null,
      "temperature": 0,
      "thinking": "on, effort low except MMLU",
      "input_tokens": null,
      "output_tokens": null,
      "cache_state": null
    },
    "corrected_EXL3": {
      "model": "GLM-5.3-Flash",
      "model_artifact": "turboderp/GLM-5.3-Flash-exl3",
      "model_revision": "2a30229e67012798ba9f0cd832bb78abf4c363d5",
      "engine": "vLLM 0.30.1rc1.dev709+g21d93d0d8 plus cuda-exl3 and 14 patches",
      "base_engine_revision": "21d93d0d8c0e9627900020382bfce4730e61cab7",
      "runtime_execution_revision": null,
      "production_plugin_revision": "448f1d6b8964b96e7d4e299494538bfc240149ef",
      "executed_image_digest": null,
      "production_image_id": "sha256:8c4f560d92ba190ed84983bdf12e9975c5b4f1ec6dc380b5b93041b2a19ac261",
      "system": "three DGX Spark GB10",
      "node_count": 3,
      "parallelism": "TP3+EP",
      "weight_precision": "EXL3 4.05 bpw full scope",
      "kv_dtype": "FP8",
      "experiment_date": "2026-10-07 to 2026-10-08",
      "concurrency": null,
      "input_tokens": 21072,
      "scored_positions": 21066,
      "assistant_positions": 6601,
      "quality_panel": "six author-written texts; text set and per-position logits not published",
      "reference": "exllamav3 1.5.4 running SAME quantized checkpoint, not BF16",
      "metric_denominator": "assistant-token KL, reference top512 + remainder bin; lower-bound estimate",
      "temperature": null,
      "thinking": "reference panel has reasoning effort low / clear_thinking",
      "cache_state": "prefill target log probabilities, before DFlash proposals",
      "memory_utilization": 0.75,
      "host_peak_RAM": null,
      "current_task_benchmark_score": null,
      "old_EXL3_task_scores_withdrawn": true,
      "TP2_corrected_or_tested": false
    }
  },
  "generation": {
    "text_draft": "local Spark Qwen, then Codex factual review and rewrite; actual receipt in editorial-batch8 report",
    "review": "Run AI Here with Codex",
    "image": "deterministic matplotlib data figures and evidence maps",
    "local_first": true
  },
  "image": {
    "url": "/assets/batch8-glmflash-hero.webp",
    "alt": "GLM Flash: keep the scoring line attached. No NVFP4 score is assigned to the corrected EXL3 deployment. No cross-stack ranking.",
    "width": 1600,
    "height": 900,
    "sha256": "34148f9fadc7ad5f415ad084beb2014c1c1e335782d5c6d2d5d81571894295da",
    "caption": "No NVFP4 score is assigned to the corrected EXL3 deployment. No cross-stack ranking."
  },
  "case_reports": [
    {
      "id": "NVFP4-harness-only-reruns",
      "classification": "source_reported_harness_timeouts",
      "MMLU_initial_progress_fraction": 0.81,
      "MMLU_initial_error": "aiohttp total timeout includes queue",
      "MMLU_rerun_timeout_seconds": 36000,
      "needle_initial_timeout_seconds": 120,
      "needle_rerun_timeout_seconds": 3600,
      "same_content_and_seed": true,
      "reported_engine_errors": 0,
      "initial_partial_not_full_score": true,
      "source_document_id": "glm-tp3-scores",
      "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "source_line": 35,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L35"
    },
    {
      "id": "EXL3-router-bias-missing",
      "classification": "source_reported_correctness_bug_and_retraction",
      "plugin_pre_fix_short": "754421f or earlier",
      "missing_parameter": "e_score_correction_bias",
      "source_scope": "EXL3 plugin line only; no evidence that separate NVFP4 line shares defect",
      "old_task_scores_withdrawn": true,
      "current_task_score": null,
      "source_document_id": "glm-exl3-readme",
      "source_document_sha256": "a399f1dd57e66933e4b947eb4f14212815dc0526deee69942624759b178038d0",
      "source_line": 18,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md#L18"
    }
  ],
  "quality_comparisons": [
    {
      "id": "NVFP4-historical-task-panel",
      "record_id": "glm53-flash-nvfp4-tp3-scores",
      "task_scores_current_EXL3": false,
      "task_scores_withdrawn_by_EXL3_notice": false,
      "source_scope": "separate NVFP4 repository, September 3 measurements",
      "tool_trial_scores": [
        88,
        88,
        86,
        87,
        88,
        88,
        88,
        89
      ],
      "tool_mean": 87.8,
      "source_SD": 0.9,
      "source_CI95": [
        87.1,
        88.2
      ],
      "MMLU_accuracy": 0.8594217347956131,
      "MMLU_standard_error": 0.0028276391244761038,
      "IFEval_evaluator_crosschecked": false,
      "proven_IFEval_lower_bound": false
    },
    {
      "id": "EXL3-retraction-and-rerun-gap",
      "record_id": "glm53-flash-exl3-router-correction",
      "withdrawn_on": "2026-10-08",
      "old_task_scores_withdrawn": true,
      "current_task_accuracy": null,
      "current_task_benchmarks_rerun": false,
      "router_bias_loaded_after_fix": true,
      "production_KL": 0.00828,
      "production_PPL": null,
      "bias_fix_PPL": 2.222,
      "reference_PPL": 2.224,
      "BF16_reference": false,
      "public_text_set_available": false,
      "full_per_position_logits_public": false,
      "TP2_fixed_or_validated": false
    }
  ],
  "memory_observations": [
    {
      "kind": "historical operational free host memory, not peak",
      "value_range": [
        6.4,
        10
      ],
      "unit": "GB, source approximate",
      "historical_NVFP4_only": true,
      "peak_RAM": null,
      "source_document_id": "glm-tp3-scores",
      "source_document_sha256": "e60fb038ca4ab3785a68abe0cdf0906452052e091443c201f90d93ea7590a495",
      "source_line": 28,
      "evidence_url": "https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md#L28"
    }
  ],
  "schema_version": "1.0.0",
  "record_type": "source_backed_explainer",
  "evidence_state": "source_reported",
  "site_reproduced": false,
  "metrics": null,
  "performance_ranking_allowed": false,
  "body_markdown": "# GLM Flash Scores: Historical NVFP4 and the EXL3 Retraction\n\n![GLM Flash: keep the scoring line attached. No NVFP4 score is assigned to the corrected EXL3 deployment. No cross-stack ranking.](/assets/batch8-glmflash-hero.webp)\n\n*No NVFP4 score is assigned to the corrected EXL3 deployment. No cross-stack ranking.*\n\nA deployment's scores need a version history just as much as its launch command does. Two original GLM-5.3-Flash reports illustrate why: a September NVFP4 task campaign and an October EXL3 correctness investigation. Both use three DGX Sparks, but they differ in quantization, plugin, engine and measurement purpose. The EXL3 author withdrew its old task scores after finding a missing router bias. Reading both reports is useful only if their boundaries survive the summary. Run AI Here has not rerun either deployment or assigned them a combined score.\n\n## The historical NVFP4 campaign has its own identity\n\nThe [September 3 benchmark ledger](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/docs/06-benchmarks.md) describes a continuous 09:08–18:08 UTC campaign using three GB10 nodes, TP3 plus expert parallelism, Marlin MoE, FP8 KV and DFlash2 k7. The image is `harem/glm53-lil:t10`, with a short engine identity. The configured context ceiling is **1,000,000 decimal tokens**, max sequences eight and batched tokens 2,048. Memory utilization for these benchmark runs is **0.85**, rather than the later production 0.88 setting.\n\nThe [NVFP4 project README](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/README.md) says this line stopped maintenance on September 6 and moved to EXL3. The old measurements remain historical observations of that stack. Their checkpoint revision and executed image digest are not pinned, so neither is manufactured from the documentation commit. Most generation tests use temperature zero with low reasoning effort; MMLU is the explicit exception because it performs loglikelihood scoring without generation or sampling.\n\n## Full MMLU is different from a small generation subset\n\nThe saved [raw MMLU result](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/mmlu/mmlu-full-lm-eval-results.json) gives accuracy **0.8594217347956131** and standard error **0.0028276391244761038**. This is the full 14,042-question, 57-subject, zero-shot run under lm-eval 0.4.9. The uncertainty field is a harness standard error, not a repeated-trial standard deviation or a confidence interval. Replacing that meaning with a generic plus/minus label would lose useful statistical information.\n\nThe harness uses concurrency eight, a 36,000-second timeout and six retries. An initial attempt stalls around 81 percent because aiohttp's total timeout includes queued work. The repeated full run keeps the content and reports no engine errors. The failed harness attempt is not a measured zero-accuracy model run, and raising the request timeout is not an engine OOM fix. The final result and the reason for the rerun both remain queryable.\n\n![The check that exposed the routing defect. Six texts, 21,066 scored positions; 6,601 assistant positions. Reference is not BF16.](/assets/batch8-glmflash-kl.webp)\n\n*Six texts, 21,066 scored positions; 6,601 assistant positions. Reference is not BF16.*\n\nThe GSM8K result is instead 188 correct out of 200 selected questions, with eight-shot chain-of-thought prompts and low effort. It is 94.0 percent on that subset, not the full 1,319-question test and not a matched comparison to a zero-shot, thinking-off deployment. Different prompting and answer extraction can change what a score means even before hardware enters the comparison.\n\n## Reliability and evaluator limits are part of the score\n\nThe [raw tool-evaluation report](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/tool-eval-bench/tool-eval-bench-8-trials.json) uses tool-eval-bench 2.6.1.dev39+gd3352edf5, 88 hard scenarios and eight trials. The listed trial scores are 88, 88, 86, 87, 88, 88, 88 and 89. The publisher reports a rounded mean 87.8, SD 0.9 and 95 percent interval [87.1, 88.2]; the interval calculation is not independently verified here. The top-level final score is the last trial, not the mean. Pass^8 is 76.1 percent, a stricter repeated-success view than the average score.\n\nThe same ledger's IFEval uses a custom evaluator: strict prompt accuracy is 427/541, or 78.9 percent, and instruction-level accuracy is 710/834, or 85.1 percent. A known suspicious title-format failure motivates checking the evaluator against an official implementation. One suspected false negative does not prove that every discrepancy goes in one direction, so this guide does not certify the score as a mathematical lower bound. Its evaluator remains unverified.\n\nThe [needle report](https://github.com/NNNtrance/GLM-5.3-Flash-NVFP4-TP3-3x-DGX-Spark/blob/eb11cef295c107c5bd21402106eacea5ebd1fbf3/results/needle/needle-in-a-haystack-report.md) records 20 successful single-fact probes over four lengths and five depths, through 997,952 actual tokens. The initial 120-second timeout is raised to 3,600 seconds for the same-content rerun. That is finite retrieval evidence, not demonstrated long-context reasoning, concurrency at the full window or a whole-workload peak-memory trace.\n\n![What the historical task scores measure. MMLU uses loglikelihood and no sampling. These are heterogeneous task scores, not a composite.](/assets/batch8-glmflash-scores.webp)\n\n*MMLU uses loglikelihood and no sampling. These are heterogeneous task scores, not a composite.*\n\n## The EXL3 notice changes what can be quoted\n\nThe [October 8 correctness notice](https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/README.md) identifies an unloaded `e_score_correction_bias` in the old cuda-exl3 plugin. The router uses zeros instead of the model's balancing bias and selects wrong experts. The author withdraws the old EXL3 repository's task scores, including its prior MMLU, tool, GSM8K, IFEval and needle panels, until they are rerun. Old speed numbers also do not define corrected routing's speed.\n\nThat notice concerns the EXL3 plugin line. It is not evidence that a separate Marlin NVFP4 implementation has the same defect. Conversely, the separate NVFP4 scores cannot validate the corrected EXL3 model. Keeping those two statements together avoids both discarding unrelated measurements without evidence and filling a corrected stack's missing task scores with a sibling stack's results.\n\nThe [new TP3 main recipe](https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/tracks/tp3-main/README.md) pins target weights to `2a30229e67012798ba9f0cd832bb78abf4c363d5`, base vLLM to `21d93d0d8c0e9627900020382bfce4730e61cab7` and production cuda-exl3 to `448f1d6b8964b96e7d4e299494538bfc240149ef`. It adds a boot gate for the bias. The reported local image ID is retained as an ID, not mislabelled a registry digest. Published patch comments differ from the files in that image, so a byte-identical rebuild is not promised. TP2 has not been ported or tested on this corrected path.\n\n## The rerun is probability health, not a new task leaderboard\n\nThe detailed [KL investigation](https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/kl-health.md) compares the **same quantized checkpoint** to exllamav3 1.5.4, not to BF16. Six author-written texts have 21,072 tokens, 21,066 scored positions and 6,601 assistant positions. The partitioned reference-top-512-plus-remainder estimate is a lower-bound KL measurement; assistant positions form the headline because other regions behave differently.\n\nPre-fix assistant KL is 0.1033; production a4i is 0.00828. The detailed table's perplexity 2.222 belongs to the **bias-fix a4g** arm, against reference 2.224. Production a4i perplexity is unreported in that table, so it remains null even where a short summary groups the numbers together. Neither value gives task accuracy. The prepared text set and full per-position logits are not public, which limits independent reproduction.\n\n![Retractions and reruns belong in the result. Bias-fix perplexity 2.222 belongs to a4g. Production a4i perplexity remains null in the detailed table.](/assets/batch8-glmflash-reruns.webp)\n\n*Bias-fix perplexity 2.222 belongs to a4g. Production a4i perplexity remains null in the detailed table.*\n\nThe [current quality-gate document](https://github.com/NNNtrance/GLM-5.3-Flash-EXL3-DGX-Spark/blob/83ed1eccd93a15ae9efe495dbee016cd57823c5f/results/main-stack/quality-gates.md) explicitly says task benchmarks have not been rerun on the corrected stack. The useful publication is therefore a correction with a missing-result ledger: historical NVFP4 tasks retain their own date and harness; withdrawn EXL3 tasks stay withdrawn; corrected EXL3 probability checks retain their reference; future task reruns remain pending. A search or AI summary should return those states alongside any numeric excerpt, rather than selecting the most flattering score from an older table.\n"
}