# Spark community quality cohort: v7, 12 October

A dated 80-scenario quality cohort with two repeats, model-card sampling and per-deployment reports.

- Type: Performance report
- Evidence: source_reported; site reproduced: false
- Model family: Multiple
- Model: Multiple local deployments and hosted API references
- Original author: Weschera
- Original project: https://github.com/Weschera/spark-bench
- Pinned document: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md
- Source revision: 3eaaeb66aaef7ccf96f3b6e39803817e987cf934
- Source commit time (not experiment date): 2026-10-13T03:42:23Z
- Checked: 2026-10-17
- Experiment date: 2026-10-12
- Runtime topics: Multiple runtimes
- Hardware topics: spark, nvidia, apple
- Reported setup: {"system": "DGX Spark / Mac Studio / hosted APIs, per row", "node_count": null, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": "2026-10-12"}
- Performance figures: attributed selections with per-point conditions; no cross-source ranking

## Conditions

Unknown is null. Recipe options are not measured benchmark settings.

```json
{
  "conditions": {
    "model_revision": null,
    "weight_hash": null,
    "runtime_execution_revision": null,
    "input_tokens": null,
    "output_tokens": null,
    "concurrency": null,
    "cache_state": null,
    "quantization": null,
    "speculative_decoding": null,
    "temperature": null,
    "top_p": null
  },
  "test_profiles": [
    {
      "label": "v7.0 cohort only",
      "evidence_url": "https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md",
      "conditions": {
        "model_revision": null,
        "weight_hash": null,
        "runtime_execution_revision": null,
        "input_tokens": null,
        "output_tokens": "uncapped within each deployment context",
        "concurrency": null,
        "cache_state": null,
        "quantization": null,
        "speculative_decoding": null,
        "temperature": "each model card; per deployment",
        "top_p": null,
        "repetitions": 2,
        "seeds": [
          2001,
          2002
        ],
        "scenario_count": 80,
        "domain_count": 13,
        "thinking": "on/off separate rows",
        "hardware": "per-row local hardware; hosted hardware unknown"
      }
    }
  ],
  "recipe_parameters": null,
  "deployment_options": []
}
```

## Reading notes

- The cohort uses 13 domains and seeds 2001–2002.
- Thinking-on and thinking-off rows are separate.

## Limits

- v7 scores are not comparable with v6.x.
- Hosted API latency includes the network and is not a controlled hardware comparison.

## License declaration

{"declared_spdx": null, "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."}


## Editorial answer

The dated comparison contains 33 valid full deployment runs of v7.0, with 80 scenarios across 13 domains and two repeated seeds. A row identifies a model, quantization, engine/recipe, hardware and thinking mode. TrueScore is the project’s suite score, not a percent-correct rate and not a measure of generation speed.

By Run AI Here editorial · 2026-10-17

AI-assisted source reading and original editorial synthesis; no independent model execution.

## What is the unit of this quality cohort?

The dated comparison contains 33 valid full deployment runs of v7.0, with 80 scenarios across 13 domains and two repeated seeds. A row identifies a model, quantization, engine/recipe, hardware and thinking mode. TrueScore is the project’s suite score, not a percent-correct rate and not a measure of generation speed.

Sources: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L3

## Why retain thinking mode and deployment identity?

The GLM Flash example is 88.9 with thinking enabled and 84.8 with thinking disabled under the named two-Spark EXL3/TensorFold recipe. Those are two source rows, not a general estimate of the value of reasoning across all tasks or runtimes. Different quantizations, engines or hardware in other rows create additional differences.

Sources: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L7

## How are incomplete and failed runs handled?

The cohort requires its golden gate and a transport-error limit below five percent. Three hosted runs each had one timeout. The document excludes incomplete runs and describes reasoning-channel parsing failures separately. Counting only the successful top-line score would discard information the author deliberately retained about the evaluation process.

Sources: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L70

## What comparisons are unsupported?

The author says v7 is not comparable with v6.x and notes that easy shared scenarios compress the upper scores. Hosted APIs have uncontrolled hardware and network latency. This page therefore excerpts two contextual examples rather than republishing the full leaderboard or converting it into a site-wide ranking. No license declaration was found in repository metadata; the original report remains the full source.

Sources: https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L62

## Selected source-reported figures

Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking.

```json
[
  {
    "label": "GLM-5.3 Flash, thinking on",
    "metric": "TrueScore",
    "value": 88.9,
    "unit": "suite score",
    "statistic": "reported value",
    "uncertainty": null,
    "conditions": {
      "system": "DGX Spark / Mac Studio / hosted APIs, per row",
      "node_count": null,
      "parallelism": null,
      "kv_dtype": null,
      "context_limit_tokens": null,
      "experiment_date": "2026-10-12",
      "suite": "v7.0 full, 80 scenarios / 13 domains",
      "repetitions": 2,
      "seeds": [
        2001,
        2002
      ],
      "sampling": "model-card settings",
      "output_budget": "uncapped within each deployment context",
      "hardware": "2 Sparks TP2",
      "quantization": "EXL3 4bpw",
      "runtime": "TensorFold v0.6.0 + DFlash2, Mia recipe v1.8",
      "thinking": true
    },
    "evidence_state": "source_reported",
    "site_reproduced": false,
    "evidence_url": "https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L12",
    "source_document_sha256": "10ceff97d360c33a22e002db2e746a7d7de2aa9bd31051813a8d77e18f60c2cc"
  },
  {
    "label": "GLM-5.3 Flash, thinking off",
    "metric": "TrueScore",
    "value": 84.8,
    "unit": "suite score",
    "statistic": "reported value",
    "uncertainty": null,
    "conditions": {
      "system": "DGX Spark / Mac Studio / hosted APIs, per row",
      "node_count": null,
      "parallelism": null,
      "kv_dtype": null,
      "context_limit_tokens": null,
      "experiment_date": "2026-10-12",
      "suite": "v7.0 full, 80 scenarios / 13 domains",
      "repetitions": 2,
      "seeds": [
        2001,
        2002
      ],
      "sampling": "model-card settings",
      "output_budget": "uncapped within each deployment context",
      "hardware": "2 Sparks TP2",
      "quantization": "EXL3 4bpw",
      "runtime": "TensorFold v0.6.0 + DFlash2, Mia recipe v1.8",
      "thinking": false
    },
    "evidence_state": "source_reported",
    "site_reproduced": false,
    "evidence_url": "https://github.com/Weschera/spark-bench/blob/3eaaeb66aaef7ccf96f3b6e39803817e987cf934/results/comparisons/2026-10-12-v7-ladder/README.md#L26",
    "source_document_sha256": "10ceff97d360c33a22e002db2e746a7d7de2aa9bd31051813a8d77e18f60c2cc"
  }
]
```