# MLX LM: M4 Max precision and efficiency reference

Maintainer benchmark document with M4 Max hardware and explicit package versions, separating capability and efficiency columns.

- Type: Performance report
- Evidence: source_reported; site reproduced: false
- Model family: Qwen
- Model: Qwen3-4B / Qwen3-30B-A3B Instruct-2507
- Original author: ml-explore
- Original project: https://github.com/ml-explore/mlx-lm
- Pinned document: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md
- Source revision: 93e6d4a6447b3080861985947f0820e76fcb957d
- Source commit time (not experiment date): 2026-10-16T02:48:11Z
- Checked: 2026-10-17
- Experiment date: None
- Runtime topics: MLX
- Hardware topics: apple
- Reported setup: {"system": "Apple M4 Max / 64 GB unified memory", "node_count": 1, "parallelism": null, "kv_dtype": null, "context_limit_tokens": null, "experiment_date": null}
- Performance figures: attributed selections with per-point conditions; no cross-source ranking

## Conditions

Unknown is null. Recipe options are not measured benchmark settings.

```json
{
  "conditions": {
    "model_revision": null,
    "weight_hash": null,
    "runtime_execution_revision": null,
    "input_tokens": null,
    "output_tokens": null,
    "concurrency": null,
    "cache_state": null,
    "quantization": null,
    "speculative_decoding": null,
    "temperature": null,
    "top_p": null
  },
  "test_profiles": [
    {
      "label": "Efficiency benchmark",
      "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md",
      "conditions": {
        "model_revision": null,
        "weight_hash": null,
        "runtime_execution_revision": null,
        "input_tokens": 2048,
        "output_tokens": 128,
        "concurrency": null,
        "cache_state": null,
        "quantization": "BF16 / q8 / q6 / q5 / q4, per model row",
        "speculative_decoding": null,
        "temperature": null,
        "top_p": null,
        "runtime_version": "mlx 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2 / macOS 26.1"
      }
    }
  ],
  "recipe_parameters": null,
  "deployment_options": []
}
```

## Reading notes

- The efficiency command specifies 2048 prompt and 128 generated tokens.
- Per-model precision rows retain separate memory and quality measurements.

## Limits

- This dated package stack is not a measurement of current MLX or Qwen3.8.
- Missing, failed and untested table cells do not mean zero performance.

## License declaration

{"declared_spdx": "MIT", "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."}


## Editorial answer

The efficiency command uses 2,048 prompt tokens and 128 generated tokens on an M4 Max with 64 GB unified memory. It records a development MLX build, mlx-lm 0.28.2 and macOS 26.1. Its separate evaluation command runs MMLU Pro. Timing and quality therefore come from different programs, even when printed on the same model row.

By Run AI Here editorial · 2026-10-17

AI-assisted source reading and original editorial synthesis; no independent model execution.

## What does the maintainer document fix?

The efficiency command uses 2,048 prompt tokens and 128 generated tokens on an M4 Max with 64 GB unified memory. It records a development MLX build, mlx-lm 0.28.2 and macOS 26.1. Its separate evaluation command runs MMLU Pro. Timing and quality therefore come from different programs, even when printed on the same model row.

Sources: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L14

## What is the observed precision tradeoff?

For Qwen3-4B Instruct-2507, the selected BF16 and q4 rows report 52.47 and 134.52 generation tokens/s. The corresponding MMLU Pro values are 64.05 and 60.72. Faster decoding does not establish unchanged capability, and a task score cannot be interpreted as a decode rate. Group-size-specific q4 and q5 rows are separate artifacts.

Sources: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L29

## How should missing cells be represented?

The 30B-A3B BF16 row contains unavailable efficiency cells while retaining a quality value. Those cells remain absent rather than being converted to zero or to a fabricated failure diagnosis. The memory column is labeled GB by the source and is not silently relabeled GiB. Repetitions and exact weight revisions remain unspecified here.

Sources: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L51

## What is the appropriate use of this reference?

Use it as a dated maintainer example of a performance/quality tradeoff on an identified stack. It is not a benchmark of current MLX, Qwen3.8 or a different Apple chip. Preserve the conversion repository, precision/group size, model revision and command parameters before comparing another measurement.

Sources: https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L23

## Selected source-reported figures

Each point retains its own configuration, measurement unit and source. No site reproduction or cross-source ranking.

```json
[
  {
    "label": "Qwen3-4B Instruct-2507 BF16",
    "metric": "generation",
    "value": 52.47,
    "unit": "tokens/s",
    "statistic": "reported value",
    "uncertainty": null,
    "conditions": {
      "system": "Apple M4 Max / 64 GB unified memory",
      "node_count": 1,
      "parallelism": null,
      "kv_dtype": null,
      "context_limit_tokens": null,
      "experiment_date": null,
      "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2",
      "os": "macOS 26.1",
      "input_tokens": 2048,
      "output_tokens": 128,
      "repetitions": null,
      "cache_state": null,
      "concurrency": null,
      "model": "Qwen/Qwen3-4B-Instruct-2507",
      "quantization": "BF16"
    },
    "evidence_state": "source_reported",
    "site_reproduced": false,
    "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L31",
    "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
  },
  {
    "label": "Qwen3-4B Instruct-2507 q4",
    "metric": "generation",
    "value": 134.52,
    "unit": "tokens/s",
    "statistic": "reported value",
    "uncertainty": null,
    "conditions": {
      "system": "Apple M4 Max / 64 GB unified memory",
      "node_count": 1,
      "parallelism": null,
      "kv_dtype": null,
      "context_limit_tokens": null,
      "experiment_date": null,
      "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2",
      "os": "macOS 26.1",
      "input_tokens": 2048,
      "output_tokens": 128,
      "repetitions": null,
      "cache_state": null,
      "concurrency": null,
      "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
      "quantization": "q4"
    },
    "evidence_state": "source_reported",
    "site_reproduced": false,
    "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L37",
    "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
  },
  {
    "label": "Qwen3-4B Instruct-2507 BF16",
    "metric": "MMLU_Pro",
    "value": 64.05,
    "unit": "reported score",
    "statistic": "reported value",
    "uncertainty": null,
    "conditions": {
      "system": "Apple M4 Max / 64 GB unified memory",
      "node_count": 1,
      "parallelism": null,
      "kv_dtype": null,
      "context_limit_tokens": null,
      "experiment_date": null,
      "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2",
      "os": "macOS 26.1",
      "input_tokens": null,
      "output_tokens": null,
      "repetitions": null,
      "cache_state": null,
      "concurrency": null,
      "model": "Qwen/Qwen3-4B-Instruct-2507",
      "quantization": "BF16",
      "harness": "mlx_lm.evaluate --task mmlu_pro"
    },
    "evidence_state": "source_reported",
    "site_reproduced": false,
    "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L31",
    "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
  },
  {
    "label": "Qwen3-4B Instruct-2507 q4",
    "metric": "MMLU_Pro",
    "value": 60.72,
    "unit": "reported score",
    "statistic": "reported value",
    "uncertainty": null,
    "conditions": {
      "system": "Apple M4 Max / 64 GB unified memory",
      "node_count": 1,
      "parallelism": null,
      "kv_dtype": null,
      "context_limit_tokens": null,
      "experiment_date": null,
      "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2",
      "os": "macOS 26.1",
      "input_tokens": null,
      "output_tokens": null,
      "repetitions": null,
      "cache_state": null,
      "concurrency": null,
      "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
      "quantization": "q4",
      "harness": "mlx_lm.evaluate --task mmlu_pro"
    },
    "evidence_state": "source_reported",
    "site_reproduced": false,
    "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L37",
    "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
  }
]
```