MLX LM: M4 Max precision and efficiency reference

What does the maintainer document fix?

The efficiency command uses 2,048 prompt tokens and 128 generated tokens on an M4 Max with 64 GB unified memory. It records a development MLX build, mlx-lm 0.28.2 and macOS 26.1. Its separate evaluation command runs MMLU Pro. Timing and quality therefore come from different programs, even when printed on the same model row.

Source: ml-explore · pinned document ↗

What is the observed precision tradeoff?

For Qwen3-4B Instruct-2507, the selected BF16 and q4 rows report 52.47 and 134.52 generation tokens/s. The corresponding MMLU Pro values are 64.05 and 60.72. Faster decoding does not establish unchanged capability, and a task score cannot be interpreted as a decode rate. Group-size-specific q4 and q5 rows are separate artifacts.

Source: ml-explore · pinned document ↗

How should missing cells be represented?

The 30B-A3B BF16 row contains unavailable efficiency cells while retaining a quality value. Those cells remain absent rather than being converted to zero or to a fabricated failure diagnosis. The memory column is labeled GB by the source and is not silently relabeled GiB. Repetitions and exact weight revisions remain unspecified here.

Source: ml-explore · pinned document ↗

What is the appropriate use of this reference?

Use it as a dated maintainer example of a performance/quality tradeoff on an identified stack. It is not a benchmark of current MLX, Qwen3.8 or a different Apple chip. Preserve the conversion repository, precision/group size, model revision and command parameters before comparing another measurement.

Source: ml-explore · pinned document ↗

Reported figures and their conditions

Selected facts reported by the original author. Each row has its own configuration and measurement boundary. Run AI Here has not reproduced these tests.