Mac Weight Precision: Speed, First Token and Memory

Weight precision is a useful deployment choice only when the model and workload stay attached to the result. A smaller memory footprint does not tell you how accurate the answers remain. A fast generation rate does not tell you how long the first token takes.

This note selects the 4B and 9B rows from weklund’s pinned Qwen3.5 MLX report. They provide a useful same-machine example. They are historical Qwen3.5 measurements, not measurements of the latest Qwen checkpoint or every Mac.

Figure: deterministically drawn from the 18 selected source values. Rate and TTFT bars show the reported standard deviation; memory bars have no error bars because per-cell dispersion was not supplied. The workloads differ between panels. Full values and conditions are also available in text and JSON.

The conditions that make this comparison meaningful

The report identifies an M4 Pro with 10P+4E CPU cores, 20 GPU cores and 64 GB unified memory; MLX 0.31.0 and mlx-lm 0.30.7; experiment date 6 March 2026; one warmup followed by 20 measurements. Its rate table uses 256 input and 100 generated tokens. The TTFT and memory tables use 4,096 input tokens. These are the report’s selected workloads, rather than one common request profile.

For the two long-input tables, the selected sections do not specify output length. That field stays unknown. The general methodology’s maximum generation length does not establish the exact length used for every section.

Same model, three weight formats

Attribution: weklund; generation rows, TTFT rows, memory rows. ± means standard deviation, not a confidence interval. The memory column is a reported peak, not a model-file size or total-system free-memory reading.

Read the tradeoff against your task

In these selected rows, int4 has the higher generation rate and lower reported memory, while bf16 has the shorter first-token wait. That pattern explains why “the fastest precision” needs a stated objective.

For a long output, investigate sustained delivery and total completion time. For a short answer after a long prompt, investigate first-token wait. For a Mac used alongside other applications, record actual system pressure and swap during a representative request. The table can narrow a test plan; it cannot replace those observations.

Avoid subtracting the GiB peak from the machine’s GB specification to advertise “free RAM.” The units and measurement boundaries differ. Likewise, do not combine the short-input rate with the long-input TTFT to calculate an end-to-end response time: they describe different workloads.

What is still unknown

The selected report does not pin exact converted-weight hashes, conversion execution revisions, KV dtype, concurrency, sampling or cache state. Those fields remain null in the export. The reported cooldown policy is useful context, but it is not a thermal trace.

There is no matched task-quality or perplexity evaluation here. The original document also includes broader recommendations; this note does not turn those recommendations into measured quantization accuracy. Before choosing a smaller format for extraction, coding or reasoning, compare its outputs against the same task set and acceptance criteria.

How to use the evidence without overextending it

Keep one candidate model fixed, record its conversion and runtime, and choose representative input and output lengths. Check correctness separately from responsiveness and memory. Record warmup, cache, concurrency and repeated-run variation. Only then decide which format serves the task well enough.

Run AI Here has not independently benchmarked these models. The adjacent Markdown and JSON retain the source revision, row anchors, uncertainty and missing fields so an AI reader can preserve the same limits.