{
  "id": "mlx-lm-m4max-reference",
  "title": "MLX LM: M4 Max precision and efficiency reference",
  "kind": "benchmark",
  "source_type": "maintainer_benchmark",
  "summary": "Maintainer benchmark document with M4 Max hardware and explicit package versions, separating capability and efficiency columns.",
  "runtime": "MLX",
  "runtime_topics": [
    "MLX"
  ],
  "hardware_topics": [
    "apple"
  ],
  "platform_topics": [
    "mac"
  ],
  "model_label": "Qwen3-4B / Qwen3-30B-A3B Instruct-2507",
  "model_family": "Qwen",
  "model_id": null,
  "model_release_date": null,
  "model_artifact": null,
  "evidence_state": "source_reported",
  "site_reproduced": false,
  "record_type": "curated_reference",
  "performance_ranking_allowed": false,
  "metrics": null,
  "metrics_status": "site_metrics_unmeasured; attributed_source_figures_in_editorial.reported_results",
  "conditions": {
    "model_revision": null,
    "weight_hash": null,
    "runtime_execution_revision": null,
    "input_tokens": null,
    "output_tokens": null,
    "concurrency": null,
    "cache_state": null,
    "quantization": null,
    "speculative_decoding": null,
    "temperature": null,
    "top_p": null
  },
  "reported_setup": {
    "system": "Apple M4 Max / 64 GB unified memory",
    "node_count": 1,
    "parallelism": null,
    "kv_dtype": null,
    "context_limit_tokens": null,
    "experiment_date": null
  },
  "test_profiles": [
    {
      "label": "Efficiency benchmark",
      "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md",
      "conditions": {
        "model_revision": null,
        "weight_hash": null,
        "runtime_execution_revision": null,
        "input_tokens": 2048,
        "output_tokens": 128,
        "concurrency": null,
        "cache_state": null,
        "quantization": "BF16 / q8 / q6 / q5 / q4, per model row",
        "speculative_decoding": null,
        "temperature": null,
        "top_p": null,
        "runtime_version": "mlx 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2 / macOS 26.1"
      }
    }
  ],
  "deployment_options": [],
  "recipe_parameters": null,
  "measurement_scope": "inference_speed",
  "notes": [
    "The efficiency command specifies 2048 prompt and 128 generated tokens.",
    "Per-model precision rows retain separate memory and quality measurements."
  ],
  "limitations": [
    "This dated package stack is not a measurement of current MLX or Qwen3.8.",
    "Missing, failed and untested table cells do not mean zero performance."
  ],
  "comparison_status": "not_comparable",
  "editorial_status": "source_derived_detailed_article",
  "public_benchmark_snapshot": false,
  "source": {
    "publisher": "ml-explore",
    "url": "https://github.com/ml-explore/mlx-lm",
    "repository": "ml-explore/mlx-lm",
    "platform": "github",
    "revision": "93e6d4a6447b3080861985947f0820e76fcb957d",
    "document_path": "mlx_lm/BENCHMARKS.md",
    "pinned_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md",
    "report_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md",
    "document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218",
    "last_commit_at": "2026-10-16T02:48:11Z",
    "checked_on": "2026-10-17",
    "metadata_status": "checked"
  },
  "license": {
    "declared_spdx": "MIT",
    "scope": "Repository metadata only; per-file, model, data and conversion terms remain separate."
  },
  "editorial": {
    "version": 1,
    "published_on": "2026-10-17",
    "modified_on": "2026-10-17",
    "byline": "Run AI Here editorial",
    "creation_method": "AI-assisted source reading and original editorial synthesis; no independent model execution.",
    "answer": "The efficiency command uses 2,048 prompt tokens and 128 generated tokens on an M4 Max with 64 GB unified memory. It records a development MLX build, mlx-lm 0.28.2 and macOS 26.1. Its separate evaluation command runs MMLU Pro. Timing and quality therefore come from different programs, even when printed on the same model row.",
    "sections": [
      {
        "heading": "What does the maintainer document fix?",
        "paragraphs": [
          "The efficiency command uses 2,048 prompt tokens and 128 generated tokens on an M4 Max with 64 GB unified memory. It records a development MLX build, mlx-lm 0.28.2 and macOS 26.1. Its separate evaluation command runs MMLU Pro. Timing and quality therefore come from different programs, even when printed on the same model row."
        ],
        "citations": [
          "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L14"
        ]
      },
      {
        "heading": "What is the observed precision tradeoff?",
        "paragraphs": [
          "For Qwen3-4B Instruct-2507, the selected BF16 and q4 rows report 52.47 and 134.52 generation tokens/s. The corresponding MMLU Pro values are 64.05 and 60.72. Faster decoding does not establish unchanged capability, and a task score cannot be interpreted as a decode rate. Group-size-specific q4 and q5 rows are separate artifacts."
        ],
        "citations": [
          "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L29"
        ]
      },
      {
        "heading": "How should missing cells be represented?",
        "paragraphs": [
          "The 30B-A3B BF16 row contains unavailable efficiency cells while retaining a quality value. Those cells remain absent rather than being converted to zero or to a fabricated failure diagnosis. The memory column is labeled GB by the source and is not silently relabeled GiB. Repetitions and exact weight revisions remain unspecified here."
        ],
        "citations": [
          "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L51"
        ]
      },
      {
        "heading": "What is the appropriate use of this reference?",
        "paragraphs": [
          "Use it as a dated maintainer example of a performance/quality tradeoff on an identified stack. It is not a benchmark of current MLX, Qwen3.8 or a different Apple chip. Preserve the conversion repository, precision/group size, model revision and command parameters before comparing another measurement."
        ],
        "citations": [
          "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L23"
        ]
      }
    ],
    "facts": [],
    "reported_results": [
      {
        "label": "Qwen3-4B Instruct-2507 BF16",
        "metric": "generation",
        "value": 52.47,
        "unit": "tokens/s",
        "statistic": "reported value",
        "uncertainty": null,
        "conditions": {
          "system": "Apple M4 Max / 64 GB unified memory",
          "node_count": 1,
          "parallelism": null,
          "kv_dtype": null,
          "context_limit_tokens": null,
          "experiment_date": null,
          "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2",
          "os": "macOS 26.1",
          "input_tokens": 2048,
          "output_tokens": 128,
          "repetitions": null,
          "cache_state": null,
          "concurrency": null,
          "model": "Qwen/Qwen3-4B-Instruct-2507",
          "quantization": "BF16"
        },
        "evidence_state": "source_reported",
        "site_reproduced": false,
        "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L31",
        "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
      },
      {
        "label": "Qwen3-4B Instruct-2507 q4",
        "metric": "generation",
        "value": 134.52,
        "unit": "tokens/s",
        "statistic": "reported value",
        "uncertainty": null,
        "conditions": {
          "system": "Apple M4 Max / 64 GB unified memory",
          "node_count": 1,
          "parallelism": null,
          "kv_dtype": null,
          "context_limit_tokens": null,
          "experiment_date": null,
          "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2",
          "os": "macOS 26.1",
          "input_tokens": 2048,
          "output_tokens": 128,
          "repetitions": null,
          "cache_state": null,
          "concurrency": null,
          "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
          "quantization": "q4"
        },
        "evidence_state": "source_reported",
        "site_reproduced": false,
        "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L37",
        "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
      },
      {
        "label": "Qwen3-4B Instruct-2507 BF16",
        "metric": "MMLU_Pro",
        "value": 64.05,
        "unit": "reported score",
        "statistic": "reported value",
        "uncertainty": null,
        "conditions": {
          "system": "Apple M4 Max / 64 GB unified memory",
          "node_count": 1,
          "parallelism": null,
          "kv_dtype": null,
          "context_limit_tokens": null,
          "experiment_date": null,
          "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2",
          "os": "macOS 26.1",
          "input_tokens": null,
          "output_tokens": null,
          "repetitions": null,
          "cache_state": null,
          "concurrency": null,
          "model": "Qwen/Qwen3-4B-Instruct-2507",
          "quantization": "BF16",
          "harness": "mlx_lm.evaluate --task mmlu_pro"
        },
        "evidence_state": "source_reported",
        "site_reproduced": false,
        "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L31",
        "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
      },
      {
        "label": "Qwen3-4B Instruct-2507 q4",
        "metric": "MMLU_Pro",
        "value": 60.72,
        "unit": "reported score",
        "statistic": "reported value",
        "uncertainty": null,
        "conditions": {
          "system": "Apple M4 Max / 64 GB unified memory",
          "node_count": 1,
          "parallelism": null,
          "kv_dtype": null,
          "context_limit_tokens": null,
          "experiment_date": null,
          "runtime": "MLX 0.29.2.dev20251008+85a8824a8 / mlx-lm 0.28.2",
          "os": "macOS 26.1",
          "input_tokens": null,
          "output_tokens": null,
          "repetitions": null,
          "cache_state": null,
          "concurrency": null,
          "model": "mlx-community/Qwen3-4B-Instruct-2507-4bit",
          "quantization": "q4",
          "harness": "mlx_lm.evaluate --task mmlu_pro"
        },
        "evidence_state": "source_reported",
        "site_reproduced": false,
        "evidence_url": "https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/BENCHMARKS.md#L37",
        "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
      }
    ],
    "source_document_sha256": "924047e9fa7e680490ceff2792c03390e7a13576a73280ef81cefa1380447218"
  }
}