Skip to content

docs(benchmarks): refresh M5 Max benchmarks for 0.4.0-rc.1 (cooldown 30) - #756

Merged
inureyes merged 1 commit into
mainfrom
docs/bench-m5max-0.4.0-rc.1
Jul 12, 2026
Merged

docs(benchmarks): refresh M5 Max benchmarks for 0.4.0-rc.1 (cooldown 30)#756
inureyes merged 1 commit into
mainfrom
docs/bench-m5max-0.4.0-rc.1

Conversation

@inureyes

Copy link
Copy Markdown
Member

Summary

Full text + VLM re-benchmark of M5 Max on mlxcel 0.4.0-rc.1 (MLX pin 57c66cac), version-change refresh from the 0.2.1 sweep. Both sweeps used --cooldown 30 --big-cooldown 30.

Why cooldowns

The 0.4.0-rc.1 model set grew ~25 heavier OCR/VLM/119B checkpoints that sort alphabetically ahead of the Qwen block. A no-cooldown bench_decode.sh all accumulates enough heat to thermally throttle the mid-sweep Qwen cluster (a fake 40-77% drop on decode and prefill together), which then recovers on the smaller models after it. With cooldowns the throttle is gone; isolated and Qwen-only-cool re-measurements match the 0.2.1 baseline within 5%, confirming no Qwen code regression. (Time Machine was also disabled during the run, since its first-backup I/O starves unified-memory bandwidth.)

Results

  • 175 text dirs (160 measured), 75 VLM rows. No decode regressions vs 0.2.1.
  • Improvements: gemma-4-31b-it-nvfp4 +119% (native NVFP4 Metal path), lfm2-8b-a1b +11%, paligemma2-3b +10%.
  • New checkpoints: mistral-small-4-119b, qwen3-next-80b, qwen3-omni, phi-3-small, gemma-2-9b, llada2.0-mini, and the OCR/VLM wave (deepseek-ocr, idefics2/3, kimi-vl, lfm2-vl, granite-vision, paddleocr-vl, smolvlm, and more).
  • Cross-hardware decode table is now all-0.4.0-rc.1: M1 Ultra and M5 Max columns refreshed from the merged 2026-07-12 sweeps (GB10 already current). M5 Max vs M1 Ultra per-model column recomputed against metal_m1ultra_2026-07-12.csv.

pixtral note (not a regression)

The sweep first showed pixtral-12b VLM decode at 25.58 tok/s, but re-measurement with a steady llava control showed it is intermittent/bistable (~20 vs ~68 tok/s across identical runs), while M1 Ultra and GB10 are flat. The representative ~68 value (matching the 0.2.1 baseline) is recorded, with a low-priority Known Issues note. Only pixtral's large ~4100-token image context shows it.

Files

  • benchmarks/metal_m5max_2026-07-12.csv, benchmarks/metal_m5max_vlm_2026-07-12.csv (source of truth)
  • docs/benchmark_results/model_tests_m5max.md, docs/benchmark_results/model_tests.md

Validation

Real-checkpoint sweeps on M5 Max hardware; thermal/contention confounds identified and controlled (cooldowns, Time Machine off, contention watch); regression candidates re-measured in isolation with controls before recording.

Full text + VLM re-benchmark on mlxcel 0.4.0-rc.1 (MLX pin 57c66cac) using
`bench_decode.sh all --cooldown 30 --big-cooldown 30`. Cooldowns are required
on this host at 0.4.0-rc.1: the model set grew ~25 heavier OCR/VLM/119B
checkpoints that sort ahead of the Qwen block, so a no-cooldown sweep
thermally throttles the mid-sweep Qwen cluster (a fake 40-77% drop on decode
and prefill together that recovers on the smaller models after it). Isolated
and Qwen-only-cool runs match the 0.2.1 baseline within 5%, confirming no Qwen
code regression.

- 175 text dirs (160 measured), 75 VLM rows; no decode regressions vs 0.2.1.
- Improvements: gemma-4-31b-it-nvfp4 +119% (native NVFP4 Metal path),
  lfm2-8b-a1b +11%, paligemma2-3b +10%.
- New checkpoints added: mistral-small-4-119b, qwen3-next-80b, qwen3-omni,
  phi-3-small, gemma-2-9b, llada2.0-mini, and the OCR/VLM wave (deepseek-ocr,
  idefics2/3, kimi-vl, lfm2-vl, granite-vision, paddleocr-vl, smolvlm, ...).
- Cross-hardware decode table is now all-0.4.0-rc.1: M1 Ultra and M5 Max
  columns refreshed from the merged 2026-07-12 sweeps (GB10 already current);
  M5 Max vs M1 Ultra column recomputed against metal_m1ultra_2026-07-12.csv.
- pixtral M5 VLM decode is intermittent (~20 vs ~68 tok/s, llava control
  steady, M1 Ultra/GB10 flat); recorded the representative ~68 with a
  low-priority Known Issues note, not a regression.
@inureyes
inureyes merged commit 5a9ea36 into main Jul 12, 2026
5 checks passed
@inureyes
inureyes deleted the docs/bench-m5max-0.4.0-rc.1 branch July 12, 2026 10:43
inureyes added a commit that referenced this pull request Jul 12, 2026
Main advanced past this branch's base with the M5 Max 0.4.0-rc.1 refresh (#756), which touched the same Cross-Hardware Decode Comparison table and footnotes in docs/benchmark_results/model_tests.md that this PR's Nemotron-H-30B GB10 update also touches.

Resolved by keeping main's refreshed M1 Ultra/M5 Max columns and prose, and layering this PR's GB10 Nemotron-H-30B value (87.41, post-reboot single per #755) and its footnote/harness-note update on top. docs/benchmark_results/model_tests_gb10.md and the 13 new benchmark CSVs from this branch are untouched by the merge.
@inureyes inureyes self-assigned this Aug 31, 2026
@inureyes inureyes added status:done Completed type:docs Documentation improvements or additions labels Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:done Completed type:docs Documentation improvements or additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant