perf(cuda): single-dtype decode graph to eliminate per-token AsType - #732
Merged
Conversation
Add the AsType-per-token counter that is the "done" metric for the CUDA single-dtype decode graph (issue #636), inventory the remaining conversion sources, and remove the one reducible source on the sampling path. The three inventory models already emit 0 AsType per greedy decode step on CUDA: the merged patches-cuda/dtype.cpp bf16 promotion patch (bf16 + fp32 -> bf16) collapses the diffuse scalar/constant promotions that the moe-decode-gap investigation counted at ~773/token on Apple/Metal. The counter makes that state measurable and guards it against regression. Tooling: new C++ traversal count_astype_nodes_pair / astype_breakdown_pair walk the unevaluated (token, logprob) decode graph and count AsType nodes (and a per src->dst dtype breakdown), exposed through the cxx bridge and wired into the decode loop behind MLXCEL_TRACE_ASTYPE. Traversal only, no extra eval, and entirely skipped when the env var is unset. Cross-checked against MLXCEL_EXPORT_DECODE_DOT (grep -c AsType), which agrees at 0. Sampler hygiene: the fused sampler built its temperature, top-k/top-p/min-p sentinel, and threshold scalars as f32, which the CUDA bf16 promotion patch then cast to the logit dtype every decode step. Building each scalar in the logit dtype keeps the chain single-dtype: bit-identical for bf16 logits (the old path cast the scalar to bf16 anyway) and avoids upcasting the whole vocab tensor to f32 for f16 logits. Temperature-sampling AsType drops from 4 to 1 (the residual u32 -> f32 is intrinsic to random::categorical). Greedy uses the untouched argmax path, so the three inventory models stay at 0. Load-time normalization: MLXCEL_CUDA_F16_NORMALIZE adds an opt-in CUDA-only bf16 -> f16 cast of non-quantized weights for fixed-topology / CUDA-graph-reuse experiments, with a conservative f16-fragile exception list (gemma, cohere/command-r, apertus, gpt-oss, and any softcap or logit_scale config) that keeps bf16 regardless. Off by default because the CUDA decode graph is already single-dtype in bf16 and CUDA has native bf16 ALUs, so f16 gives no measured throughput gain (qwen2.5-0.5b-bf16 decode 208 vs 207 tok/s) while narrowing dynamic range. Quantized checkpoints are unaffected, so leaving it unset keeps every bf16 model available. Metal/Apple Silicon numerics are untouched: the always-on Apple bf16 -> f16 policy is unchanged and the new path is gated on the CUDA backend. Validated on GB10 (SM 12.1): three inventory models at 0 AsType greedy before and after; 40-token greedy parity byte-identical for llama-3.1-8b-4bit, qwen3-30b-a3b-4bit (env no-op) and llama-3.1-8b-bf16 (bf16 vs f16); env-on/off decode A/B within noise on qwen3-30b-a3b-4bit and qwen2.5-0.5b-bf16; 46 mlxcel-core sampling unit tests pass. Counts and tables in docs/benchmark_results/single-dtype-decode-astype-gb10-2026-07-10.md. Refs #636
The issue #636 sampler-chain change built the fused sampler's temperature, top-k/top-p/min-p sentinel, and threshold scalars in the logit dtype unconditionally, which altered Metal temperature-sampling numerics. On Apple Silicon bf16 weights load as f16, logits are f16, and patches-cuda/dtype.cpp deliberately leaves f16+f32 -> f32 unpatched, so main's Metal path upcasts the sampler chain to an f32 softmax via the first bare f32 scalar. Building the scalars in f16 kept the Metal chain in f16, which #636 forbids (Metal numerics must be untouched). Gate the scalar-dtype choice on a runtime `!mlx::core::metal::is_available()` flag (`single_dtype_scalars`), matching the kernel-dispatch pattern already used in this file. On Metal the scalars stay bare `array(f32)` exactly as main; on CUDA (and other non-Metal backends) they are built in the logit dtype so the bf16 promotion patch inserts no per-step AsType. `fused_sample` hoists the flag once and threads it into `top_p_filter` and the compiled min-p kernel (captured at build time, since the backend is process-constant). On CUDA this is bit-identical for bf16 logits (the old path cast the f32 scalar to bf16 anyway); a natively-f16 CUDA checkpoint now runs temperature sampling in f16, which is the intended single-dtype behavior and within fp16 tolerance for the stochastic categorical draw, now noted in the benchmark note. Verified: `cargo check --features cuda --lib --tests` clean; `cargo test --release --features cuda -p mlxcel-core --lib sampling::` 46 passed; CUDA MLXCEL_TRACE_ASTYPE still reports 1 on the qwen3-8b-4bit temp+top-p path (the intrinsic u32->f32) and 0 on greedy. Refs #636
… guard Security-review follow-ups: the softcap/logit-scale sniff in is_f16_fragile_family only inspected top-level config keys while the model_type detection already fell back to text_config, so a multimodal checkpoint carrying softcapping under text_config could slip past the guard and get f16-normalized. The key loop now applies the same text_config fallback. Also align the doc comment with actual behavior (unknown families are normalized, which is why the path is opt-in and default-off), fix a comment that claimed the metal::is_available gate already existed in this file, and add the missing cstdint include.
Add a table-driven unit test for the CUDA f16-normalization fragile-family guard (issue #636), covering a known-fragile model_type (gemma2), a top-level softcap key, the c1d1c33 text_config-nested softcap fallback that the security review flagged as a multimodal blind spot, and a plain non-fragile llama config. Also runs cargo fmt on the file, which reflowed the pre-existing FRAGILE_SUBSTRINGS const array to match rustfmt's line-width rule. Validation: - cargo check --features cuda --lib --tests -p mlxcel: clean - cargo test --release --features cuda -p mlxcel --lib is_f16_fragile_family -- --test-threads=1: 1 passed - cargo fmt --check on touched Rust files: clean Refs #636
5 tasks
This was referenced Jul 12, 2026
inureyes
added a commit
that referenced
this pull request
Jul 12, 2026
…measurement (#757) ## Summary Closes #755 with the post-reboot re-measurement it gated on, and a verdict: **none of the sweep drops is a code regression**. The gemma-4-26b pair was depressed by the stale ~5.5-day host/driver state and recovers decisively on a fresh boot; glm4-flash and mamba2-130m were never regressed, their sweep readings are single draws from a bimodal host-level throughput distribution whose fast mode reproduces the 0.3.1 records; and the same session's SSM-cluster re-verification shows the 1.4-3.4x "anomaly" is the intended #727 fused-SSM-kernel improvement, misdiagnosed as environmental in the sweep notes. Doc rows for all 13 re-measured models now carry the post-reboot values, and the 21 committed runs live in `benchmarks/cuda_gb10_2026-07-12_postreboot_single_*.csv`. Host: fresh boot 2026-07-12 19:25 (uptime 7 min at first check), mlxcel 0.4.0-rc.1 at `fcb4c20`, MLX pin `57c66cac` (0.32.1), incremental `cargo build --release --features cuda` rebuilt clean before measuring. All script runs used `--cooldown 30`. ## Measurements ### #755 subjects and controls (decode tok/s) | Model | 2026-06-17 (0.3.1) | 2026-07-12 sweep | Post-reboot | Verdict | |---|---|---|---|---| | gemma-4-26b-a4b-it-4bit | 58.59 | 50.19 | **59.88** (n=3: 59.57-60.07) | recovered, environmental | | gemma-4-26b-a4b-it-qat-4bit | 50.33 | 45.29 | **53.68** (n=3: 53.48-53.71) | recovered, environmental | | glm4-flash-4bit | 53.33 | 45.72 | **47.42** (n=3 templated: 46.42-50.07) | within noise floor (below) | | mamba2-130m | 181.05 | 162.23 | **202.98** (n=3: 147.98-204.77) | within noise floor | | qwen3-30b-a3b-4bit (control) | 90.70 | 92.41 | 94.39 | stable | | lfm2-8b-a1b-4bit (control) | 157.73 | 161.87 | 165.53 | stable | ### SSM-cluster re-verification (decode tok/s) | Model | 2026-06-17 | 2026-07-09 single | 2026-07-12 sweep | Post-reboot | |---|---|---|---|---| | granite-4.0-h-350m-4bit | 64.00 | 86.60 | 259.69 | 171.22 | | granite-4.0-h-tiny-4bit | 33.84 | - | 100.28 | 101.37 | | falcon-h1-tiny-90m-instruct-4bit | 102.99 | 110.42 | 413.00 | 354.43 | | nemotron-h-30b-4bit | 40.32 | - | 79.94 | 87.41 | | nemotron-nas-30b-4bit | 37.33 | - | 82.72 | 85.91 | | nemotron-3-nano-omni-30b (reasoning) | 38.45 | - | 80.86 | 82.88 | | plamo-2-1b | 34.36 | 35.14 | 44.54 | 46.84 | The cluster's gains persist on a fresh host, so they are release numbers, not an artifact. The attribution is a timeline fact the sweep notes missed: the low 2026-07-09 singles ran at 15:03, and #727 (fused single-token SSM decode kernel, CUDA port; its own PR measured granite-350m at 4.5x) merged the next day at 16:45 on 2026-07-10. The "no SSM-related code has landed since" premise behind the environmental explanation was simply false, and this PR corrects it on the record. ## Attribution evidence for the non-recovering pair glm4-flash decode over twelve **identical** 100-token greedy runs on the freshly booted host (same binary, same prompt, `--no-chat-template`, identical generated tokens): 39.23, 54.80, 42.33, 52.33, 51.48, 52.86, 40.51, 42.12, 41.28, 49.29, 52.24, 42.54. The distribution is bimodal (a ~41 mode and a ~52.5 mode), and the 0.3.1 record (53.33) sits inside the fast mode. mamba2-130m shows the same flapping (147.98, then 202.98 / 204.77 / 208.25 back-to-back, vs the 181.05 record). There is no stable deficit to attribute; a single sweep run draws one sample from this distribution. The code suspects named in the issue are each ruled out: - **#740 multirow qmv**: `MLXCEL_QMV_MULTIROW=0` reads 49.55, inside the baseline envelope; by design the path keeps `M*B == 1` classic decode on the stock kernel, so B=1 benches never exercise it. - **#727 fused SSM kernel**: `MLXCEL_SSM_CUDA_KERNEL=0` on mamba2-130m reads 208.25 (unchanged); pure mamba2 never uses the fused kernel, exactly as the #727 PR stated. - **#732 single-dtype decode graph**: shipped trace tooling (`MLXCEL_TRACE_ASTYPE`) plus a default-off opt-in normalization (`MLXCEL_CUDA_F16_NORMALIZE`); nothing changes by default. The slow mode is not SM clock capping: sampling `clocks.sm` at 2 Hz during runs caught slow runs (41.28, 42.54 tok/s) at a pinned 2411-2424 MHz, the same clocks as fast runs. The CPU governor is `performance` at max frequency. The host-level mechanism behind the two modes remains unidentified, but the fast mode reproducing the 0.3.1 number rules out a code-level regression, which is what #755 needed to establish. Two further observations recorded in the docs: - The 26B gemma MoE repeats within ±0.5% while the launch-bound small models flap by up to ±25%, so the noise floor is model-class-dependent: single-run deltas on small dense/SSM checkpoints and small MoEs should not be read as regressions without repeats. - glm4-flash's templated greedy output shortened from 100 tokens (0.3.1) to 18 (rc.1), a different greedy continuation rather than a failure, so its sweep-to-sweep decode averages additionally stopped being length-comparable. ## Changes - `benchmarks/cuda_gb10_2026-07-12_postreboot_single_*.csv` (13 files, 21 runs): the post-reboot singles, including the n=3 repeat sets for both gemma variants, glm4-flash, and mamba2-130m. Env-modified A/B runs are excluded from the CSVs and reported here instead. - `docs/benchmark_results/model_tests_gb10.md`: the 13 model rows now carry the post-reboot values (medians where n=3); the SSM-cluster notable-changes bullet is rewritten with the #727 attribution and the correction of the earlier environmental claim; the moderate-drops bullet is rewritten as resolved with the evidence above and the noise-floor guidance. - `docs/benchmark_results/model_tests.md`: Nemotron-H-30B cross-hardware row updated to 87.41 and its footnote now carries the #727/#755 attribution instead of the re-verify caveat. ## Acceptance criteria (from #755) - [x] Post-reboot single-model measurements recorded for the three subjects, mamba2-130m, and both controls (singles CSVs committed; numbers above) - [x] Verdict recorded: environmental for the gemma pair; no-regression (host run-to-run variance) for glm4-flash and mamba2-130m; the SSM-cluster side settled as a real #727 improvement - [x] Doc rows and the unattributed-drop notes updated to the post-reboot values, with this issue referenced - [x] Kill-switch A/B and clock-sampling evidence recorded for the persisting readings (no code fix needed; nothing to fix) - [x] The notes in `docs/benchmark_results/model_tests_gb10.md` reference #755 ## Test plan - [x] Incremental `cargo build --release --features cuda` clean before measuring (no source changes in this PR; docs and benchmark data only) - [x] 13 post-reboot singles + 8 repeat/variance runs + 3 kill-switch A/B runs + 6 clock-sampled runs on GB10 - [x] Doc tables cross-checked against the committed CSV rows
inureyes
added a commit
that referenced
this pull request
Sep 5, 2026
Gemma 4 reached the load-time dtype policy through no call site at all. Its four load paths, text, VLM, unified, and tensor-parallel, all bypass `load_text_weights` and `finish_vlm_weights_common`, because MLX's native `load_safetensors` crashes on Gemma 4 shards and the family needs the selective mmap loader. On pre-Ampere CUDA that meant the whole family ran bf16 and paid the full emulation penalty, and no environment variable could change it: `MLXCEL_CUDA_F16_NORMALIZE` in either direction did nothing, because nothing on that path read it. The symptom was easy to misread, and was misread. Perplexity came back bit-identical between the f16 and bf16 arms, which looks exactly like the f16-fragile list correctly declining to convert a family it excludes, and Gemma is on that list twice over. Two hypotheses fit one observation: a policy that declines, and a policy nobody consults. Forcing conversion past the fragile list separates them, and all three arms stayed bit-identical, which only the second explains. Locating the seam took three wrong guesses because `load_gemma4_text_weights_with_backing` carried its own copy of `load_gemma4_family_weights_with_backing`'s body, differing only in the `keep` filter, while the VLM and unified variants delegated. The policy landed in the copy the 26B does not use, and each function reads as correct on its own. The copy is deleted; the policy now lives once, in the function every Gemma 4 variant shares. Measured on a Tesla V100 with gemma-4-26b-a4b-it-4bit: 1370 bf16 tensors convert, decode goes from 6.12 to 9.79 tok/s, and no non-finite value appears at any perplexity window. Output quality was checked directly rather than inferred from a metric: across four prompts at 200 greedy tokens the arms diverge in wording, as two different numeric types must eventually, while both stay correct, including the arithmetic prompt where both derive 396. `MLXCEL_CUDA_F16_FRAGILE` is added as an opt-in escape for pre-Ampere users who want that speed on a family the fragile list excludes. The list's default is unchanged: it is inherited from #732, which adopted it for Ampere and later where f16 offered, in that commit's words, "no AsType reduction and no throughput gain", and whose test plan ran no fragile family in f16. Below sm_80 the same exclusion is not free, but flipping its default needs evidence this change does not yet have.
inureyes
added a commit
that referenced
this pull request
Sep 5, 2026
The f16-fragile list keeps a family on bf16 even when normalization is on. It arrived in #732 for Ampere and later, where that commit records f16 as yielding "no AsType reduction and no throughput gain", and where the list is described in its own words as conservative. With no upside, excluding a family on a suspicion costs nothing, and that PR's test plan ran no fragile family in f16 at all. Below sm_80 the same exclusion costs 1.60x on decode, so it has to be earned rather than inherited. Measured on a Tesla V100 with `gemma-4-26b-a4b-it-4bit`. Converting its 1370 bf16 tensors takes decode from 6.12 to 9.79 tok/s, no window of a perplexity sweep produces a non-finite value, and output quality holds across four prompts at 200 greedy tokens: identical Python, equally accurate history, and both arms deriving 396 for `17 * 23 + 45 / 9`. The mechanism agrees with the measurement. Gemma's trigger is softcapping, and `tanh(x / c) * c` bounds a value rather than growing it, which is the opposite of what exhausts an f16 exponent. Cohere/Command-R, Apertus and gpt-oss stay fragile at every architecture. The argument that acquits Gemma does not transfer to them: Apertus squares through xIELU and gpt-oss carries a wide dynamic range, and neither has been measured here. `MLXCEL_CUDA_F16_FRAGILE` remains for whoever wants to take that measurement. Ampere and later are untouched. `is_f16_fragile_family` keeps its full list there, because the trade that makes the exemption worth taking below sm_80 does not exist above it. The exemption also has to precede both of Gemma's triggers, the `"gemma"` model_type substring and `final_logit_softcapping`, and has to look under `text_config`, since a multimodal Gemma carries `gemma4` at the top level and `gemma4_text` beneath it.
inureyes
added a commit
that referenced
this pull request
Sep 5, 2026
…below sm_80 Every family on the f16-fragile list has now been measured on a Tesla V100 against wikitext-2, f16 against bf16, and only one of them breaks. Below sm_80 the list becomes a positive list of what failed rather than the sm_80+ list minus exemptions, because there is no longer an unmeasured entry for the subtractive form to protect. Apertus stays. Its xIELU squares its input, and f16 produces NaN at every one of 508 scored positions from the first token while bf16 scores 24.27 with none. This is the entry the list was right about, and the mechanism is the reason: squaring grows a value past f16's 65504 into inf, and a subtraction of infinities is NaN. Gemma, gpt-oss and Cohere leave. Gemma moves perplexity by +0.09% over 10208 tokens with no non-finite value, takes decode from 6.12 to 9.79 tok/s, and holds output across four prompts at 200 greedy tokens. gpt-oss disagrees in sign between window sizes, +3.92% at 128 and -0.95% at 512, for a token-weighted -0.02%; an earlier 508-token sample had shown +3.83% and looked systematic, and it was one short window. Cohere needs no conversion to answer the question: `c4ai-command-r7b-12-2024-4bit` ships 483 F16 tensors and zero bf16, so it already executes in f16 on every backend while the list calls f16 unsafe for it, and it scores 20.25 and 13.15 at the two windows. The generic softcap and `logit_scale` config triggers are dropped below sm_80 as well. Both families carrying them measured clean, and a cap bounds the value it is applied to, which is the opposite of what exhausts an f16 exponent. Ampere and later keep the full list unchanged. The trade that makes this worth taking below sm_80 does not exist above it, where #732 measured f16 as offering no throughput gain at all. A family this list does not know, whose activations genuinely grow, would now convert and could reproduce the Apertus failure. `MLXCEL_CUDA_F16_NORMALIZE=0` is the way out, and the perplexity harness fails on a single non-finite value so that such a family is caught before it ships rather than after. That check is what surfaced Apertus: its perplexity came back as 1.0000, a meaningless number produced by every log-probability being NaN, and only the non-finite count showed the real state.
inureyes
added a commit
that referenced
this pull request
Sep 5, 2026
…1656) * fix(bench): detect the host and inline media in bench_embeddings The harness hardcoded `hardware` to `NVIDIA_GB10_122GB`, so a CSV produced anywhere else silently carried the GB10 label and could not be trusted beside the GB10 ladder it exists to sit next to. It now detects the host the way `bench_decode.sh` does: `Apple_M5_Max_128GB` here, an `nvidia-smi`-derived string on CUDA. Image cells sent the fixture as an absolute `file://` URL and every one returned HTTP 400 once #1481 confined request media. `resolve_media_file_in` strips the leading separator and joins onto the media root, matching llama-server b10621, so an absolute path is probed under the root and reported missing while a relative one resolves. The fixture is now inlined as a base64 `data:` URI, which needs no `--media-path` and keeps the ladder runnable on a host where nobody set one. Visual token counts match the GB10 pass exactly: 71 on Qwen3-VL-Embedding, 264 on Llama-Nemotron-VL-Embed. Refs #1612 * docs(benchmarks): refresh M5 Max results for 0.6.0 Full text and VLM sweep on 0.6.0 with `--cooldown 30 --big-cooldown 30`, plus speculative, batched-serving, and embedding/rerank passes. 175 text checkpoints walked, 161 measured, and no decode regression across the 159 comparable with 0.4.0-rc.1. The sweep now runs under a 90 GB weight budget (`BENCH_MEM_OVERHEAD_FACTOR=1.209`), so `deepseek-v3-4bit` at 99.96 GiB records `SKIP:oom_estimate` instead of being launched and recorded as a failure. Two fixes that landed after 0.4.0-rc.1 changed the measured condition rather than the speed, and the docs now say so before any delta is read: #792 stopped upscaling every image to a fixed square, and the chat template now renders Llama's official prompt at 42 tokens against 98 before, matching the canonical tokenization exactly. Both shorten the prompt, and `prefill_tok_s` is prompt-length sensitive, which accounts for 10 of the 13 double-digit prefill drops. Three stale claims are corrected: the GLM-5 pair is an interrupted local download rather than a load-path defect, `minicpm-v-4.6-mxfp4` passes now, and `deepseek-v3-4bit` is a capacity exclusion. The 12 duplicate checkpoint directories are recorded so they stop inflating the model counts. `docs/benchmarks.md` command examples now match the actual script interfaces; `bench_decode.sh` takes a positional path and has no `-m` or `--runs`. Refs #1612, #1613, #1614, #1615 * feat(bench): record the 8-char source revision in every CSV `mlx_version` reads `Cargo.toml`, which does not move between releases, so every sweep taken across a development cycle records the same version however far `main` has travelled. A cross-hardware table assembled over weeks then cannot tell a hardware difference from a code difference, which is exactly the shape this branch is for: it collects the other hosts' sweeps, taken at different times, while being rebased forward. `bench_decode.sh` now writes an 8-character `commit` column, with `-dirty` appended when tracked files were modified. Untracked files are excluded, since a stray note does not change the measured binary. The column is appended last so positional readers of column 11 (`mlx_version`) keep working, and it sits before any trailing `SKIP:` or `FAIL:` token so classification still reads the last field. `bench_embeddings.py` pins its existing commit capture to 8 characters to match. The five CSVs from this sweep are backfilled with `b2ff1eee`, the commit the embeddings CSV had already recorded on its own. `docs/benchmarks.md` states the rule: check that hosts share a commit before attributing a gap to hardware, and say so where they do not. * docs(benchmarks): tie the M5 Max sweep to v0.7.0-beta.1 The branch now carries `v0.7.0-beta.1` while the CSVs record `mlx_version` 0.6.0, which reads as stale data unless the relationship is written down. The sweep ran at `b2ff1eee`, when `Cargo.toml` still said 0.6.0, and the `commit` column records that. The tag (`64f5d9b4`) is exactly one commit later and touches no compiled source: the diff is manifests, `Cargo.lock`, `CHANGELOG.md`, `CITATION.cff`, `README.md`, `docs/environment-variables.md`, the recipes registry, and an issue template. `git diff --name-only b2ff1eee v0.7.0-beta.1 | grep -E '\.(rs|cpp|metal|cu|h|hpp)$'` returns nothing, so these are the beta's numbers rather than a release behind. Sweeps taken from here record `0.7.0-beta.1` in `mlx_version`, which leaves the `commit` column as the discriminator within the cycle. * refactor(bench): split the overloaded version column by meaning One CSV column named `mlx_version` carried three different things depending on which script wrote it: the mlxcel crate version from `bench_decode.sh` and `bench_embeddings.py`, an actual MLX release in files written before the value stopped being hardcoded on 2026-06-12, and `mlx-lm-<v>` or `mlx-vlm-<v>` from `bench_mlxlm.py`. The command doc had to document the mismatch as a quirk to preserve. Columns are now named for what they hold: `mlxcel_version` (88 files), `mlx_version` kept only where it really is an MLX release (15), and `baseline_version` for the Python baselines (26). Every header was decided per file from its own values rather than by position, and an audit confirms no column is misnamed. The five CSVs from this campaign additionally gain `mlx_commit`, the pinned MLX C++ revision, because a pin bump changes kernels without moving either mlxcel field; `commit` becomes `mlxcel_commit` for symmetry. Those five are relabelled `0.7.0-beta.1`. They ran at `b2ff1eee` when `Cargo.toml` still said 0.6.0, but the tag is one commit later and touches no compiled source, and `mlxcel_commit` pins the revision regardless. `mlxcel_version` stays at column 11, so the positional readers keep working. * chore(bench): stop tracking per-run embedding server logs `scripts/bench_embeddings.py` writes one server log per checkpoint to its default `--logdir`, `benchmarks/perf_logs/`. Those are transient run artifacts rather than results, they embed absolute local paths, and the GB10 embedding pass did not commit them either. An overly broad `git add -A` swept 18 of them in. Untracked and added to `.gitignore` so the harness's default output cannot be committed again. * fix: dedup checkpoint aliases in bench_decode.sh all mode `bench_decode.sh all` enumerated every directory under `MODELS_DIR` with no notion of checkpoint identity, so a checkpoint present under two directory names (a re-download alias, or a symlink into a shared model store) was loaded, prefilled, decoded, and cooled down twice and landed in the CSV as two rows. On the M1 Ultra host `models/mlx` holds 15 such duplicate groups covering 16 redundant directories and 124.5 GB of redundant weights, present in every committed sweep back to `metal_m5max_2026-04-04.csv` and reaching `docs/benchmark_results/model_tests_m1ultra.md` and `docs/benchmark_results/model_tests_m5max.md` as separate table rows for three of the groups. `all` mode now computes a cheap checkpoint identity key before the first model runs: sha256(config.json) plus the sorted (basename, byte size) list of every *.safetensors shard, with no weight hashing. This does not collapse checkpoints that differ only in quantization or weight dtype, since their config.json differs in the quantization block (verified against the bitnet-b1.58-2b-4t / bitnet-b1.58-2b-4t-4bit pair). Directories that resolve via realpath to the same physical path are also collapsed, independent of content. A directory with no config.json is never grouped by content and is always measured. The survivor in a group is the first directory the sweep's own glob enumeration reaches; every other member is skipped and recorded as its own CSV row with the trailing status SKIP:duplicate_of=<survivor-name>, so total row count is unchanged. The collapsed groups print to stderr once before the first measurement. --no-dedup restores the prior behavior exactly, and single-model mode is untouched. Verified the grouping helper against MODELS_DIR=models/mlx: it found the same 15 groups and 16 redundant directories the issue's evidence table lists, byte-for-byte. Built a temporary MODELS_DIR of symlinks covering two small duplicate groups (gemma-3-1b-it-4bit/gemma3-1b-4bit and qwen2-0.5b/qwen2.5-0.5b-4bit) plus one control model and ran a real `all` sweep both with and without --no-dedup: the dedup run measured one row per group plus the two SKIP:duplicate_of alias rows, and --no-dedup measured all five directories individually. Ran bash -n and shellcheck on the script; no cargo command was run since nothing under src/ changed. Refs #1615 * docs(benchmarks): update the media-path note for the #1612 fix The "Why the image cells use a data URI" section described a rule that PR #1620 changes. It said an absolute `file://` URL can never resolve under `--media-path`, which was true when this pass ran and is no longer true: the concatenation stays primary, and an absolute reference whose concatenated candidate does not resolve is now canonicalized and put through the same containment check, so a file inside the root is read and one outside it is refused as an escape rather than as a missing file. The observations from the run are kept in the past tense, since they record what the M5 Max pass actually saw. The reason the harness still sends a base64 `data:` URI survives the fix unchanged: a data URI needs no server flag, which keeps the ladder reproducible on a host where the operator never set `--media-path`. The stale docstring on `_image_data_uri()` in `scripts/bench_embeddings.py` is named as a residue rather than fixed here, because that file is one of the many this PR already rewrites and the docstring is still right about why the harness uses a data URI. It changes no measurement. Refs #1612 * docs(bench): refresh the file:// notes in bench_embeddings.py for #1612 The `_image_data_uri` docstring and the `--media-path` comment both described the pre-#1612 server, where only a relative `file://` URL resolved and an absolute one was concatenated onto the root and reported missing. PR #1620 kept that concatenation as the primary resolution and added a fallback that canonicalizes an absolute reference and puts it through the same containment check, so both spellings resolve now. Comments only. The harness still sends a base64 `data:` URI, for the reason it always did: that needs no server flag, so the ladder stays reproducible on a host where the operator never set `--media-path`. Refs #1612 * docs(bench): record the M1 Ultra speculative sweep for #1613 `speculative_bench` drove only a Gemma 4 Unified target until #1613 replaced that match with per-variant adapter selection and added the Qwen 3.8 27B pairings to `REACHABLE_PAIRINGS`. This records the first sweep run against that binary. Measured on the Mac Studio M1 Ultra 128 GB (Metal), mlxcel 0.7.0-beta.1 at `a74348f8`, MLX pin `9a795735`, `speculative_bench --sweep --batch 1 --max-tokens 128`: 16 rows, 4 baselines, 9 measured MTP rows at K=2/4/8 across Gemma 4 31B, Gemma 4 Unified 12B and Qwen 3.8 27B, and the 3 DFlash rows still deferred on their own blocker. Every MTP pairing reads below 1.00x on this host and acceptance is not the reason: Gemma 4 Unified 12B accepts more here than on M5 Max at the same K (39.6% and mean accepted length 1.19 at K=4 against 35.0% and 1.05) and still lands at 0.74x where M5 Max reads 1.57x. The verify round is the difference, at 2.70 classic decode steps against 1.27, which is the reading already recorded in `speculative-decoding-m1ultra-2026-08-19.md` and the basis of the static gate that declines B=1 MTP on Apple GPU generation 13. K=8 splits by drafter family. The Gemma 4 assistants clamp to their configured block size of 4, so acceptance and mean accepted length are identical at K=4 and K=8. The `qwen3_5_mtp` head honors the wider request instead, proposing 7 per round against 3 while accepting the same 1.51, which drops acceptance to 21.6% and throughput to 9.3 tok/s. The M5 Max matrix keeps its own numbers. Its `unsupported target` cell and the two paragraphs explaining it now say the harness restriction was lifted and that an M5 Max re-run against a binary carrying the fix has not been made, rather than describing a limitation that no longer exists. The CSV index row and the overall-status row are updated for the same reason, and the M1 Ultra reachable-pairings table keeps its older reading with a pointer to the new matrix. Refs #1613 * docs(bench): record the M1 Ultra MoE batched-decode attribution for #1616 Issue #1616 read the M5 Max batched-serving rows as the MoE decode path declining the fused kernel at B>=2 and falling back to `gather_qmm`. Profiling on M1 Ultra found a different cause. `Qwen3MoeModel` never overrode `forward_batched`, so the server's batched decode ran the `LanguageModel` default: one single-sequence `forward` per row, evaluated together. Each of those rows is a single routed token, so the fused kernel was reached on every one of them and the token-count gate was never what stood in the way. The attribution document records the op-level measurements that decided what to build. Per-token fused launches scale linearly (3.9, 7.5, 14.7, 29.2 ms per 48 layers at n=1/2/4/8); identical and disjoint expert sets cost the same at every n, so expert-plane traffic is not the limiter and deduplicating expert ids across the cohort buys nothing; and a prototype of the batched fused kernel the issue proposed, bit-identical to per-token launches, loses to `gather_qmm` from n=4 (11.1 against 9.3 to 9.7 ms). That kernel was therefore not built, which is the outcome the issue's acceptance criteria allow when the profile does not support it. The M1 Ultra ladder is in `benchmarks/metal_m1ultra_batch_2026-09-04.csv`, whose `mlxcel_commit` column separates the two passes: `bf1cdb72` is main before the change and `4aee0fb9` is the code PR. The after pass was re-measured on the final commit after a `cargo build --release` showed an earlier pass had been taken from a binary predating the last edit to the branch; the conclusion is unchanged and the numbers move by about 1%. The dense-versus-MoE contrast this issue is named after is much narrower on this host than on M5 Max. There dense reads 3.17x and 3.25x against the MoE model's 1.55x; here dense reads 1.77x and 2.09x against 1.86x after the change and 1.72x before it. The M5 Max section is left as measured with a pointer to this work, because re-running that ladder is what would show the change's effect there. Refs #1616 * docs(bench): correct the stated dedup survivor rule The script header comment said the survivor within a duplicate group is "the first name in sort order". The code picks the first directory the sweep's own glob reaches, and that glob yields paths with a trailing slash, so where one name is a prefix of another the longer name wins: `pixtral-12b-4bit` survives over `pixtral-12b`, not the other way round. Six of the fifteen groups on the M1 Ultra store are prefix pairs, so the stated rule predicted the wrong survivor in 40% of cases. `docs/benchmarks.md` already described the behavior correctly but did not say what the enumeration order implies for prefix pairs, which is the only case where the two readings diverge. Both now state it and name an example. Comments and prose only; no behavior change. Refs #1615 * perf(cuda): convert bf16 weights to f16 at load on pre-Ampere GPUs Volta (sm_70) and Turing (sm_75) have no bf16 ALU, no bf16 tensor-core MMA atom, and no cuBLAS bf16 GEMM path, so every bf16 operand is converted before it can be executed. Measured on a Tesla V100: dense f32 reaches 14.0 TFLOPS and f16 reaches 95 to 100, while bf16 manages 9.6. Converting the weights once at load time therefore buys back an order of magnitude that per-operation conversion cannot. On qwen3.8-27B-4bit this takes prefill from 85.3 s to 4.8 s and decode from 118.1 to 49.2 ms per token. The mechanism is `qmm_naive`, which runs an identical 994 launches in both dtypes and spends 199.1 s against 11.4 s; the dispatch does not change, only the operand cost inside the inner loop. Decode moves only on quantized checkpoints, because a dense model's batch-1 decode is bandwidth-bound and the two dtypes occupy the same bytes. The policy itself now lives in one function, `bf16_to_f16_at_load`. It had been reimplemented at four sites that had drifted apart: the text loader carried the opt-in from issue #636, while the VLM and embedding loaders kept every family on bf16 regardless, and the selective-materialization arm picked its leaf dtype from an Apple-Silicon-only predicate. Consolidating them is what makes the policy apply at all rather than to one path in three. Each site now names the others in a comment, because the drift is invisible from any one call site. Apple Silicon behavior is unchanged, and Ampere and later keep the opt-in from #636. `MLXCEL_CUDA_F16_NORMALIZE=0` opts out on pre-Ampere, and the f16-fragile family list is still honored. The conversion message no longer claims to be an Apple Silicon optimization when it fires on a CUDA device. Verified on sm_70 only: prefill and decode on a quantized VLM and a dense text model, greedy answer quality in both dtypes, and embedding cosine matrices on two checkpoints where the paraphrase-versus-unrelated separation holds and the largest dtype disagreement is 0.0020. No Ampere-or-later device is present on this host, so the sm_80+ non-regression criteria are not verified here. * refactor(cuda): delegate the load-time dtype policy instead of restating it `bf16_to_f16_at_load` was introduced to stop the pre-Ampere rule from being reimplemented per load path, and then restated the predicates of `should_convert_bf16_to_f16` and `cuda_f16_normalize_for_config` inline rather than calling them, leaving both dead. Clippy caught it under `-D warnings`. It now delegates to each, so there is one copy of the CUDA rule and one of the Apple Silicon rule. The selective-materialization arm in `tensor_view_to_array` goes back to the Apple Silicon predicate it used before. Switching it to the shared policy was wrong: that function has no `config` in scope, so it cannot honor the f16-fragile family exception, and an Apertus or BitNet checkpoint reaching it on pre-Ampere would have been converted against the policy's intent. It leaves a bf16 leaf on CUDA and `convert_bf16_weights` applies the real rule a moment later with the config in hand, which costs one extra pass over the bf16 tensors and is the correct trade. The comment there says so, because the next reader will otherwise make the same substitution. The distinction that matters is not which sites touch the load-time dtype but which sites can see the config; only those can carry the policy. * test(cuda): add the pre-Ampere f16 perplexity gate from #1542 The load-time bf16 to f16 conversion trades f16's 5-bit exponent for bf16's 8-bit one in exchange for an order of magnitude of throughput on hardware with no bf16 ALU. That trade is only defensible if the range it gives up is range the model never used, and #1542 makes measuring it a blocking criterion rather than an assumption. The policy is applied once at load and cannot be toggled on a loaded model, so the two arms are two processes and the comparison is made across their printed tables. Non-finite log-likelihoods fail a single run on their own, because an f16 exponent overflow is a hard failure rather than a quality regression. Measured on a Tesla V100. On Meta-Llama-3.1-8B-Instruct-bf16, where 291 tensors convert, perplexity moves by -0.46, -0.08 and -0.06 percent at the 128, 512 and 2048 windows. On qwen3.8-27B-4bit, where 1682 convert, by +0.03 and -0.12 percent at 128 and 512. Every point is inside the 1 percent gate and no window in either arm produced a non-finite value. Evaluation time at the 27B's 512 window fell from 265.4 s to 16.1 s, which reproduces the `qmm_naive` profile's 17.5x through a code path that has nothing to do with the benchmark it is confirming. Two criteria stay unmet and are documented at the constants that encode them rather than left for a reader to rediscover. The MoE family cannot be covered here: the only local MoE checkpoint is Gemma, which the f16-fragile list excludes twice over, by family substring and by `final_logit_softcapping`, so both arms run identical bf16 and agree to four decimal places while testing nothing. The 27B's 2048 window exhausts a 32 GB card inside `evaluate_loglikelihoods`, identically in both arms, which is what establishes capacity rather than dtype as the cause. Refs #1542 * fix(cuda): apply the load-time dtype policy to the Gemma 4 family Gemma 4 reached the load-time dtype policy through no call site at all. Its four load paths, text, VLM, unified, and tensor-parallel, all bypass `load_text_weights` and `finish_vlm_weights_common`, because MLX's native `load_safetensors` crashes on Gemma 4 shards and the family needs the selective mmap loader. On pre-Ampere CUDA that meant the whole family ran bf16 and paid the full emulation penalty, and no environment variable could change it: `MLXCEL_CUDA_F16_NORMALIZE` in either direction did nothing, because nothing on that path read it. The symptom was easy to misread, and was misread. Perplexity came back bit-identical between the f16 and bf16 arms, which looks exactly like the f16-fragile list correctly declining to convert a family it excludes, and Gemma is on that list twice over. Two hypotheses fit one observation: a policy that declines, and a policy nobody consults. Forcing conversion past the fragile list separates them, and all three arms stayed bit-identical, which only the second explains. Locating the seam took three wrong guesses because `load_gemma4_text_weights_with_backing` carried its own copy of `load_gemma4_family_weights_with_backing`'s body, differing only in the `keep` filter, while the VLM and unified variants delegated. The policy landed in the copy the 26B does not use, and each function reads as correct on its own. The copy is deleted; the policy now lives once, in the function every Gemma 4 variant shares. Measured on a Tesla V100 with gemma-4-26b-a4b-it-4bit: 1370 bf16 tensors convert, decode goes from 6.12 to 9.79 tok/s, and no non-finite value appears at any perplexity window. Output quality was checked directly rather than inferred from a metric: across four prompts at 200 greedy tokens the arms diverge in wording, as two different numeric types must eventually, while both stay correct, including the arithmetic prompt where both derive 396. `MLXCEL_CUDA_F16_FRAGILE` is added as an opt-in escape for pre-Ampere users who want that speed on a family the fragile list excludes. The list's default is unchanged: it is inherited from #732, which adopted it for Ampere and later where f16 offered, in that commit's words, "no AsType reduction and no throughput gain", and whose test plan ran no fragile family in f16. Below sm_80 the same exclusion is not free, but flipping its default needs evidence this change does not yet have. * test(cuda): prefix BOS in the perplexity harness and cover Gemma Windows were scored without a leading BOS, so each one started mid-stream in a state the model was never trained to see. Every window now carries the model's BOS when it declares one, and the BOS position is excluded from the score so both arms still cover exactly `window_len - 1` corpus positions and the denominator does not move. On Llama-3.1-8B this lowers perplexity at all three windows, 17.5357 to 15.6685, 11.6873 to 11.4849 and 7.2963 to 7.2743, which is the direction more context should move it. That matters here beyond tidiness: this file exists to find f16 exponent overflow, and overflow has to be looked for in the distribution the model actually runs in. Gemma is added as its own test now that the family reaches the policy at all. It is also the case the harness handles worst: its perplexity is implausible in absolute terms and rises with window length, 5111 at 128 against 31950 at 512, where Llama falls monotonically on the same code. Both arms show it identically, so it is not a dtype effect, and the Llama control above rules out the BOS change as the cause. Something in the Gemma evaluation path is wrong and is not diagnosed here. The numbers are recorded rather than trusted, and any perplexity conclusion about Gemma has to wait for that. * fix(bench): measure every model over the same interval Decode throughput was timed over a model-dependent number of tokens: a 6-token chat prompt, up to 100 tokens, stopping at EOS. On the 2026-09-04 M1 Ultra sweep only 25% of models reached the budget and 16% stopped under 20 tokens. The bias is not one-directional, which is why it went unnoticed: re-measuring moved `granite-4.1-8b-4bit` +117% (1 token, charging first-token latency to throughput) and `gemma2-2b-4bit` -45% (18 tokens, enjoying an almost-empty KV cache). Rankings invert under it, so per-model numbers were never comparable. Both harnesses now use a deterministic 512-token synthetic prompt and exactly 128 generated tokens with end-of-generation suppressed: llama-bench's pp512/tg128, already what `scripts/bench_serving_concurrency.py` uses. `mlxcel-bench-decode` grows `--ignore-eos` as the `-inf` token bias the server uses for #1436, not a cleared stop set, because the generator merges the model's own `eos_token_ids()` with `SamplingConfig::stop_token_ids`. `bench_mlxlm.py` suppresses EOS through `logits_processors` and synthesizes its prompt from a corpus byte-identical to the Rust one, and resolves its interpreter through `MLXLM_PYTHON`, the uv-created `.venv-mlxlm`, `uv run`, then `python3`, since the system Python is 3.14 and has no mlx wheels. Two classification gaps found while testing: a directory with no `config.json` or no readable `*.safetensors` now reports `SKIP:not_a_checkpoint` or `SKIP:missing_weights` before any model load rather than the `FAIL:bench` a real defect also produces, which covered 5 of 21 failures last sweep; and the #1615 dedup grouped two weightless directories as duplicates of each other because an empty shard list compared equal, hiding both. Validated on `qwen3-0.6b-4bit`: both sides report `prompt_tokens=512`, `gen_tokens=128`, mlx-lm `finish_reason=length`. * docs(server): the thinking-token budget is not Qwen3-only `--reasoning-budget` was documented in ten places as applying to "Qwen3-family models", but `resolve_thinking_token_ids` has recognized three delimiter pairs for some time: `<think>` / `</think>`, `<|content_thinking|>` / `<|end_message|>`, and Gemma 4's `<|channel>` / `<channel|>`. A reader checking whether the flag covers their model got the wrong answer from every doc comment and from the CLI help. `startup.rs` was additionally wrong about when the budget applies, not just to what. It said a model lacking the token IDs resolves the pair to `None` and the budget is ignored, which is true and incomplete: a checkpoint can carry the tokens and still never open a block, and there the budget is a silent no-op rather than a failure. `gemma-4-26b-a4b-it-4bit` is exactly that case. It carries `<|channel>` at id 100 and `<channel|>` at 101, and emits neither: with `--show-reasoning` no channel appears at all, and `thinking_budget_tokens` of 0 against -1 produces 469 tokens against 491 with the same answer. That gap between "has the tokens" and "uses them" is what the comment now states, because reading the old one leads to reporting the flag as working when it has done nothing. * test(cuda): record why Gemma cannot be scored by perplexity here The Gemma arm added earlier is removed, along with the one-shot diagnostics used to work out why it behaved as it did, and replaced by the conclusion those diagnostics reached. `gemma-4-26b-a4b-it` scores between 255 and 88553 on this harness depending on the chunk, against Llama's 15.67 on the same corpus and the same code, with top-1 next-token accuracy of 20.5% against Llama's 59.1%. That reads as a broken evaluation path and was pursued as one through six hypotheses, all wrong: the caches are reset and chunk 0 scores identically alone or as the first of four; the sliding window is 1024 while the collapse is already total at 32; the final-logit softcap is applied; the tokenizer round-trips exactly; the logits are `[1, T, V]` with rows matching the input; and row `i` predicts token `i+1` at 20.5% against 2.3% for token `i`, so the alignment is right. The explanation is that the checkpoint cannot continue raw text. With `--no-chat-template` it answers "Its most important consequence was" with "ownce of consequence-wise-wise-wise-..." and never recovers, while Llama-3.1-8B-Instruct on the identical invocation continues correctly through Gutenberg and the 1440s, and Gemma inside its chat template answers the same question well. Wikitext perplexity measures the ability this model no longer has, so it is an inapplicable metric rather than an inaccurate one, and a dtype comparison computed inside it carries no information. This is worth a comment rather than a deletion because the failure looks so much like a bug. Two pieces of the diagnostic scaffolding are kept because they earn their place: `MLXCEL_PPL_CORPUS` scores a different file, which is what separated a model defect from a domain mismatch, and `MLXCEL_PPL_PER_CHUNK` reports each chunk, which is what disproved the cache-reuse theory by comparing the same chunk rather than two runs over different text. * perf(sampling): stop synchronizing the device on every biased decode step `apply_token_bias_stage` read the pre-bias argmax id back to the host on every decode step to feed the B9 suppression counters. That `eval` plus `item_i32` breaks the async lookahead pipeline the decode loop is built on, and it costs the same regardless of vocabulary size, which is what makes it a per-step tax rather than a per-token one. Measured on M1 Ultra as the delta between `--ignore-eos` and no flag, both at pp512/tg128: +65.4% decode time on `qwen3-0.6b-4bit`, +30.4% on `llama-3.2-1b-4bit`, +24.0% on `qwen3-30b-a3b-4bit` and +13.6% on `llama-3.1-8b-4bit`. After gating the readback behind `MLXCEL_LANG_BIAS_COUNTERS=1` the same deltas are -0.6%, -1.2%, -0.4% and -1.0%, so the cost is gone rather than moved. `LANG_BIAS_APPLIED_TOTAL` stays always-on because incrementing it is free; only the two suppression counters, which need the argmax id, became opt-in. `apply_token_bias` also no longer uploads a vocab-sized f32 vector per step to bias a handful of ids; it reads the biased positions with `take_along_axis`, adds, and writes back with `put_along_axis`, so the work is O(biased ids). That was the first hypothesis for the slowdown and was not the dominant cost, but the op is correct on its own terms and the 15 `token_bias` unit tests cover the additive semantics. Nothing is affected unless a request or server opts in: `logit_bias`, `ignore_eos`, or `--lang-bias` / `LLAMA_ARG_LANG_BIAS`. An empty bias map still returns before any graph node is built, so the default path is unchanged. Language steering was the worst case, since the map stays resident for the whole session and every token paid. Found while fixing the benchmark harness, whose new `--ignore-eos` made the instrument distort the measurement it was added to take. * docs: add CMake to the CONTRIBUTING.md minimum development environment (#1642) * docs(examples): fix dead link, placeholder, and stale in-progress claim in surgery examples (#1643) * docs: document four missing MLXCEL_* server variables in environment-variables.md (#1640) * docs(python): document the LLM/AsyncLLM constructor parameters and server_kwargs (#1646) * docs(python): link the python/examples scripts from the README and client docs (#1645) * chore(cli): port the Model store and Embeddings/Reranking help sections to mlxcel serve (#1650) * docs: update the SECURITY.md supported-versions table to the 0.7.x line (#1639) * docs: update the SECURITY.md supported-versions table to the 0.7.x line * docs: keep the SECURITY.md recommendation consistent with a beta-only line The supported-versions table now names 0.7.x, whose only shipped tag is `v0.7.0-beta.1`, so the closing sentence telling operators to run "the latest stable release" pointed at a release that does not exist on that line. It now says "latest published release" and names both the current tag and the last tag without a pre-release suffix, so an operator can tell which is which. Also squares the table's End of Support column padding, which the previous row had one character wide. Refs #1630 --------- Co-authored-by: Jeongkyu Shin <jshin@lablup.com> * docs: describe all six make verify gates in CONTRIBUTING.md (#1641) * docs: describe all six make verify gates in CONTRIBUTING.md * docs: say which command in the gate block `make verify` does not run The enumeration covers the six prerequisites, but the code block above it lists four commands, so "the last three are the fmt, clippy, and test commands above" leaves a contributor to work out on their own that `cargo deny check` is the one `make verify` skips. Naming it, and naming PR CI as what gates it instead, is the point of the issue: a first failure should be self-explanatory in both directions. Refs #1625 --------- Co-authored-by: Jeongkyu Shin <jshin@lablup.com> * docs: point --distributed-config examples at the checked-in TOML templates (#1644) * docs: point --distributed-config examples at the checked-in TOML templates * docs: keep the CLI help's write-config step off the checked-in template The `--help` blocks in `src/main.rs` and `src/bin/mlx_server.rs` are self-contained: step 1 runs `benchmark_pipeline_remote_rollout.sh write-config` and steps 2 through 4 consume what it wrote, so the runtime-generated name was correct there. Renaming its target to `pipeline_remote_2node_tcp.toml` made step 1 overwrite the checked-in template this PR introduces, which both contradicts the new `examples/distributed/README.md` and silently reverts an operator's edited addresses on the next `git checkout`. Those eight sites go back to `generated_pipeline_remote_2node_tcp.toml`, which is also the script's own default output path. The two `docs/distributed.md` sites keep the PR's change: that page shows the servers started with no generation step in front of them, which is the case issue #1633 actually reported. Adds a `.gitignore` rule for `examples/distributed/generated_*.toml` so following the help does not leave the tree dirty. The checked-in templates do not match the pattern and stay tracked. Refs #1633 --------- Co-authored-by: Jeongkyu Shin <jshin@lablup.com> * fix(server): stamp the speculative burst's first token at its prefill end so `timings` splits prompt from decode (#1609) * fix(server): stamp the burst's first token at its prefill end so timings split prompt from decode (#1592) On the speculative burst path the native `timings` block reported almost the whole request under `prompt_ms` and 0 to 1 ms under `predicted_ms`, so `predicted_per_second` read 95000 on a 96-token DFlash request. The burst runs every verify round before any token reaches the sequence, and the first-token stamp was taken when the finished token vector was replayed, after drafter load, prefill and all rounds. `BurstSuccess` now carries `prefill_end: Instant`, the moment the target prefill finished and round 0 began, in place of the two millisecond floats `finalize_burst_success` discarded. DFlash B=1 derives it from the timers it already kept, MTP B=1 from `MtpGenerator`'s own prefill timer, and the batched window from the batched prefill (DFlash stamps it after the prefill forward; MTP subtracts the round loop's decode slice), so the window stops passing `0.0, 0.0`. `finalize_burst_success` stamps the sequence with it through the new `SequenceInfo::mark_first_token_at`, which also stamps `StreamingDecodeState`, so the streamed `Prefill` snapshot and the final `prompt_eval_ms` derive from one instant and the replay's own `mark_first_token` is a no-op. The drafter's once-per-process disk load is excluded from every timings field: `WorkerDrafterSlot::ensure_loaded` returns `Some(elapsed)` when it loaded, and the drivers and the slice start shift `seq.created_at` by it. `created_at` feeds only the timing fields, so this is the issue's origin shift without a new `SequenceInfo` field. Four unit tests on a backdated sequence pin the split, the origin shift, the zero clamp, and `Ok(None)` on a resident slot. The real-drafter measurements the issue asks for were not run here: no checkpoints and no Metal toolchain on this host. Verified on a CPU-only build: `cargo fmt --all -- --check`, `cargo clippy --profile test-fast -p mlxcel --lib --tests -- -D warnings`, and `cargo test --profile test-fast -p mlxcel --lib server::batch::speculative_burst` (66 passed). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y * test(server): pin the classic path's first-token stamp too The fix changes the classic path in exactly one way: `mark_first_token()` now delegates to `mark_first_token_at(Instant::now())`, which also stamps the decode state, so the streamed `PrefillStats.prompt_ms` and the final `prompt_eval_ms` derive from one instant rather than from two calls a few dozen lines apart. The PR's four new tests all drive the supplied-instant form, so that classic-path change had no coverage: a later reordering that let `on_token` win the stamp would move classic `prompt_eval_ms` with nothing to catch it. The new test drives the classic ordering (`mark_first_token()`, a measurable gap, then `on_token`) and asserts the streamed snapshot equals the final frame, that `prompt_eval_ms` comes from the mark rather than the later token, and that the gap lands in `generation_only_ms`. Verified by reversion rather than by passing: with `decode_state.stamp_first_token(at)` removed from `mark_first_token_at`, this test and `mark_first_token_at_splits_prompt_time_from_generation_time` both fail; restored, all 67 tests in the module pass. Refs #1592 --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Jeongkyu Shin <inureyes@gmail.com> Co-authored-by: Jeongkyu Shin <jshin@lablup.com> * refactor: deduplicate the byte-size formatters (#1651) * refactor: deduplicate the byte-size formatters Reuse execution::memory_estimate::format_bytes in quant_advisor, and make KvCacheDimensions::total_size_str delegate to the file-local format_bytes in kv_cache_transfer/benchmark.rs (adopting its no-space unit spelling, which the format_bytes_ranges test already pins). * refactor: record the spacing pick and pin the delegation in the test The issue allows `total_size_str` to change spelling as long as the pick is deliberate and noted; the note was missing, so the only record that `256.0 MiB` became `256.0MiB` was a one-character edit to an assertion. The doc comment now states which spelling this is, that the inlined copy differed in two ways rather than one (it was spaced and it had no sub-KiB arm), and that reintroducing the space means moving two tests. `bench_config_total_size_str` now asserts the delegation itself, `total_size_str() == format_bytes(total_bytes())`, so re-inlining a differently spelled copy fails here rather than only in `format_bytes_ranges`, and covers the `< 1024` arm the old copy did not have: a sub-KiB total used to render as `0.0 KiB` and now renders in bytes. Refs #1626 --------- Co-authored-by: Jeongkyu Shin <jshin@lablup.com> Co-authored-by: Jeongkyu Shin <inureyes@gmail.com> * test: assert every flag named in `src/commands` exists in some `--help` (#1604) * test(cli): assert every flag named in src/commands exists in some --help (#1131) `validate_pipeline_parallel_args` once told the user to pass `--tensor-parallel-size`, a flag no binary accepts (#1112). The sweep that found it lived only in a PR description, so #1131 asked for it as a machine-checked invariant in `tests/cli_help_consistency.rs`. The new test collects the known flags from the built binaries: `--help` of every subcommand in `mlxcel --help`'s `Commands:` block, `mlxcel-server --help`, and the `--dump-flag-surface` JSON of both server entry points. The dump is what makes this work without an allowlist: the llama-server b10621 compatibility groups are `hide = true`, and with `--help` alone all 13 mentions of them in `serve.rs` (`--spec-type`, `--mmproj`, `--numa`, ...) came back as unknown. It then scans every `--flag` token in `src/commands/*.rs` other than `*_tests.rs`, comments included, and fails with one `file:line: --flag` per unknown name. Injecting the #1112 defect (one `--tp-size` in `generate.rs` rewritten to `--tensor-parallel-size`) fails the test at `src/commands/generate.rs:130`; current `main` passes clean, so no source change ships with it. Verified on a CPU-only build (no Metal toolchain on this host): `cargo fmt --all -- --check`, `cargo clippy --profile test-fast --test cli_help_consistency -- -D warnings`, and `cargo test --profile test-fast --test cli_help_consistency` (26 passed). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y * test(cli): take known flags from help signatures, not prose, and pin the scanner Review of #1604 found two holes. The known set was scanned from the whole help text, so a flag named only inside a description whitelisted itself and a wrapped description contributed fragments such as `--embedding-`; it now comes from `all_documented_spellings`, the file's existing signature-line parser. The subcommand list took the first word of every indented line under `Commands:`, so a wrapped description would have been spawned as a subcommand; it now reads the name column only. `long_flags` gets the meta-test every other parser in this file has, covering separators, `=`-joined values, a URL, an uppercase token, and a trailing bare `--`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y * test: walk src/commands recursively so a submodule cannot go unscanned The sweep used a single `read_dir`, which covers `src/commands` exactly as long as it stays flat. It is flat today (24 files, no subdirectories), so the shallow scan passes and says nothing; the day a subcommand grows a submodule directory, the flags inside it stop being checked and the test stays green while covering less than its name claims. That is the same shape of silent blindness the test exists to prevent, so it now walks the tree. The `!sources.is_empty()` assertion already guarded against the directory moving; nothing guarded against it nesting. Verified by construction rather than by passing: a throwaway `src/commands/nested_probe/probe.rs` naming `--definitely-not-a-real-flag` is reported as `src/commands/nested_probe/probe.rs:2: --definitely-not-a-real-flag` and fails the test; removed, the suite is 27 passed. Refs #1131 --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Jeongkyu Shin <inureyes@gmail.com> Co-authored-by: Jeongkyu Shin <jshin@lablup.com> * fix(server): let a malformed `MLXCEL_PAGED_SLAB_BLOCKS` fall back to the derived slab size (#1602) * fix(server): let a malformed MLXCEL_PAGED_SLAB_BLOCKS fall back to the derived slab size (#1137) `resolve_paged_slab_blocks` returned early on an unparseable value, which is the `0` pin: the pool kept the historical 32-block slab while the warning said the derived slab size was in use. A typo in the variable therefore quietly made the fused paged decode path unreachable past one slab, and the log pointed the operator away from the cause. This takes option (b) from the issue: the parse-error arm warns and falls through to the derivation, so a malformed value behaves exactly like an unset one. `0` still pins the pool default. The warning now says exactly that, in the structured form the file's other env-var warnings use, rather than promising a derived value that the geometry probe below it may still decline for a model that is not pool-eligible. A unit test pins all arms against each other: unset and malformed derive the same value, `0` returns `None`, and an explicit integer is used verbatim, including below the 32-block floor the derived path applies. The `MLXCEL_PAGED_SLAB_BLOCKS` doc row and the constant's rustdoc describe the fall-through and the verbatim rule. Verified on a CPU-only build (no Metal toolchain on this host): `cargo fmt --all -- --check`, `cargo clippy --profile test-fast -p mlxcel --all-targets -- -D warnings`, and `cargo test --profile test-fast -p mlxcel --lib memory_estimate::tests` (39 passed, also at `--test-threads=8`). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y * docs: say `0` pins the pool default in the MLXCEL_PAGED_SLAB_BLOCKS values column "Non-negative integer, used verbatim" contradicted the row's own notes: `0` is non-negative and is not used verbatim, it pins the 32-block default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Jeongkyu Shin <inureyes@gmail.com> * chore(build): surface maintenance targets in make help (#1648) ## Summary Six annotated targets (`update`, `tree`, `outdated`, `verify-versions`, `verify-kernel-dtype-keys`, `bump-version`) carried `##` help text that `make help` never printed because they matched none of the hardcoded keyword buckets; they now appear under a new `Maintenance Targets:` bucket. The bucket regex is also widened from `^[a-zA-Z_-]+:` to `^[a-zA-Z0-9_.-]+:` in every bucket so a future target name containing a digit or dot cannot silently vanish from help. Verified by replaying the help pipeline over the Makefile: the new bucket lists exactly the six targets, and the widened regex extracts the identical target set as before (no bucket regressions). ## Related issues Closes #1637 ## Type of change - [ ] `feat` — new user-visible feature - [ ] `fix` — bug fix - [ ] `perf` — performance improvement (include before/after numbers in the PR body) - [ ] `refactor` — internal restructuring without behavior change - [x] `chore` — build, CI, dependencies, release infrastructure - [ ] `docs` — documentation only - [ ] `test` — tests only * docs(cli): document the .sock listener rule (#1647) ## Summary The `--host`/`--port` help text in both binaries and two passages in `docs/python-client.md` still taught the deprecated `--port 0` Unix-socket spelling. Per `src/server/transport.rs` the rule is: a `--host` ending in `.sock` selects a Unix domain socket whatever `--port` says (the b10621 rule), `--port 0` on TCP binds an ephemeral port, and the legacy `--port 0` + path spelling emits a deprecation warning. The four flag doc-comments and both doc examples now describe the `.sock` rule. ## Related issues Closes #1631 ## Type of change - [ ] `feat` — new user-visible feature - [ ] `fix` — bug fix - [ ] `perf` — performance improvement (include before/after numbers in the PR body) - [ ] `refactor` — internal restructuring without behavior change - [ ] `chore` — build, CI, dependencies, release infrastructure - [x] `docs` — documentation only - [ ] `test` — tests only * fix(core): name --kv-bits in the bits validation error (#1652) ## Summary The `--kv-bits` negative-value validation error leaked the internal config struct name (`BatchKvQuantConfig: bits must be non-negative (got N)`). It now names the flag with the same punctuation as every sibling arm in the `match`: `--kv-bits must be non-negative, got N`. ## Related issues Closes #1629 ## Type of change - [ ] `feat` — new user-visible feature - [x] `fix` — bug fix - [ ] `perf` — performance improvement (include before/after numbers in the PR body) - [ ] `refactor` — internal restructuring without behavior change - [ ] `chore` — build, CI, dependencies, release infrastructure - [ ] `docs` — documentation only - [ ] `test` — tests only * test(cli): cover infill and batch KV arguments (#1649) ## Summary `src/cli/infill_args.rs` and `src/cli/batch_quant_args.rs` were the only modules in `src/cli/` without an inline test block. This adds them in the sibling `#[path = "..._tests.rs"]` style: parse tests for `--spm-infill` (absent → false, present → true, takes no value), and for the KV quant group a test pinning the hand-written `Default` impl to what clap produces from an empty argv — the two independent statements of the same defaults could previously fork silently — plus a flags-override test. The env-bound group takes the crate-wide env lock like its siblings. ## Related issues Closes #1634 ## Type of change - [ ] `feat` — new user-visible feature - [ ] `fix` — bug fix - [ ] `perf` — performance improvement (include before/after numbers in the PR body) - [ ] `refactor` — internal restructuring without behavior change - [ ] `chore` — build, CI, dependencies, release infrastructure - [ ] `docs` — documentation only - [x] `test` — tests only * perf(cuda): drop Gemma from the f16-fragile list below sm_80 The f16-fragile list keeps a family on bf16 even when normalization is on. It arrived in #732 for Ampere and later, where that commit records f16 as yielding "no AsType reduction and no throughput gain", and where the list is described in its own words as conservative. With no upside, excluding a family on a suspicion costs nothing, and that PR's test plan ran no fragile family in f16 at all. Below sm_80 the same exclusion costs 1.60x on decode, so it has to be earned rather than inherited. Measured on a Tesla V100 with `gemma-4-26b-a4b-it-4bit`. Converting its 1370 bf16 tensors takes decode from 6.12 to 9.79 tok/s, no window of a perplexity sweep produces a non-finite value, and output quality holds across four prompts at 200 greedy tokens: identical Python, equally accurate history, and both arms deriving 396 for `17 * 23 + 45 / 9`. The mechanism agrees with the measurement. Gemma's trigger is softcapping, and `tanh(x / c) * c` bounds a value rather than growing it, which is the opposite of what exhausts an f16 exponent. Cohere/Command-R, Apertus and gpt-oss stay fragile at every architecture. The argument that acquits Gemma does not transfer to them: Apertus squares through xIELU and gpt-oss carries a wide dynamic range, and neither has been measured here. `MLXCEL_CUDA_F16_FRAGILE` remains for whoever wants to take that measurement. Ampere and later are untouched. `is_f16_fragile_family` keeps its full list there, because the trade that makes the exemption worth taking below sm_80 does not exist above it. The exemption also has to precede both of Gemma's triggers, the `"gemma"` model_type substring and `final_logit_softcapping`, and has to look under `text_config`, since a multimodal Gemma carries `gemma4` at the top level and `gemma4_text` beneath it. * perf(cuda): reduce the f16-fragile list to what measurement supports below sm_80 Every family on the f16-fragile list has now been measured on a Tesla V100 against wikitext-2, f16 against bf16, and only one of them breaks. Below sm_80 the list becomes a positive list of what failed rather than the sm_80+ list minus exemptions, because there is no longer an unmeasured entry for the subtractive form to protect. Apertus stays. Its xIELU squares its input, and f16 produces NaN at every one of 508 scored positions from the first token while bf16 scores 24.27 with none. This is the entry the list was right about, and the mechanism is the reason: squaring grows a value past f16's 65504 into inf, and a subtraction of infinities is NaN. Gemma, gpt-oss and Cohere leave. Gemma moves perplexity by +0.09% over 10208 tokens with no non-finite value, takes decode from 6.12 to 9.79 tok/s, and holds output across four prompts at 200 greedy tokens. gpt-oss disagrees in sign between window sizes, +3.92% at 128 and -0.95% at 512, for a token-weighted -0.02%; an earlier 508-token sample had shown +3.83% and looked systematic, and it was one short window. Cohere needs no conversion to answer the question: `c4ai-command-r7b-12-2024-4bit` ships 483 F16 tensors and zero bf16, so it already executes in f16 on every backend while the list calls f16 unsafe for it, and it scores 20.25 and 13.15 at the two windows. The generic softcap and `logit_scale` config triggers are dropped below sm_80 as well. Both families carrying them measured clean, and a cap bounds the value it is applied to, which is the opposite of what exhausts an f16 exponent. Ampere and later keep the full list unchanged. The trade that makes this worth taking below sm_80 does not exist above it, where #732 measured f16 as offering no throughput gain at all. A family this list does not know, whose activations genuinely grow, would now convert and could reproduce the Apertus failure. `MLXCEL_CUDA_F16_NORMALIZE=0` is the way out, and the perplexity harness fails on a single non-finite value so that such a family is caught before it ships rather than after. That check is what surfaced Apertus: its perplexity came back as 1.0000, a meaningless number produced by every log-probability being NaN, and only the non-finite count showed the real state. --------- Co-authored-by: Sujin Kim <kimsujin@lablup.com> Co-authored-by: Jeongseok Kang <jskang@lablup.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
inureyes
added a commit
that referenced
this pull request
Sep 6, 2026
* fix(bench): detect the host and inline media in bench_embeddings
The harness hardcoded `hardware` to `NVIDIA_GB10_122GB`, so a CSV produced anywhere else silently carried the GB10 label and could not be trusted beside the GB10 ladder it exists to sit next to. It now detects the host the way `bench_decode.sh` does: `Apple_M5_Max_128GB` here, an `nvidia-smi`-derived string on CUDA.
Image cells sent the fixture as an absolute `file://` URL and every one returned HTTP 400 once #1481 confined request media. `resolve_media_file_in` strips the leading separator and joins onto the media root, matching llama-server b10621, so an absolute path is probed under the root and reported missing while a relative one resolves. The fixture is now inlined as a base64 `data:` URI, which needs no `--media-path` and keeps the ladder runnable on a host where nobody set one. Visual token counts match the GB10 pass exactly: 71 on Qwen3-VL-Embedding, 264 on Llama-Nemotron-VL-Embed.
Refs #1612
* docs(benchmarks): refresh M5 Max results for 0.6.0
Full text and VLM sweep on 0.6.0 with `--cooldown 30 --big-cooldown 30`, plus speculative, batched-serving, and embedding/rerank passes. 175 text checkpoints walked, 161 measured, and no decode regression across the 159 comparable with 0.4.0-rc.1.
The sweep now runs under a 90 GB weight budget (`BENCH_MEM_OVERHEAD_FACTOR=1.209`), so `deepseek-v3-4bit` at 99.96 GiB records `SKIP:oom_estimate` instead of being launched and recorded as a failure.
Two fixes that landed after 0.4.0-rc.1 changed the measured condition rather than the speed, and the docs now say so before any delta is read: #792 stopped upscaling every image to a fixed square, and the chat template now renders Llama's official prompt at 42 tokens against 98 before, matching the canonical tokenization exactly. Both shorten the prompt, and `prefill_tok_s` is prompt-length sensitive, which accounts for 10 of the 13 double-digit prefill drops.
Three stale claims are corrected: the GLM-5 pair is an interrupted local download rather than a load-path defect, `minicpm-v-4.6-mxfp4` passes now, and `deepseek-v3-4bit` is a capacity exclusion. The 12 duplicate checkpoint directories are recorded so they stop inflating the model counts.
`docs/benchmarks.md` command examples now match the actual script interfaces; `bench_decode.sh` takes a positional path and has no `-m` or `--runs`.
Refs #1612, #1613, #1614, #1615
* feat(bench): record the 8-char source revision in every CSV
`mlx_version` reads `Cargo.toml`, which does not move between releases, so every sweep taken across a development cycle records the same version however far `main` has travelled. A cross-hardware table assembled over weeks then cannot tell a hardware difference from a code difference, which is exactly the shape this branch is for: it collects the other hosts' sweeps, taken at different times, while being rebased forward.
`bench_decode.sh` now writes an 8-character `commit` column, with `-dirty` appended when tracked files were modified. Untracked files are excluded, since a stray note does not change the measured binary. The column is appended last so positional readers of column 11 (`mlx_version`) keep working, and it sits before any trailing `SKIP:` or `FAIL:` token so classification still reads the last field. `bench_embeddings.py` pins its existing commit capture to 8 characters to match.
The five CSVs from this sweep are backfilled with `b2ff1eee`, the commit the embeddings CSV had already recorded on its own.
`docs/benchmarks.md` states the rule: check that hosts share a commit before attributing a gap to hardware, and say so where they do not.
* docs(benchmarks): tie the M5 Max sweep to v0.7.0-beta.1
The branch now carries `v0.7.0-beta.1` while the CSVs record `mlx_version` 0.6.0, which reads as stale data unless the relationship is written down.
The sweep ran at `b2ff1eee`, when `Cargo.toml` still said 0.6.0, and the `commit` column records that. The tag (`64f5d9b4`) is exactly one commit later and touches no compiled source: the diff is manifests, `Cargo.lock`, `CHANGELOG.md`, `CITATION.cff`, `README.md`, `docs/environment-variables.md`, the recipes registry, and an issue template. `git diff --name-only b2ff1eee v0.7.0-beta.1 | grep -E '\.(rs|cpp|metal|cu|h|hpp)$'` returns nothing, so these are the beta's numbers rather than a release behind.
Sweeps taken from here record `0.7.0-beta.1` in `mlx_version`, which leaves the `commit` column as the discriminator within the cycle.
* refactor(bench): split the overloaded version column by meaning
One CSV column named `mlx_version` carried three different things depending on which script wrote it: the mlxcel crate version from `bench_decode.sh` and `bench_embeddings.py`, an actual MLX release in files written before the value stopped being hardcoded on 2026-06-12, and `mlx-lm-<v>` or `mlx-vlm-<v>` from `bench_mlxlm.py`. The command doc had to document the mismatch as a quirk to preserve.
Columns are now named for what they hold: `mlxcel_version` (88 files), `mlx_version` kept only where it really is an MLX release (15), and `baseline_version` for the Python baselines (26). Every header was decided per file from its own values rather than by position, and an audit confirms no column is misnamed. The five CSVs from this campaign additionally gain `mlx_commit`, the pinned MLX C++ revision, because a pin bump changes kernels without moving either mlxcel field; `commit` becomes `mlxcel_commit` for symmetry.
Those five are relabelled `0.7.0-beta.1`. They ran at `b2ff1eee` when `Cargo.toml` still said 0.6.0, but the tag is one commit later and touches no compiled source, and `mlxcel_commit` pins the revision regardless.
`mlxcel_version` stays at column 11, so the positional readers keep working.
* chore(bench): stop tracking per-run embedding server logs
`scripts/bench_embeddings.py` writes one server log per checkpoint to its default `--logdir`, `benchmarks/perf_logs/`. Those are transient run artifacts rather than results, they embed absolute local paths, and the GB10 embedding pass did not commit them either. An overly broad `git add -A` swept 18 of them in.
Untracked and added to `.gitignore` so the harness's default output cannot be committed again.
* fix: dedup checkpoint aliases in bench_decode.sh all mode
`bench_decode.sh all` enumerated every directory under `MODELS_DIR` with no notion of checkpoint identity, so a checkpoint present under two directory names (a re-download alias, or a symlink into a shared model store) was loaded, prefilled, decoded, and cooled down twice and landed in the CSV as two rows. On the M1 Ultra host `models/mlx` holds 15 such duplicate groups covering 16 redundant directories and 124.5 GB of redundant weights, present in every committed sweep back to `metal_m5max_2026-04-04.csv` and reaching `docs/benchmark_results/model_tests_m1ultra.md` and `docs/benchmark_results/model_tests_m5max.md` as separate table rows for three of the groups.
`all` mode now computes a cheap checkpoint identity key before the first model runs: sha256(config.json) plus the sorted (basename, byte size) list of every *.safetensors shard, with no weight hashing. This does not collapse checkpoints that differ only in quantization or weight dtype, since their config.json differs in the quantization block (verified against the bitnet-b1.58-2b-4t / bitnet-b1.58-2b-4t-4bit pair). Directories that resolve via realpath to the same physical path are also collapsed, independent of content. A directory with no config.json is never grouped by content and is always measured. The survivor in a group is the first directory the sweep's own glob enumeration reaches; every other member is skipped and recorded as its own CSV row with the trailing status SKIP:duplicate_of=<survivor-name>, so total row count is unchanged. The collapsed groups print to stderr once before the first measurement. --no-dedup restores the prior behavior exactly, and single-model mode is untouched.
Verified the grouping helper against MODELS_DIR=models/mlx: it found the same 15 groups and 16 redundant directories the issue's evidence table lists, byte-for-byte. Built a temporary MODELS_DIR of symlinks covering two small duplicate groups (gemma-3-1b-it-4bit/gemma3-1b-4bit and qwen2-0.5b/qwen2.5-0.5b-4bit) plus one control model and ran a real `all` sweep both with and without --no-dedup: the dedup run measured one row per group plus the two SKIP:duplicate_of alias rows, and --no-dedup measured all five directories individually. Ran bash -n and shellcheck on the script; no cargo command was run since nothing under src/ changed.
Refs #1615
* docs(benchmarks): update the media-path note for the #1612 fix
The "Why the image cells use a data URI" section described a rule that PR #1620 changes. It said an absolute `file://` URL can never resolve under `--media-path`, which was true when this pass ran and is no longer true: the concatenation stays primary, and an absolute reference whose concatenated candidate does not resolve is now canonicalized and put through the same containment check, so a file inside the root is read and one outside it is refused as an escape rather than as a missing file.
The observations from the run are kept in the past tense, since they record what the M5 Max pass actually saw. The reason the harness still sends a base64 `data:` URI survives the fix unchanged: a data URI needs no server flag, which keeps the ladder reproducible on a host where the operator never set `--media-path`.
The stale docstring on `_image_data_uri()` in `scripts/bench_embeddings.py` is named as a residue rather than fixed here, because that file is one of the many this PR already rewrites and the docstring is still right about why the harness uses a data URI. It changes no measurement.
Refs #1612
* docs(bench): refresh the file:// notes in bench_embeddings.py for #1612
The `_image_data_uri` docstring and the `--media-path` comment both described the pre-#1612 server, where only a relative `file://` URL resolved and an absolute one was concatenated onto the root and reported missing. PR #1620 kept that concatenation as the primary resolution and added a fallback that canonicalizes an absolute reference and puts it through the same containment check, so both spellings resolve now.
Comments only. The harness still sends a base64 `data:` URI, for the reason it always did: that needs no server flag, so the ladder stays reproducible on a host where the operator never set `--media-path`.
Refs #1612
* docs(bench): record the M1 Ultra speculative sweep for #1613
`speculative_bench` drove only a Gemma 4 Unified target until #1613 replaced that match with per-variant adapter selection and added the Qwen 3.8 27B pairings to `REACHABLE_PAIRINGS`. This records the first sweep run against that binary. Measured on the Mac Studio M1 Ultra 128 GB (Metal), mlxcel 0.7.0-beta.1 at `a74348f8`, MLX pin `9a795735`, `speculative_bench --sweep --batch 1 --max-tokens 128`: 16 rows, 4 baselines, 9 measured MTP rows at K=2/4/8 across Gemma 4 31B, Gemma 4 Unified 12B and Qwen 3.8 27B, and the 3 DFlash rows still deferred on their own blocker.
Every MTP pairing reads below 1.00x on this host and acceptance is not the reason: Gemma 4 Unified 12B accepts more here than on M5 Max at the same K (39.6% and mean accepted length 1.19 at K=4 against 35.0% and 1.05) and still lands at 0.74x where M5 Max reads 1.57x. The verify round is the difference, at 2.70 classic decode steps against 1.27, which is the reading already recorded in `speculative-decoding-m1ultra-2026-08-19.md` and the basis of the static gate that declines B=1 MTP on Apple GPU generation 13.
K=8 splits by drafter family. The Gemma 4 assistants clamp to their configured block size of 4, so acceptance and mean accepted length are identical at K=4 and K=8. The `qwen3_5_mtp` head honors the wider request instead, proposing 7 per round against 3 while accepting the same 1.51, which drops acceptance to 21.6% and throughput to 9.3 tok/s.
The M5 Max matrix keeps its own numbers. Its `unsupported target` cell and the two paragraphs explaining it now say the harness restriction was lifted and that an M5 Max re-run against a binary carrying the fix has not been made, rather than describing a limitation that no longer exists. The CSV index row and the overall-status row are updated for the same reason, and the M1 Ultra reachable-pairings table keeps its older reading with a pointer to the new matrix.
Refs #1613
* docs(bench): record the M1 Ultra MoE batched-decode attribution for #1616
Issue #1616 read the M5 Max batched-serving rows as the MoE decode path declining the fused kernel at B>=2 and falling back to `gather_qmm`. Profiling on M1 Ultra found a different cause. `Qwen3MoeModel` never overrode `forward_batched`, so the server's batched decode ran the `LanguageModel` default: one single-sequence `forward` per row, evaluated together. Each of those rows is a single routed token, so the fused kernel was reached on every one of them and the token-count gate was never what stood in the way.
The attribution document records the op-level measurements that decided what to build. Per-token fused launches scale linearly (3.9, 7.5, 14.7, 29.2 ms per 48 layers at n=1/2/4/8); identical and disjoint expert sets cost the same at every n, so expert-plane traffic is not the limiter and deduplicating expert ids across the cohort buys nothing; and a prototype of the batched fused kernel the issue proposed, bit-identical to per-token launches, loses to `gather_qmm` from n=4 (11.1 against 9.3 to 9.7 ms). That kernel was therefore not built, which is the outcome the issue's acceptance criteria allow when the profile does not support it.
The M1 Ultra ladder is in `benchmarks/metal_m1ultra_batch_2026-09-04.csv`, whose `mlxcel_commit` column separates the two passes: `bf1cdb72` is main before the change and `4aee0fb9` is the code PR. The after pass was re-measured on the final commit after a `cargo build --release` showed an earlier pass had been taken from a binary predating the last edit to the branch; the conclusion is unchanged and the numbers move by about 1%.
The dense-versus-MoE contrast this issue is named after is much narrower on this host than on M5 Max. There dense reads 3.17x and 3.25x against the MoE model's 1.55x; here dense reads 1.77x and 2.09x against 1.86x after the change and 1.72x before it. The M5 Max section is left as measured with a pointer to this work, because re-running that ladder is what would show the change's effect there.
Refs #1616
* docs(bench): correct the stated dedup survivor rule
The script header comment said the survivor within a duplicate group is "the first name in sort order". The code picks the first directory the sweep's own glob reaches, and that glob yields paths with a trailing slash, so where one name is a prefix of another the longer name wins: `pixtral-12b-4bit` survives over `pixtral-12b`, not the other way round. Six of the fifteen groups on the M1 Ultra store are prefix pairs, so the stated rule predicted the wrong survivor in 40% of cases.
`docs/benchmarks.md` already described the behavior correctly but did not say what the enumeration order implies for prefix pairs, which is the only case where the two readings diverge. Both now state it and name an example.
Comments and prose only; no behavior change.
Refs #1615
* perf(cuda): convert bf16 weights to f16 at load on pre-Ampere GPUs
Volta (sm_70) and Turing (sm_75) have no bf16 ALU, no bf16 tensor-core MMA atom, and no cuBLAS bf16 GEMM path, so every bf16 operand is converted before it can be executed. Measured on a Tesla V100: dense f32 reaches 14.0 TFLOPS and f16 reaches 95 to 100, while bf16 manages 9.6. Converting the weights once at load time therefore buys back an order of magnitude that per-operation conversion cannot.
On qwen3.8-27B-4bit this takes prefill from 85.3 s to 4.8 s and decode from 118.1 to 49.2 ms per token. The mechanism is `qmm_naive`, which runs an identical 994 launches in both dtypes and spends 199.1 s against 11.4 s; the dispatch does not change, only the operand cost inside the inner loop. Decode moves only on quantized checkpoints, because a dense model's batch-1 decode is bandwidth-bound and the two dtypes occupy the same bytes.
The policy itself now lives in one function, `bf16_to_f16_at_load`. It had been reimplemented at four sites that had drifted apart: the text loader carried the opt-in from issue #636, while the VLM and embedding loaders kept every family on bf16 regardless, and the selective-materialization arm picked its leaf dtype from an Apple-Silicon-only predicate. Consolidating them is what makes the policy apply at all rather than to one path in three. Each site now names the others in a comment, because the drift is invisible from any one call site.
Apple Silicon behavior is unchanged, and Ampere and later keep the opt-in from #636. `MLXCEL_CUDA_F16_NORMALIZE=0` opts out on pre-Ampere, and the f16-fragile family list is still honored. The conversion message no longer claims to be an Apple Silicon optimization when it fires on a CUDA device.
Verified on sm_70 only: prefill and decode on a quantized VLM and a dense text model, greedy answer quality in both dtypes, and embedding cosine matrices on two checkpoints where the paraphrase-versus-unrelated separation holds and the largest dtype disagreement is 0.0020. No Ampere-or-later device is present on this host, so the sm_80+ non-regression criteria are not verified here.
* refactor(cuda): delegate the load-time dtype policy instead of restating it
`bf16_to_f16_at_load` was introduced to stop the pre-Ampere rule from being reimplemented per load path, and then restated the predicates of `should_convert_bf16_to_f16` and `cuda_f16_normalize_for_config` inline rather than calling them, leaving both dead. Clippy caught it under `-D warnings`. It now delegates to each, so there is one copy of the CUDA rule and one of the Apple Silicon rule.
The selective-materialization arm in `tensor_view_to_array` goes back to the Apple Silicon predicate it used before. Switching it to the shared policy was wrong: that function has no `config` in scope, so it cannot honor the f16-fragile family exception, and an Apertus or BitNet checkpoint reaching it on pre-Ampere would have been converted against the policy's intent. It leaves a bf16 leaf on CUDA and `convert_bf16_weights` applies the real rule a moment later with the config in hand, which costs one extra pass over the bf16 tensors and is the correct trade. The comment there says so, because the next reader will otherwise make the same substitution.
The distinction that matters is not which sites touch the load-time dtype but which sites can see the config; only those can carry the policy.
* test(cuda): add the pre-Ampere f16 perplexity gate from #1542
The load-time bf16 to f16 conversion trades f16's 5-bit exponent for bf16's 8-bit one in exchange for an order of magnitude of throughput on hardware with no bf16 ALU. That trade is only defensible if the range it gives up is range the model never used, and #1542 makes measuring it a blocking criterion rather than an assumption.
The policy is applied once at load and cannot be toggled on a loaded model, so the two arms are two processes and the comparison is made across their printed tables. Non-finite log-likelihoods fail a single run on their own, because an f16 exponent overflow is a hard failure rather than a quality regression.
Measured on a Tesla V100. On Meta-Llama-3.1-8B-Instruct-bf16, where 291 tensors convert, perplexity moves by -0.46, -0.08 and -0.06 percent at the 128, 512 and 2048 windows. On qwen3.8-27B-4bit, where 1682 convert, by +0.03 and -0.12 percent at 128 and 512. Every point is inside the 1 percent gate and no window in either arm produced a non-finite value. Evaluation time at the 27B's 512 window fell from 265.4 s to 16.1 s, which reproduces the `qmm_naive` profile's 17.5x through a code path that has nothing to do with the benchmark it is confirming.
Two criteria stay unmet and are documented at the constants that encode them rather than left for a reader to rediscover. The MoE family cannot be covered here: the only local MoE checkpoint is Gemma, which the f16-fragile list excludes twice over, by family substring and by `final_logit_softcapping`, so both arms run identical bf16 and agree to four decimal places while testing nothing. The 27B's 2048 window exhausts a 32 GB card inside `evaluate_loglikelihoods`, identically in both arms, which is what establishes capacity rather than dtype as the cause.
Refs #1542
* fix(cuda): apply the load-time dtype policy to the Gemma 4 family
Gemma 4 reached the load-time dtype policy through no call site at all. Its four load paths, text, VLM, unified, and tensor-parallel, all bypass `load_text_weights` and `finish_vlm_weights_common`, because MLX's native `load_safetensors` crashes on Gemma 4 shards and the family needs the selective mmap loader. On pre-Ampere CUDA that meant the whole family ran bf16 and paid the full emulation penalty, and no environment variable could change it: `MLXCEL_CUDA_F16_NORMALIZE` in either direction did nothing, because nothing on that path read it.
The symptom was easy to misread, and was misread. Perplexity came back bit-identical between the f16 and bf16 arms, which looks exactly like the f16-fragile list correctly declining to convert a family it excludes, and Gemma is on that list twice over. Two hypotheses fit one observation: a policy that declines, and a policy nobody consults. Forcing conversion past the fragile list separates them, and all three arms stayed bit-identical, which only the second explains.
Locating the seam took three wrong guesses because `load_gemma4_text_weights_with_backing` carried its own copy of `load_gemma4_family_weights_with_backing`'s body, differing only in the `keep` filter, while the VLM and unified variants delegated. The policy landed in the copy the 26B does not use, and each function reads as correct on its own. The copy is deleted; the policy now lives once, in the function every Gemma 4 variant shares.
Measured on a Tesla V100 with gemma-4-26b-a4b-it-4bit: 1370 bf16 tensors convert, decode goes from 6.12 to 9.79 tok/s, and no non-finite value appears at any perplexity window. Output quality was checked directly rather than inferred from a metric: across four prompts at 200 greedy tokens the arms diverge in wording, as two different numeric types must eventually, while both stay correct, including the arithmetic prompt where both derive 396.
`MLXCEL_CUDA_F16_FRAGILE` is added as an opt-in escape for pre-Ampere users who want that speed on a family the fragile list excludes. The list's default is unchanged: it is inherited from #732, which adopted it for Ampere and later where f16 offered, in that commit's words, "no AsType reduction and no throughput gain", and whose test plan ran no fragile family in f16. Below sm_80 the same exclusion is not free, but flipping its default needs evidence this change does not yet have.
* test(cuda): prefix BOS in the perplexity harness and cover Gemma
Windows were scored without a leading BOS, so each one started mid-stream in a state the model was never trained to see. Every window now carries the model's BOS when it declares one, and the BOS position is excluded from the score so both arms still cover exactly `window_len - 1` corpus positions and the denominator does not move. On Llama-3.1-8B this lowers perplexity at all three windows, 17.5357 to 15.6685, 11.6873 to 11.4849 and 7.2963 to 7.2743, which is the direction more context should move it. That matters here beyond tidiness: this file exists to find f16 exponent overflow, and overflow has to be looked for in the distribution the model actually runs in.
Gemma is added as its own test now that the family reaches the policy at all. It is also the case the harness handles worst: its perplexity is implausible in absolute terms and rises with window length, 5111 at 128 against 31950 at 512, where Llama falls monotonically on the same code. Both arms show it identically, so it is not a dtype effect, and the Llama control above rules out the BOS change as the cause. Something in the Gemma evaluation path is wrong and is not diagnosed here. The numbers are recorded rather than trusted, and any perplexity conclusion about Gemma has to wait for that.
* fix(bench): measure every model over the same interval
Decode throughput was timed over a model-dependent number of tokens: a 6-token chat prompt, up to 100 tokens, stopping at EOS. On the 2026-09-04 M1 Ultra sweep only 25% of models reached the budget and 16% stopped under 20 tokens. The bias is not one-directional, which is why it went unnoticed: re-measuring moved `granite-4.1-8b-4bit` +117% (1 token, charging first-token latency to throughput) and `gemma2-2b-4bit` -45% (18 tokens, enjoying an almost-empty KV cache). Rankings invert under it, so per-model numbers were never comparable.
Both harnesses now use a deterministic 512-token synthetic prompt and exactly 128 generated tokens with end-of-generation suppressed: llama-bench's pp512/tg128, already what `scripts/bench_serving_concurrency.py` uses. `mlxcel-bench-decode` grows `--ignore-eos` as the `-inf` token bias the server uses for #1436, not a cleared stop set, because the generator merges the model's own `eos_token_ids()` with `SamplingConfig::stop_token_ids`. `bench_mlxlm.py` suppresses EOS through `logits_processors` and synthesizes its prompt from a corpus byte-identical to the Rust one, and resolves its interpreter through `MLXLM_PYTHON`, the uv-created `.venv-mlxlm`, `uv run`, then `python3`, since the system Python is 3.14 and has no mlx wheels.
Two classification gaps found while testing: a directory with no `config.json` or no readable `*.safetensors` now reports `SKIP:not_a_checkpoint` or `SKIP:missing_weights` before any model load rather than the `FAIL:bench` a real defect also produces, which covered 5 of 21 failures last sweep; and the #1615 dedup grouped two weightless directories as duplicates of each other because an empty shard list compared equal, hiding both.
Validated on `qwen3-0.6b-4bit`: both sides report `prompt_tokens=512`, `gen_tokens=128`, mlx-lm `finish_reason=length`.
* docs(server): the thinking-token budget is not Qwen3-only
`--reasoning-budget` was documented in ten places as applying to "Qwen3-family models", but `resolve_thinking_token_ids` has recognized three delimiter pairs for some time: `<think>` / `</think>`, `<|content_thinking|>` / `<|end_message|>`, and Gemma 4's `<|channel>` / `<channel|>`. A reader checking whether the flag covers their model got the wrong answer from every doc comment and from the CLI help.
`startup.rs` was additionally wrong about when the budget applies, not just to what. It said a model lacking the token IDs resolves the pair to `None` and the budget is ignored, which is true and incomplete: a checkpoint can carry the tokens and still never open a block, and there the budget is a silent no-op rather than a failure. `gemma-4-26b-a4b-it-4bit` is exactly that case. It carries `<|channel>` at id 100 and `<channel|>` at 101, and emits neither: with `--show-reasoning` no channel appears at all, and `thinking_budget_tokens` of 0 against -1 produces 469 tokens against 491 with the same answer. That gap between "has the tokens" and "uses them" is what the comment now states, because reading the old one leads to reporting the flag as working when it has done nothing.
* test(cuda): record why Gemma cannot be scored by perplexity here
The Gemma arm added earlier is removed, along with the one-shot diagnostics used to work out why it behaved as it did, and replaced by the conclusion those diagnostics reached.
`gemma-4-26b-a4b-it` scores between 255 and 88553 on this harness depending on the chunk, against Llama's 15.67 on the same corpus and the same code, with top-1 next-token accuracy of 20.5% against Llama's 59.1%. That reads as a broken evaluation path and was pursued as one through six hypotheses, all wrong: the caches are reset and chunk 0 scores identically alone or as the first of four; the sliding window is 1024 while the collapse is already total at 32; the final-logit softcap is applied; the tokenizer round-trips exactly; the logits are `[1, T, V]` with rows matching the input; and row `i` predicts token `i+1` at 20.5% against 2.3% for token `i`, so the alignment is right.
The explanation is that the checkpoint cannot continue raw text. With `--no-chat-template` it answers "Its most important consequence was" with "ownce of consequence-wise-wise-wise-..." and never recovers, while Llama-3.1-8B-Instruct on the identical invocation continues correctly through Gutenberg and the 1440s, and Gemma inside its chat template answers the same question well. Wikitext perplexity measures the ability this model no longer has, so it is an inapplicable metric rather than an inaccurate one, and a dtype comparison computed inside it carries no information. This is worth a comment rather than a deletion because the failure looks so much like a bug.
Two pieces of the diagnostic scaffolding are kept because they earn their place: `MLXCEL_PPL_CORPUS` scores a different file, which is what separated a model defect from a domain mismatch, and `MLXCEL_PPL_PER_CHUNK` reports each chunk, which is what disproved the cache-reuse theory by comparing the same chunk rather than two runs over different text.
* perf(sampling): stop synchronizing the device on every biased decode step
`apply_token_bias_stage` read the pre-bias argmax id back to the host on every decode step to feed the B9 suppression counters. That `eval` plus `item_i32` breaks the async lookahead pipeline the decode loop is built on, and it costs the same regardless of vocabulary size, which is what makes it a per-step tax rather than a per-token one.
Measured on M1 Ultra as the delta between `--ignore-eos` and no flag, both at pp512/tg128: +65.4% decode time on `qwen3-0.6b-4bit`, +30.4% on `llama-3.2-1b-4bit`, +24.0% on `qwen3-30b-a3b-4bit` and +13.6% on `llama-3.1-8b-4bit`. After gating the readback behind `MLXCEL_LANG_BIAS_COUNTERS=1` the same deltas are -0.6%, -1.2%, -0.4% and -1.0%, so the cost is gone rather than moved. `LANG_BIAS_APPLIED_TOTAL` stays always-on because incrementing it is free; only the two suppression counters, which need the argmax id, became opt-in.
`apply_token_bias` also no longer uploads a vocab-sized f32 vector per step to bias a handful of ids; it reads the biased positions with `take_along_axis`, adds, and writes back with `put_along_axis`, so the work is O(biased ids). That was the first hypothesis for the slowdown and was not the dominant cost, but the op is correct on its own terms and the 15 `token_bias` unit tests cover the additive semantics.
Nothing is affected unless a request or server opts in: `logit_bias`, `ignore_eos`, or `--lang-bias` / `LLAMA_ARG_LANG_BIAS`. An empty bias map still returns before any graph node is built, so the default path is unchanged. Language steering was the worst case, since the map stays resident for the whole session and every token paid.
Found while fixing the benchmark harness, whose new `--ignore-eos` made the instrument distort the measurement it was added to take.
* docs: add CMake to the CONTRIBUTING.md minimum development environment (#1642)
* docs(examples): fix dead link, placeholder, and stale in-progress claim in surgery examples (#1643)
* docs: document four missing MLXCEL_* server variables in environment-variables.md (#1640)
* docs(python): document the LLM/AsyncLLM constructor parameters and server_kwargs (#1646)
* docs(python): link the python/examples scripts from the README and client docs (#1645)
* chore(cli): port the Model store and Embeddings/Reranking help sections to mlxcel serve (#1650)
* docs: update the SECURITY.md supported-versions table to the 0.7.x line (#1639)
* docs: update the SECURITY.md supported-versions table to the 0.7.x line
* docs: keep the SECURITY.md recommendation consistent with a beta-only line
The supported-versions table now names 0.7.x, whose only shipped tag is `v0.7.0-beta.1`, so the closing sentence telling operators to run "the latest stable release" pointed at a release that does not exist on that line. It now says "latest published release" and names both the current tag and the last tag without a pre-release suffix, so an operator can tell which is which. Also squares the table's End of Support column padding, which the previous row had one character wide.
Refs #1630
---------
Co-authored-by: Jeongkyu Shin <jshin@lablup.com>
* docs: describe all six make verify gates in CONTRIBUTING.md (#1641)
* docs: describe all six make verify gates in CONTRIBUTING.md
* docs: say which command in the gate block `make verify` does not run
The enumeration covers the six prerequisites, but the code block above it lists four commands, so "the last three are the fmt, clippy, and test commands above" leaves a contributor to work out on their own that `cargo deny check` is the one `make verify` skips. Naming it, and naming PR CI as what gates it instead, is the point of the issue: a first failure should be self-explanatory in both directions.
Refs #1625
---------
Co-authored-by: Jeongkyu Shin <jshin@lablup.com>
* docs: point --distributed-config examples at the checked-in TOML templates (#1644)
* docs: point --distributed-config examples at the checked-in TOML templates
* docs: keep the CLI help's write-config step off the checked-in template
The `--help` blocks in `src/main.rs` and `src/bin/mlx_server.rs` are self-contained: step 1 runs `benchmark_pipeline_remote_rollout.sh write-config` and steps 2 through 4 consume what it wrote, so the runtime-generated name was correct there. Renaming its target to `pipeline_remote_2node_tcp.toml` made step 1 overwrite the checked-in template this PR introduces, which both contradicts the new `examples/distributed/README.md` and silently reverts an operator's edited addresses on the next `git checkout`. Those eight sites go back to `generated_pipeline_remote_2node_tcp.toml`, which is also the script's own default output path.
The two `docs/distributed.md` sites keep the PR's change: that page shows the servers started with no generation step in front of them, which is the case issue #1633 actually reported.
Adds a `.gitignore` rule for `examples/distributed/generated_*.toml` so following the help does not leave the tree dirty. The checked-in templates do not match the pattern and stay tracked.
Refs #1633
---------
Co-authored-by: Jeongkyu Shin <jshin@lablup.com>
* fix(server): stamp the speculative burst's first token at its prefill end so `timings` splits prompt from decode (#1609)
* fix(server): stamp the burst's first token at its prefill end so timings split prompt from decode (#1592)
On the speculative burst path the native `timings` block reported almost
the whole request under `prompt_ms` and 0 to 1 ms under `predicted_ms`,
so `predicted_per_second` read 95000 on a 96-token DFlash request. The
burst runs every verify round before any token reaches the sequence, and
the first-token stamp was taken when the finished token vector was
replayed, after drafter load, prefill and all rounds.
`BurstSuccess` now carries `prefill_end: Instant`, the moment the target
prefill finished and round 0 began, in place of the two millisecond
floats `finalize_burst_success` discarded. DFlash B=1 derives it from the
timers it already kept, MTP B=1 from `MtpGenerator`'s own prefill timer,
and the batched window from the batched prefill (DFlash stamps it after
the prefill forward; MTP subtracts the round loop's decode slice), so the
window stops passing `0.0, 0.0`. `finalize_burst_success` stamps the
sequence with it through the new `SequenceInfo::mark_first_token_at`,
which also stamps `StreamingDecodeState`, so the streamed `Prefill`
snapshot and the final `prompt_eval_ms` derive from one instant and the
replay's own `mark_first_token` is a no-op.
The drafter's once-per-process disk load is excluded from every timings
field: `WorkerDrafterSlot::ensure_loaded` returns `Some(elapsed)` when it
loaded, and the drivers and the slice start shift `seq.created_at` by
it. `created_at` feeds only the timing fields, so this is the issue's
origin shift without a new `SequenceInfo` field.
Four unit tests on a backdated sequence pin the split, the origin shift,
the zero clamp, and `Ok(None)` on a resident slot. The real-drafter
measurements the issue asks for were not run here: no checkpoints and no
Metal toolchain on this host.
Verified on a CPU-only build: `cargo fmt --all -- --check`, `cargo clippy
--profile test-fast -p mlxcel --lib --tests -- -D warnings`, and `cargo
test --profile test-fast -p mlxcel --lib server::batch::speculative_burst`
(66 passed).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y
* test(server): pin the classic path's first-token stamp too
The fix changes the classic path in exactly one way: `mark_first_token()` now delegates to `mark_first_token_at(Instant::now())`, which also stamps the decode state, so the streamed `PrefillStats.prompt_ms` and the final `prompt_eval_ms` derive from one instant rather than from two calls a few dozen lines apart. The PR's four new tests all drive the supplied-instant form, so that classic-path change had no coverage: a later reordering that let `on_token` win the stamp would move classic `prompt_eval_ms` with nothing to catch it.
The new test drives the classic ordering (`mark_first_token()`, a measurable gap, then `on_token`) and asserts the streamed snapshot equals the final frame, that `prompt_eval_ms` comes from the mark rather than the later token, and that the gap lands in `generation_only_ms`.
Verified by reversion rather than by passing: with `decode_state.stamp_first_token(at)` removed from `mark_first_token_at`, this test and `mark_first_token_at_splits_prompt_time_from_generation_time` both fail; restored, all 67 tests in the module pass.
Refs #1592
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Jeongkyu Shin <inureyes@gmail.com>
Co-authored-by: Jeongkyu Shin <jshin@lablup.com>
* refactor: deduplicate the byte-size formatters (#1651)
* refactor: deduplicate the byte-size formatters
Reuse execution::memory_estimate::format_bytes in quant_advisor, and make
KvCacheDimensions::total_size_str delegate to the file-local format_bytes
in kv_cache_transfer/benchmark.rs (adopting its no-space unit spelling,
which the format_bytes_ranges test already pins).
* refactor: record the spacing pick and pin the delegation in the test
The issue allows `total_size_str` to change spelling as long as the pick is deliberate and noted; the note was missing, so the only record that `256.0 MiB` became `256.0MiB` was a one-character edit to an assertion. The doc comment now states which spelling this is, that the inlined copy differed in two ways rather than one (it was spaced and it had no sub-KiB arm), and that reintroducing the space means moving two tests.
`bench_config_total_size_str` now asserts the delegation itself, `total_size_str() == format_bytes(total_bytes())`, so re-inlining a differently spelled copy fails here rather than only in `format_bytes_ranges`, and covers the `< 1024` arm the old copy did not have: a sub-KiB total used to render as `0.0 KiB` and now renders in bytes.
Refs #1626
---------
Co-authored-by: Jeongkyu Shin <jshin@lablup.com>
Co-authored-by: Jeongkyu Shin <inureyes@gmail.com>
* test: assert every flag named in `src/commands` exists in some `--help` (#1604)
* test(cli): assert every flag named in src/commands exists in some --help (#1131)
`validate_pipeline_parallel_args` once told the user to pass
`--tensor-parallel-size`, a flag no binary accepts (#1112). The sweep that
found it lived only in a PR description, so #1131 asked for it as a
machine-checked invariant in `tests/cli_help_consistency.rs`.
The new test collects the known flags from the built binaries: `--help` of
every subcommand in `mlxcel --help`'s `Commands:` block, `mlxcel-server
--help`, and the `--dump-flag-surface` JSON of both server entry points. The
dump is what makes this work without an allowlist: the llama-server b10621
compatibility groups are `hide = true`, and with `--help` alone all 13
mentions of them in `serve.rs` (`--spec-type`, `--mmproj`, `--numa`, ...)
came back as unknown. It then scans every `--flag` token in
`src/commands/*.rs` other than `*_tests.rs`, comments included, and fails
with one `file:line: --flag` per unknown name.
Injecting the #1112 defect (one `--tp-size` in `generate.rs` rewritten to
`--tensor-parallel-size`) fails the test at `src/commands/generate.rs:130`;
current `main` passes clean, so no source change ships with it.
Verified on a CPU-only build (no Metal toolchain on this host):
`cargo fmt --all -- --check`, `cargo clippy --profile test-fast --test
cli_help_consistency -- -D warnings`, and `cargo test --profile test-fast
--test cli_help_consistency` (26 passed).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y
* test(cli): take known flags from help signatures, not prose, and pin the scanner
Review of #1604 found two holes. The known set was scanned from the whole
help text, so a flag named only inside a description whitelisted itself and
a wrapped description contributed fragments such as `--embedding-`; it now
comes from `all_documented_spellings`, the file's existing signature-line
parser. The subcommand list took the first word of every indented line
under `Commands:`, so a wrapped description would have been spawned as a
subcommand; it now reads the name column only.
`long_flags` gets the meta-test every other parser in this file has,
covering separators, `=`-joined values, a URL, an uppercase token, and a
trailing bare `--`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y
* test: walk src/commands recursively so a submodule cannot go unscanned
The sweep used a single `read_dir`, which covers `src/commands` exactly as long as it stays flat. It is flat today (24 files, no subdirectories), so the shallow scan passes and says nothing; the day a subcommand grows a submodule directory, the flags inside it stop being checked and the test stays green while covering less than its name claims. That is the same shape of silent blindness the test exists to prevent, so it now walks the tree.
The `!sources.is_empty()` assertion already guarded against the directory moving; nothing guarded against it nesting.
Verified by construction rather than by passing: a throwaway `src/commands/nested_probe/probe.rs` naming `--definitely-not-a-real-flag` is reported as `src/commands/nested_probe/probe.rs:2: --definitely-not-a-real-flag` and fails the test; removed, the suite is 27 passed.
Refs #1131
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Jeongkyu Shin <inureyes@gmail.com>
Co-authored-by: Jeongkyu Shin <jshin@lablup.com>
* fix(server): let a malformed `MLXCEL_PAGED_SLAB_BLOCKS` fall back to the derived slab size (#1602)
* fix(server): let a malformed MLXCEL_PAGED_SLAB_BLOCKS fall back to the derived slab size (#1137)
`resolve_paged_slab_blocks` returned early on an unparseable value, which is
the `0` pin: the pool kept the historical 32-block slab while the warning said
the derived slab size was in use. A typo in the variable therefore quietly made
the fused paged decode path unreachable past one slab, and the log pointed the
operator away from the cause.
This takes option (b) from the issue: the parse-error arm warns and falls
through to the derivation, so a malformed value behaves exactly like an unset
one. `0` still pins the pool default. The warning now says exactly that, in
the structured form the file's other env-var warnings use, rather than
promising a derived value that the geometry probe below it may still decline
for a model that is not pool-eligible.
A unit test pins all arms against each other: unset and malformed derive the
same value, `0` returns `None`, and an explicit integer is used verbatim,
including below the 32-block floor the derived path applies. The
`MLXCEL_PAGED_SLAB_BLOCKS` doc row and the constant's rustdoc describe the
fall-through and the verbatim rule.
Verified on a CPU-only build (no Metal toolchain on this host):
`cargo fmt --all -- --check`, `cargo clippy --profile test-fast -p mlxcel
--all-targets -- -D warnings`, and `cargo test --profile test-fast -p mlxcel
--lib memory_estimate::tests` (39 passed, also at `--test-threads=8`).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y
* docs: say `0` pins the pool default in the MLXCEL_PAGED_SLAB_BLOCKS values column
"Non-negative integer, used verbatim" contradicted the row's own notes: `0`
is non-negative and is not used verbatim, it pins the 32-block default.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0134qsVMvsvs9LpZgCVH5m3Y
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Jeongkyu Shin <inureyes@gmail.com>
* chore(build): surface maintenance targets in make help (#1648)
## Summary
Six annotated targets (`update`, `tree`, `outdated`, `verify-versions`, `verify-kernel-dtype-keys`, `bump-version`) carried `##` help text that `make help` never printed because they matched none of the hardcoded keyword buckets; they now appear under a new `Maintenance Targets:` bucket. The bucket regex is also widened from `^[a-zA-Z_-]+:` to `^[a-zA-Z0-9_.-]+:` in every bucket so a future target name containing a digit or dot cannot silently vanish from help.
Verified by replaying the help pipeline over the Makefile: the new bucket lists exactly the six targets, and the widened regex extracts the identical target set as before (no bucket regressions).
## Related issues
Closes #1637
## Type of change
- [ ] `feat` — new user-visible feature
- [ ] `fix` — bug fix
- [ ] `perf` — performance improvement (include before/after numbers in the PR body)
- [ ] `refactor` — internal restructuring without behavior change
- [x] `chore` — build, CI, dependencies, release infrastructure
- [ ] `docs` — documentation only
- [ ] `test` — tests only
* docs(cli): document the .sock listener rule (#1647)
## Summary
The `--host`/`--port` help text in both binaries and two passages in `docs/python-client.md` still taught the deprecated `--port 0` Unix-socket spelling. Per `src/server/transport.rs` the rule is: a `--host` ending in `.sock` selects a Unix domain socket whatever `--port` says (the b10621 rule), `--port 0` on TCP binds an ephemeral port, and the legacy `--port 0` + path spelling emits a deprecation warning. The four flag doc-comments and both doc examples now describe the `.sock` rule.
## Related issues
Closes #1631
## Type of change
- [ ] `feat` — new user-visible feature
- [ ] `fix` — bug fix
- [ ] `perf` — performance improvement (include before/after numbers in the PR body)
- [ ] `refactor` — internal restructuring without behavior change
- [ ] `chore` — build, CI, dependencies, release infrastructure
- [x] `docs` — documentation only
- [ ] `test` — tests only
* fix(core): name --kv-bits in the bits validation error (#1652)
## Summary
The `--kv-bits` negative-value validation error leaked the internal config struct name (`BatchKvQuantConfig: bits must be non-negative (got N)`). It now names the flag with the same punctuation as every sibling arm in the `match`: `--kv-bits must be non-negative, got N`.
## Related issues
Closes #1629
## Type of change
- [ ] `feat` — new user-visible feature
- [x] `fix` — bug fix
- [ ] `perf` — performance improvement (include before/after numbers in the PR body)
- [ ] `refactor` — internal restructuring without behavior change
- [ ] `chore` — build, CI, dependencies, release infrastructure
- [ ] `docs` — documentation only
- [ ] `test` — tests only
* test(cli): cover infill and batch KV arguments (#1649)
## Summary
`src/cli/infill_args.rs` and `src/cli/batch_quant_args.rs` were the only modules in `src/cli/` without an inline test block. This adds them in the sibling `#[path = "..._tests.rs"]` style: parse tests for `--spm-infill` (absent → false, present → true, takes no value), and for the KV quant group a test pinning the hand-written `Default` impl to what clap produces from an empty argv — the two independent statements of the same defaults could previously fork silently — plus a flags-override test. The env-bound group takes the crate-wide env lock like its siblings.
## Related issues
Closes #1634
## Type of change
- [ ] `feat` — new user-visible feature
- [ ] `fix` — bug fix
- [ ] `perf` — performance improvement (include before/after numbers in the PR body)
- [ ] `refactor` — internal restructuring without behavior change
- [ ] `chore` — build, CI, dependencies, release infrastructure
- [ ] `docs` — documentation only
- [x] `test` — tests only
* perf(cuda): drop Gemma from the f16-fragile list below sm_80
The f16-fragile list keeps a family on bf16 even when normalization is on. It arrived in #732 for Ampere and later, where that commit records f16 as yielding "no AsType reduction and no throughput gain", and where the list is described in its own words as conservative. With no upside, excluding a family on a suspicion costs nothing, and that PR's test plan ran no fragile family in f16 at all. Below sm_80 the same exclusion costs 1.60x on decode, so it has to be earned rather than inherited.
Measured on a Tesla V100 with `gemma-4-26b-a4b-it-4bit`. Converting its 1370 bf16 tensors takes decode from 6.12 to 9.79 tok/s, no window of a perplexity sweep produces a non-finite value, and output quality holds across four prompts at 200 greedy tokens: identical Python, equally accurate history, and both arms deriving 396 for `17 * 23 + 45 / 9`. The mechanism agrees with the measurement. Gemma's trigger is softcapping, and `tanh(x / c) * c` bounds a value rather than growing it, which is the opposite of what exhausts an f16 exponent.
Cohere/Command-R, Apertus and gpt-oss stay fragile at every architecture. The argument that acquits Gemma does not transfer to them: Apertus squares through xIELU and gpt-oss carries a wide dynamic range, and neither has been measured here. `MLXCEL_CUDA_F16_FRAGILE` remains for whoever wants to take that measurement.
Ampere and later are untouched. `is_f16_fragile_family` keeps its full list there, because the trade that makes the exemption worth taking below sm_80 does not exist above it. The exemption also has to precede both of Gemma's triggers, the `"gemma"` model_type substring and `final_logit_softcapping`, and has to look under `text_config`, since a multimodal Gemma carries `gemma4` at the top level and `gemma4_text` beneath it.
* perf(cuda): reduce the f16-fragile list to what measurement supports below sm_80
Every family on the f16-fragile list has now been measured on a Tesla V100 against wikitext-2, f16 against bf16, and only one of them breaks. Below sm_80 the list becomes a positive list of what failed rather than the sm_80+ list minus exemptions, because there is no longer an unmeasured entry for the subtractive form to protect.
Apertus stays. Its xIELU squares its input, and f16 produces NaN at every one of 508 scored positions from the first token while bf16 scores 24.27 with none. This is the entry the list was right about, and the mechanism is the reason: squaring grows a value past f16's 65504 into inf, and a subtraction of infinities is NaN.
Gemma, gpt-oss and Cohere leave. Gemma moves perplexity by +0.09% over 10208 tokens with no non-finite value, takes decode from 6.12 to 9.79 tok/s, and holds output across four prompts at 200 greedy tokens. gpt-oss disagrees in sign between window sizes, +3.92% at 128 and -0.95% at 512, for a token-weighted -0.02%; an earlier 508-token sample had shown +3.83% and looked systematic, and it was one short window. Cohere needs no conversion to answer the question: `c4ai-command-r7b-12-2024-4bit` ships 483 F16 tensors and zero bf16, so it already executes in f16 on every backend while the list calls f16 unsafe for it, and it scores 20.25 and 13.15 at the two windows.
The generic softcap and `logit_scale` config triggers are dropped below sm_80 as well. Both families carrying them measured clean, and a cap bounds the value it is applied to, which is the opposite of what exhausts an f16 exponent.
Ampere and later keep the full list unchanged. The trade that makes this worth taking below sm_80 does not exist above it, where #732 measured f16 as offering no throughput gain at all.
A family this list does not know, whose activations genuinely grow, would now convert and could reproduce the Apertus failure. `MLXCEL_CUDA_F16_NORMALIZE=0` is the way out, and the perplexity harness fails on a single non-finite value so that such a family is caught before it ships rather than after. That check is what surfaced Apertus: its perplexity came back as 1.0000, a meaningless number produced by every log-probability being NaN, and only the non-finite count showed the real state.
* perf(cuda/quant): add a Volta tensor-core arm to qmm_naive, gated by tile_m
`make_tiled_mma` in `device/gemm_sm70.cuh` took a bool named `SM80`, which can say "tensor cores" or "no tensor cores" and nothing else. Volta needs a third answer: it has tensor cores, but only the 8x8x4 quad-pair atom and no bf16 atom at all, so a bool left it on `UniversalFMA` at this card's measured 14.0 TFLOPS f32 while the same card reaches 95 to 100 TFLOPS in f16. `qmm_naive` is 88.9 percent of prefill GPU time and 31.0 percent of a 180-token generation run on `qwen3.8-27B-4bit`, so that arm choice is most of the prefill cost. The flag becomes `MmaPath`, with a `Sm70` case selecting `SM70_8x8x4_F32F16F16F32_TN`.
This is the cheap version of the change, and deliberately so. #1543 originally scoped a port of the `qmm_sm80` CuTe pipeline, which needs `SM80_CP_ASYNC_CACHEALWAYS` and `SM75_U32x4_LDSM_N`, neither of which exists on sm_70; that port was written and thrown away once it became clear `qmm_naive` uses neither, so for this kernel the atom swap is the whole job.
Two things had to be measured rather than assumed. `SM70_8x8x4`'s ThrID is `SM70_QuadPair`, 8 threads rather than the 32 an sm_80 atom uses, so the atom layout has to be 16-way to land on the 128 threads every arm launches with; a 4-way layout gives 32 and the dispatch fails with `cuGraphAddKernelNode ... invalid argument` because the launcher sizes the block from a different arm than the kernel was compiled for. And the arm only works at small CTA tiles: `MLXCEL_TRACE_QMM_TILE` on a V100 reports 255 registers and no spill at tile_m 16, against 64 registers and 11304 bytes of spill at tile_m 64, which is the same launch failure again. Volta's atom is half the M and a quarter the K of sm_80's, so a tall tile needs far more accumulator fragments per thread. `kVoltaMaxTileM` caps it and falls back to the scalar arm above that, since scalar is the faster of the two options available there.
Correctness is verified by comparing the two arms on the same prompt, not against an expected answer. `MLXCEL_QMM_FORCE_SCALAR` exists for that comparison and is the reason it is trustworthy: an absolute check ("does the output contain 396") reads a reasoning model's suppressed channel as a wrong result, and on `qwen3.8-27B-4bit` it did, which cost a full revert of working code before `--show-reasoning` showed the model had been right all along. Both `qwen3.8-27B-4bit` and `gemma-4-12B-it-4bit` produce byte-identical output with the arm on and off.
Throughput is unchanged, and that is the honest result rather than a disappointing one. Decode moves 48.25 against 49.92 ms/tok on the 27B and 22.67 against 22.92 on the 12B, prefill deltas 6.57 against 6.56 s and 2.82 against 2.76 s, all inside noise. The cap explains it: a long prompt takes tile_m 64 and falls back to scalar, and that is exactly where `qmm_naive` spends its time. The arm is correct and reachable but not yet reaching the shapes that matter. Raising the cap needs the accumulator pressure addressed, which is the next step and is tracked in #1543.
Refs #1543
* perf(cuda/quant): bound the Volta arm by tile area, and narrow N before falling back
The Volta tensor-core arm was gated on `tile_m` alone, which is the wrong quantity and kept the arm off the shapes that matter. Raising that cap from 16 to 32 changed nothing observable, and the reason is worth stating: `qwen3.8-27B-4bit` takes `tile_m` 64 either way, so it kept falling back to scalar while looking like it had been given room.
What the accumulator actually scales with is the output tile's area. Measured with `MLXCEL_TRACE_QMM_TILE` on a Tesla V100:
| tile | area | registers | spill | result |
|---|---|---|---|---|
| 16 x 128 | 2048 | 255 | 0 | runs |
| 64 x 64 | 4096 | 226 | 0 | runs |
| 64 x 128 | 8192 | 64 | 11304 B | fails at `cuGraphAddKernelNode` |
Volta's atom is half the M and a quarter the K of sm_80's, so covering the same output tile costs far more fragments per thread, which is why sm_80 carries 64 x 128 without trouble and sm_70 cannot. `kVoltaMaxTileArea` replaces the `tile_m` cap, and over the bound the dispatch narrows `tile_n` to 64 and re-checks before considering the scalar fallback. That narrowing is the whole difference for the 27B: it reaches tensor cores through a narrower N tile instead of through the fallback.
Measured against the same binary with `MLXCEL_QMM_FORCE_SCALAR` as the control, so this is not a comparison across builds:
| model | arm | tile | prefill delta | prefill |
|---|---|---|---|---|
| qwen3.8-27B-4bit | Sm70 | 64 x 64 | 1.68 s | about 536 tok/s |
| qwen3.8-27B-4bit | Scalar | 64 x 64 | 6.54 s | about 138 tok/s |
| gemma-4-12B-it-4bit | Sm70 | 16 x 128 | 0.57 s | about 1579 tok/s |
| gemma-4-12B-it-4bit | Scalar | 16 x 128 | 2.76 s | about 326 tok/s |
Prefill is 3.9x on the 27B and 4.8x on the 12B. Both clear their scalar-path ceilings, 259 and 583 tok/s at this host's measured 14.0 TFLOPS f32, by more than double, which is what says the tensor cores are doing the work rather than the tile change alone helping. Each control matches its own earlier baseline, 6.54 against 6.56 s and 2.76 against 2.83 s.
Decode slopes do not move, 48.58 against 49.00 ms/tok on the 27B and 22.83 against 22.67 on the 12B, and should not: decode is `qmv`, and `qmm_naive` only handles a run's prompt prefill. The absolute decode time does drop, 3.63 and 9.46 s against 4.54 and 10.42 s, which is that prefill share shrinking.
Correctness is checked by comparing the two arms on the same prompt rather than against an expected answer, since the 27B now reaches the arm through the narrowed tile, a combination the earlier check never exercised. Both models produce byte-identical output with the arm on and off.
Refs #1543
---------
Co-authored-by: Sujin Kim <kimsujin@lablup.com>
Co-authored-by: Jeongseok Kang <jskang@lablup.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The CUDA single-dtype decode graph is the objective of issue #636: eliminate the per-token AsType (dtype-conversion) nodes so every weight, cache, activation, and sampler tensor sits in one dtype. This PR ships the AsType-per-token counter that is the "done" metric, inventories the remaining conversion sources on CUDA, and removes the one reducible source (the temperature-sampler chain). The headline result: the three inventory models already emit 0 AsType per greedy decode step on CUDA, because the merged
patches-cuda/dtype.cppbf16 promotion patch collapses the diffuse scalar/constant promotions that the MoE decode-gap investigation counted at ~773/token on Apple/Metal. The counter makes that state measurable and guards it against regression.What changed
count_astype_nodes_pair/astype_breakdown_pairwalks the unevaluated(token, logprob)decode graph, countsAsTypenodes, and produces a persrc->dstdtype breakdown. Exposed through the cxx bridge (lib.rs,mlx_cxx_bridge.{h,cpp}) and wired into the decode loop behindMLXCEL_TRACE_ASTYPE(generate.rs). Traversal only, no extra eval, entirely skipped when unset. Cross-checked againstMLXCEL_EXPORT_DECODE_DOT(grep -c AsType), which agrees at 0.fused_sample,top_p_filter, and the compiled min-p kernel built their temperature / sentinel / threshold scalars as f32, which the CUDA bf16 promotion patch then cast to the logit dtype every decode step. Each scalar is now built in the logit dtype on non-Metal backends only, gated on a runtime!metal::is_available()flag (the kernel-dispatch pattern already used in this file), keeping the CUDA sampler chain single-dtype. Metal keeps the bare f32 scalars exactly as main, so its temperature-sampling numerics are untouched per the issue AC. On CUDA the change is bit-identical for bf16 logits (the old path cast the f32 scalar to bf16 anyway); a natively-f16 CUDA checkpoint now runs temperature sampling in f16, which is the intended single-dtype behavior and within fp16 tolerance for the stochastic categorical draw.MLXCEL_CUDA_F16_NORMALIZEadds an opt-in, CUDA-only bf16 -> f16 cast of non-quantized weights (sanitize.rs), with a conservative f16-fragile exception list (is_f16_fragile_family) and a global env opt-out. Default off (see rationale below).docs/environment-variables.mddocumentsMLXCEL_TRACE_ASTYPEandMLXCEL_CUDA_F16_NORMALIZE;docs/benchmark_results/single-dtype-decode-astype-gb10-2026-07-10.mdrecords the inventory and before/after counts.Inventory (GB10, CUDA, SM 12.1)
patches-cuda/dtype.cpp(bf16 + fp32 -> bf16) and the consistent-dtype quant path. Nothing to reduce.f32 -> bf16)random::categoricaluniform drawu32 -> f32)f16 <-> f32, 2/layer)Before / after AsType per decode step
The greedy target reduction is vacuously >= 90% (0 before, 0 after): the objective was already met on CUDA by prior merged work. The one path with a real, reducible count (temperature sampling) drops 75%, with the residual
u32 -> f32intrinsic torandom::categorical.Quality: 40-token greedy parity (env off vs on)
Decode tok/s A/B (
mlxcel-bench-decode --prompt x --prompt-tokens 512 --max-tokens 128, 2 runs)Why CUDA f16 normalization is opt-in
The single-dtype objective is already met in bf16 on CUDA, and CUDA has native bf16 compute, so bf16 -> f16 yields no AsType reduction and no throughput gain while narrowing dynamic range. The load-time f16 path is therefore off by default, with the conservative f16-fragile exception list (gemma, cohere/command-r, apertus, gpt-oss, and any softcap/
logit_scaleconfig) keeping bf16 even when enabled. Quantized checkpoints are unaffected, so leaving it unset keeps every bf16 model available. Metal/Apple Silicon numerics are untouched: the always-on Apple policy is unchanged and the new path is gated on the CUDA backend.Test plan
cargo build --release --features cudaMLXCEL_TRACE_ASTYPEbefore/after on the three inventory models (greedy) and the temperature-sampler paths; cross-checked vsMLXCEL_EXPORT_DECODE_DOTcargo test --features cuda -p mlxcel-core --lib sampling::(46 passed)Note: the full representative sweep, perplexity spot check, and full test suite are left to the orchestrator.
Closes #636