Conversation
…olated one
`ft bench bw` already measures the CPU MoE GEMV and the PCIe gather twice: once
each standalone, and once running concurrently -- "the contention regime hybrid
decode's overlap actually lives in", as `measure_overlap_bw` puts it. But only the
fetch split used that pair. The hybrid-vs-offload verdict rode the standalone
numbers, which describe a situation that never occurs: hybrid decode never runs the
CPU GEMV alone.
It matters because contention is not symmetric. On a 2x Xeon Gold 6526Y the bf16
CPU kernel barely moved when the gather ran alongside it (71.9 -> 66.8 GB/s) while
the gather lost nearly 40% (25.1 -> 15.4). Measured ratios on that box:
format standalone contended
bf16 2.95x 4.27x
ds_fp4 2.63x 2.99x
mxfp4 1.94x 1.56x
Same verdicts here, but each is further from the 2x threshold than it looked, and
mxfp4 -- the format that flip-flops between runs on this hardware -- moves furthest
away from it, so the contended measurement is also the more decisive one.
The overlap bench now runs before the decision instead of after it. It is the
measurement that can fail on its own (it needs both paths live at once), so a
failure falls back to the standalone pair with a note rather than losing the
recommendation; `verdict_source` records which was used.
The printed bandwidth columns still show the standalone numbers -- they are the
useful hardware ceilings -- with a line saying so, and the overlapped row is marked
as the one the ratio came from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpGe2fQ5pDShGrnSuksfun
|
Re-tested against current main ( Method. This PR's head merged onto main, then the full Result: 1205 passed, 350 skipped, no new failures. 🤖 Generated with Claude Code |
|
Closing this. It was written with heavy AI assistance, and the maintainers have indicated that do not want such contributions. The description and the diff stay here for anyone who wants to pick the idea up. |
The problem
ft bench bwmeasures the CPU MoE kernel and the PCIe gather separately, each with the machine otherwise idle, then divides the two.The
hybridbackend runs both at the same time. They contend for the same memory controllers, so neither reaches its isolated figure. A ratio built from two isolated measurements describes a situation that never occurs in production.What this changes
The tool now also measures the pair running together and decides on that. It reports both, so the isolated numbers stay visible for tuning:
The second line is what the verdict now uses.
What it revealed
On this machine the isolated ratio says
hybridby a comfortable margin, 2.61x against a 2.0x threshold. Under contention the PCIe side drops to 14.8 GB/s, so hybrid can only fetch 17.2% of misses while the CPU handles the rest.That matches what the deployment does. End to end,
hybridmeasured 3.9% slower thanoffloadat one stream and 15.7% slower at eight. Those two figures come from the deployment branch, which carries several other changes, so treat them as corroboration rather than as a measurement of this PR. The isolated ratio recommended the slower backend; the overlapped measurement does not.Testing
tests/moeonmainwith this PR: 94 passed, 6 skipped, 1 failed. The failure istest_cpu_moe_q4_0.py::test_cpu_decode_q4_0_matches_ggml_mmvq, which also fails onmainwithout this PR.This PR adds no test of its own. The overlapped measurement is timing code, so a value assertion would be flaky, but the verdict selection is testable as a pure function the way #37 tests its own. Tell me if you want that before merge.