Conversation
`benchbw` builds its banks with `cudaHostAlloc`, under a comment claiming that is
"like the loaders". It is not. Production allocates each bank as a lazy anonymous
mmap, fills it, and only then `cudaHostRegister`s it -- pin-after-fill, see
`host_banks.HostBank`. The two differ for anything that depends on how the pages
were obtained: NUMA placement, transparent huge pages, page-cache interaction.
It is not a cosmetic difference. Same box, median of 3:
format cudaHostAlloc production banks
bf16 70.7 GB/s 79.8 GB/s +13%
ds_fp4 65.3 GB/s 63.1 GB/s -3%
Not even a uniform shift -- the bench and production are simply in different
memory regimes, and which is faster depends on the format.
`--production-banks` switches the allocation path; the profile records which was
used in `bank_allocator`. Default is unchanged, because pinned pages cannot be
handed back (there is no `cudaHostUnregister` binding), so every format's banks
would stay resident for the whole run instead of being freed between formats. On a
memlock-limited machine -- which is most of them -- that turns a benchmark into an
allocation failure. Making it the default needs an unregister path first.
The immediate use is investigative: placement and huge-page effects are invisible
through `cudaHostAlloc`, which pins as it allocates, so pages can never be moved
and `madvise` has nothing to act on. Wanting to measure exactly that and being
unable to is what turned this up, along with the inaccurate comment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GpGe2fQ5pDShGrnSuksfun
|
Re-tested against current main ( Method. This PR's head merged onto main, then the full Result: 1205 passed, 350 skipped, no new failures. 🤖 Generated with Claude Code |
|
Closing this. It was produced with heavy AI assistance, and I would rather withdraw it than take it through review on that basis. The description and the diff stay here for anyone who wants to pick the idea up. |
The problem
ft bench bwallocates its synthetic expert banks with a plain host allocation. The loaders allocate the real banks differently: they map the bank, fill it, and only then register it with CUDA.Registration order changes the pages the gather reads from, so the benchmark can measure a gather that the deployment never performs.
What this adds
--production-banksallocates the synthetic banks the way the loaders do, through the sameHostBankpath. The default keeps the current allocation, so existing numbers stay comparable.This is a measurement-fidelity option, not a speedup. Its value is that a number produced with it can be trusted to describe the deployed path.
Testing
tests/moeonmainwith this PR: 94 passed, 6 skipped, 1 failed. The failure istest_cpu_moe_q4_0.py::test_cpu_decode_q4_0_matches_ggml_mmvq, which also fails onmainwithout this PR.This PR adds no test of its own. That the flag routes through
HostBankis assertable without timing anything. Tell me if you want that before merge.