VQ4P's apply is an #ifdef in model.c, not a dispatch slot — which is why x86 has no kernel

[marcobambini]

vq_rows_p6 — the VQ4P apply — is vectorized behind an #ifdef in
src/model.c:2158 rather than behind the dispatch table, and that is the
structural reason x86 has no kernel for it rather than a narrow one.

#if (defined(__ARM_NEON) || defined(__aarch64__)) && !defined(WASTE_P6_SCALAR)
            if (nr == VQ_TILE) {
                /* four vqtbl4q_s8 per 16 rows over a vld3q_u8 deinterleave */
            } else
#endif
            {
                /* one row at a time, four scalar table loads per row */
            }

The two range kernels that carry the rest of the arithmetic — mvq_rows_f32
and lutb_range — moved behind waste_kernels on 2026-07-28 precisely so
that one binary adapts from CPUID and the ISA lives in its own translation
unit. This one did not move with them, so an x86 build takes the portable
else branch: T[j0] + T[en + j1] + T[2*en + j2] + T[3*en + j3], per row.
That is not a narrower kernel, it is the absence of one, on the same build
where VQ3R's table gets simd_avx512.c.

Successor to #32, which is closed: its documentation half landed in
48b8afa, and this is the
half that needs code.

Scope: who actually meets this

Narrower than #32 claimed, and worth stating before anyone prioritises it.
#32 said VQ4P is what convert.py produces by default. It is not —
--index-bits defaults to 8 and --stages to 3, so a default conversion is
VQ3R, and convert.py:903 refuses --index-bits 6 without
--stages 4 --entries 64. So this is not the path on every x86 build.

What makes it worth doing anyway is who the affected population is. That
shape is exactly the 64-entry crossover docs/LEARNED.md §50 lists as not
measured. So the people most likely to run VQ4P on x86 are the people trying
to answer that open question — and what they would measure is a scalar apply
against a vectorized one, then report an ISA difference as a table-size
result. A wrong number that looks like an answer costs more than a slow path
that announces itself.

Why a kernel is possible here and is not for VQ3R

§41 has the arithmetic. VQ3R's 256-entry stage table is 256 bytes — sixteen
vector registers of the thirty-two that exist — so it fits no byte-permute
primitive on any ISA, and blocking does not make it fit. VQ4P spends the same
24 bits per 8-weight vector at 4x64, and a 64-entry stage table is 64 bytes,
four registers, which is exactly what vqtbl4q_s8 addresses. The premise
that made the gather scalar is false for this format.

The x86 analogue is AVX-512 VBMI's vpermi2b: 128 bytes indexed across
two zmm registers, so 64 entries fit with room to spare. Zen 4 and later, Ice
Lake and later.

The work, in the order that makes it reviewable

1. Add the dispatch slot first, with the ARM body moved into it unchanged.
   waste_kernels has no entry for this kernel; add one taking
   (int b, int e, void *arg) like the other two, move the NEON body out of
   model.c into its own translation unit, and register it from the ARM path.
   This step should be bit-identical on ARM by construction and is worth
   landing on its own — it is reviewable without an x86 machine, and it is
   where a mistake would be cheapest to find.

   It also improves the self-check. -DWASTE_P6_SCALAR is a compile-time
   lever today; once the kernel is a slot, WASTE_BACKEND=cpu reaches the
   portable path at runtime, which is the lever the existing suite check
   already uses.

2. WASTE_CPU_AVX512VBMI does not exist yet. src/waste_backend.h:56-57
   has AVX512F (bit 11) and AVX512BW (bit 12); VBMI is CPUID leaf 7
   subleaf 0, ECX bit 1, and src/backend.c:123 is where its siblings are
   detected. Gate on it specifically — AVX512F && AVX512BW at
   backend.c:190 is not sufficient for vpermi2b.

3. Then the kernel, in src/simd_avx512.c, which already exists.

Acceptance

• Bit-identity against the portable path, not "close". §43 says why the
  bar is that high: an int8 table makes the engine discontinuous, so an
  approximate match is not a match. It is reachable — accumulation is int16
  within a WASTE_VQ_LUT_BLK block and only folds to fp32 at the end, so
  there is no float reordering for a different lane arrangement to expose.
  The ARM path already claims this against -DWASTE_P6_SCALAR; x86 has to
  clear the same bar against the same reference.
• The check that decides it is tests/run.sh's SIMD backend matches the
  CPU baseline, which runs WASTE_BACKEND=cpu against the dispatched path
  and compares logits.
• It has to run on an --index-bits 6 container. The suite's default
  WASTE_REF_MODEL is a VQ3R conversion, which does not execute this kernel
  at all; per CLAUDE.md, running the suite on the wrong container shape is
  exactly how a load path once stayed broken through green runs. Build one
  with convert.py --index-bits 6 --stages 4 --entries 64.

Unmeasured in both directions

Stated before anyone spends a weekend on it:

• The path may be bound by the deinterleave (vld3q_u8 on ARM, three
  interleaved index bytes per row) rather than by the lookup, in which case
  the table primitive buys much less than it looks like it should.
• Cross-lane byte permutes are not uniformly cheap across x86
  microarchitectures, and vpermi2b's cost differs between Ice Lake and
  Zen 4/5.

This is a hypothesis with an obvious first experiment, not a plan.

Nobody here can execute it

This repository has no x86 machine that is not emulated, which is also why
docs/BACKENDS.md records AVX-512 as "compiled and dispatched, never
executed", and why the gap was found by reading rather than by measurement.
Whoever writes this is also the only person who can test it, so the
measurement matters more than the patch — and step 1 above is deliberately
carved out as the part that can be reviewed here.

Related: #11 (a GPU VQ4P implementation exists off-repo and found that
building the table dominated applying it — a caution about where the time
actually goes, on a different vehicle), and #36 (the suite's non-synthetic
path does not run in CI on any platform, which is the reason a container-shape
requirement in the acceptance criteria above cannot be enforced by CI today).

[marcobambini]

Cross-reference: #37, filed today, is a 128 GB Ryzen AI Max+ 395 (Strix Halo, Zen 5) with AVX-512 and VBMI — the first non-emulated machine with the primitive this issue needs to appear anywhere near the project. @Lrrr908 offered it for a different open gate; I have pointed them here as well, with no obligation attached.

Two things follow for whoever picks this up. That machine can run the SIMD backend matches the CPU baseline check on AVX-512 for the first time, which is a prerequisite for trusting any VBMI result and is worth doing before a line of vpermi2b is written — docs/BACKENDS.md records those kernels as compiled and dispatched but never executed, so their correctness is currently an assumption rather than a measurement. And step 1 here — the dispatch slot, with the NEON body moved into it unchanged — is still the right thing to land first regardless, since it is reviewable on ARM and does not depend on anyone's hardware.

[GTSUltear]

Measured this on a non-emulated x86 machine, since #32 noted there is none upstream — Ryzen 5 3600X (Zen 2, AVX2, 6C/12T), Windows 11 native under MSYS2 UCRT64. Setup detail in #36.

Caveat first: Zen 2 has no AVX-512 and no VBMI, so this says nothing about whether vpermi2b would pay. It bounds the cost of the absent kernel, and that number is smaller than the obvious comparison suggests.

The obvious comparison overstates it

Converted the same Kimi-Linear source into VQ4P (--stages 4 --entries 64 --index-bits 6) alongside the existing VQ3R container. WASTE_XPAR=1 --budget 16G -n 200, interleaved on an idle machine:

VQ4P lands at 0.57–0.60x of VQ3R across two full runs. Against §50's 1.18x faster for VQ4P on Kimi-Linear, that is roughly a 2x platform swing, and it would be easy to attribute all of it to the #ifdef at model.c:2158.

Most of it is not that. A third container isolates the packed path — --stages 4 --entries 64 --index-bits 8, same stage count and codebook size, ordinary 8-bit indices — and it sits at 0.62–0.64x. So the 4-stage/64-entry shape accounts for most of the deficit on x86.

The specifically-packed-index portion is the VQ4P-to-control ratio, and that is the number worth quoting because it is the stable one. Two full runs, two budgets each:

	run 1	run 2
--budget 16G	0.9332	0.9319
--budget 32G	0.9321	0.9269

~6.9%, with 0.7% spread over four independent measurements. The VQ4P-to-VQ3R ratio moved more between runs (0.574 to 0.599) because VQ3R gains more than VQ4P when the machine is quieter, so I would not quote that one tightly.

On this hardware the missing kernel is worth ~7%, not ~40%.

Two things about that number. It is a floor, not an estimate: the control uses the ordinary 8-bit apply, not a vectorized p6, and on ARM the vectorized p6 beats the 8-bit path, so a real kernel could recover more. And it is not an I/O artifact — the control is 4.0 b/w reading 16.11 GB against VQ4P's 3.0 b/w and 11.55 GB, which is why both budgets are in the table above: at --budget 32G the cache is 30.72 GB and every expert set fits entirely, so capacity misses vanish. The ratio does not move. The control's extra reads are compulsory rather than capacity — first touch of a larger expert set — and cost nothing measurable, matching everything else on this machine pointing away from disk.

WASTE_XPAR is format-dependent, not only machine-dependent

Same containers and conditions:

	no XPAR	WASTE_XPAR=1	gain
VQ3R	5.145	5.213	+1.3%
VQ4P	1.995	3.077	+54%

§47 records the setting inverting between models, and #36 noted it varying between machines. It also varies between expert formats on one machine, by more than either — which retrospectively explains the +2 to +3% I measured in #36, where the container was VQ3R rather than the VQ4P the tuning table was built against.

The acceptance criterion cannot be met on this container shape

This issue names SIMD backend matches the CPU baseline as its gate. On
an --index-bits 6 container that check cannot pass, on any platform, and
not because anything is broken. Worth knowing before the kernel is
written rather than after.

Measured, with the mechanism rather than just the symptom. Full write-up in
#36; the short version:

container	stages/entries	index bits	AVX2 vs CPU
VQ3R	3 / 256	8	1.04904e-05, 0/163840
VQ4P	4 / 64	6	1.22853, 163297/163840
CTRL	4 / 64	8	1.04904e-05, 0/163840

CTRL is the control: same codebook geometry, only the index width differs.
Three passes reproduced every digit above, so it is deterministic and not a
sampled maximum.

AVX2 fills exactly two slots, so bisection is exhaustive — and either one
alone produces the divergence (1.270 with mvq_rows_f32 only, 1.633 with
lutb_range only, 0 with neither). Neither is at fault. What
index_bits == 6 alone does is quantize the float LUT to int8, and that
is a discontinuity: instrumenting the tables directly, the first LUT
built differs in exactly one entry by one step out of 73,728, and by the
end of the pass 42% of entries differ. vq_rows_p6 accumulates in int16 —
exactly — so nothing averages it back out, and 27 layers do the rest. The
control never enters that path at all.

The bearing on this issue. Step 1 here is moving the ARM body into a
dispatch slot, "bit-identical on ARM by construction". That is still the
right thing to land first. But anyone who then checks it on x86 against an
--index-bits 6 container will see max diff 1.2 and conclude they broke
something. They will not have — and no tolerance can tell the two apart,
since a genuinely broken kernel produces the same magnitude.

So the gate for this path probably has to be bit-identity of the int8 table
(or of the routes) rather than a threshold on logits, which is LEARNED §43's
point arriving somewhere concrete.

Both containers, the control, the source weights and the toolchain are set up here, so re-running anything is cheap.

—

Reported by Kirin.

🤖 Built, measured and triaged with Claude Code.