perf: NEON kernel for 5/6-bit widths — scalar path is ~4.6x slower than the 4-bit NEON path

[ajianaz]

Found during #39 implementation (E2E bench, real EmbeddingGemma d=768, n=2000, 100 queries, release)

mode	B/vec	r@1	r@10	ms/q
plain 4-bit (NEON)	514	0.910	0.958	0.92
plain 5-bit (scalar)	642	0.970	0.979	4.27
plain 6-bit (scalar)	770	0.960	0.980	4.29
4-bit+residual (NEON)	1028	0.990	0.984	1.80

Recall/size goals of #39 are met, but 5/6-bit scoring currently runs the portable scalar unpack loop instead of the batch-4 NEON kernel, costing ~4.6x query latency vs 4-bit.

Scope

• score_neon_wide on aarch64: unpack 5/6-bit codes to bytes (or 16-bit indices), LUT gather from 32/64-entry centroid tables, widen+accumulate — keep bit-identity with score_wide_scalar (existing bitwise parity test pattern).
• Wire into the batch-4 slot loop next to score_neon4 when bits != 4.
• Optional AVX2 variant (use_avx2 && bits != 4 gate already exists).
• Acceptance: 5/6-bit ms/q within ~1.5x of the 4-bit path on the same bench; neon_matches_scalar_bitwise-style parity test extended to 5/6-bit.

Notes

• The 16-bit RMW pack/unpack helpers are correct but scalar; the kernel can read the same LSB-first layout directly.
• Until this lands, users needing both recall ≥0.98 and <2 ms/q should use residual (1.80 ms/q) or wait — documented in feat(core): parameterize quantizer bit-width (5/6-bit) — dominates residual mode on size AND speed #39.

[ajianaz]

Progress: window-extraction kernel shipped (a8edd9c)

score_neon_wide4 v2: one unaligned u64 window per 8 dims (replaces 8×unpack_code), L1 gather, NEON mul/add with per-lane bucket order bit-identical to score_wide_scalar (bitwise-tested, 5/6-bit × dims {128,384}).

E2E (real, d=768, n=2000): 5-bit 4.27 → 3.23 ms/q (−24%).

Honest ceiling analysis for the remaining gap (3.23 vs 0.89 ms/q)

Per-dim cost now: vectorized mul/add (~same as 4-bit path) + scalar extraction & gather (shift/and + L1 load). The 4-bit path extracts with 2 NEON ops per 16 dims and gathers via vqtbl4 (64-byte table = 16 f32 entries).

The vector gather for 5/6-bit cannot use vqtbl directly: tables are 128/256 bytes and vqtbl addressing caps at 64 bytes. The standard workaround (range-split gather: vqtbl4 on low nibble + vqtbl4 on high bit as select + vbsl) costs ~2–3× the 4-bit gather per dim.

Projected split-layout ceiling (separate low-4-bit + high-1/2-bit streams, vector extraction, split gather): ≈ 2.2–2.7 ms/q — i.e. 5/6-bit can likely never beat the 4-bit NEON path, and at best approaches the existing residual mode (1.76 ms/q @ r@10 0.984) which needs zero new machinery.

Recommendation

• Users needing recall ≥0.98 AND <2 ms/q today: residual mode.
• Split-layout (specified above) is the only remaining lever for 5/6-bit latency; expected payoff is bounded (~1.8–3.2 ms/q → ~2.2–2.7 ms/q at best). Worth it only if a workload is pinned on plain-mode latency AND cannot rebuild as residual.
• AVX2 wide kernel: same analysis applies; scalar-wide remains correct and bit-identical on x86_64.

[ajianaz]

v2 kernel landed in develop (merged via #41, eb26d67)

score_neon_wide4 (u64-window extraction + L1 gather + NEON mul/add, bitwise-tested vs scalar) is in develop. 5-bit: 4.27 → 3.23 ms/q (−24%).

Remaining scope for this issue, in priority order:

1. Split-layout codes (separate low-nibble + high-bit streams): the only lever toward ~2.2–2.7 ms/q. Requires format v1.6 or layout-in-header; bounded payoff per ceiling analysis above.
2. AVX2 wide kernel for x86_64 hosts (currently scalar-wide, bit-identical, correct).

Keeping this open for the split-layout decision; no urgency — residual mode already covers the recall≥0.98 + <2 ms/q niche at 1.76 ms/q.

[ajianaz]

Closed: primary scope shipped, remaining scope declined with rationale

Shipped (merged via #41):

• score_neon_wide4 batch kernel for 5/6-bit — u64-window extraction + L1 gather + NEON mul/add, bitwise parity vs scalar (tested at 5/6-bit × dims {128,384}).
• 5-bit scan latency: 4.27 → 3.21 ms/q (−24%). The original "~4.6x slower" gap is now 3.6x, with extraction (not SIMD math) as the documented bottleneck.

Declined, with analysis (see ceiling comment above):

• Split-layout codes (v1.6): projected only ~2.2–2.7 ms/q best case; requires a format change + migration; and the niche it targets (recall ≥0.98 + <2 ms/q) is already served by residual mode (1.76 ms/q @ r@10 0.984, shipped Residual quantization (TurboQuant-style second pass) — recall at same bit width #23/feat(core): residual quantization — opt-in recall mode, format v1.4 (#23) #38). Reopen with a workload profile that is pinned to plain-mode latency and cannot rebuild as residual.
• AVX2 wide kernel: same extraction bottleneck applies on x86_64; scalar-wide is correct and bit-identical. Not worth it before split-layout exists.

Final landscape is documented in docs/BENCHMARK.md (width matrix) — close as completed.