perf(index): accelerate RQ scale search

[BubbleCal]

What is the performance issue?

RQ scale search generates several quantization thresholds per vector dimension and sorts them before selecting the best rescale factor. For RQ8 on 1,536-dimensional vectors, comparison-based tuple sorting was the dominant sampled cost in this part of index training.

How does this PR improve performance?

This PR packs each (positive finite f32 threshold, dimension index) pair into a u64 and sorts the threshold bits with four stable byte-wise radix passes. Positive finite IEEE-754 values have the same ordering as their bit patterns, so this removes comparison-heavy tuple sorting for the common unique-key case. Equal thresholds retain the previous quantizer behavior by falling back to the original comparison sort because their event order can affect the incrementally evaluated floating-point objective.

The implementation is isolated from the shuffle reader and scheduling changes in #8894.

Benchmark

The following supporting A/B measurement isolated this implementation on the #8894 benchmark context. Both variants ran on the same AWS m7i.4xlarge VM (16 vCPU, 64 GiB RAM, gp3 storage) against the same S3 dataset: 1,000,000 rows, 1,536 float32 dimensions, 10,000 supplied IVF centroids, and RQ8. Each value is from one fresh process.

Scenario / metric	Comparison sort (c7508ad49)	Radix sort (174663d0a)	Benefit
Shuffle elapsed (lower is better)	70.219 s	51.470 s	1.36x speedup
Full index build elapsed (lower is better)	76.831 s	58.346 s	1.32x speedup
Shuffle CPU time (lower is better)	1,012.95 core-s	718.29 core-s	1.41x less CPU time
S3 read throughput during shuffle (higher is better)	84.16 MB/s	117.61 MB/s	1.40x higher

The two measured commits differed only by the initial RQ radix implementation, but they were on the stacked #8894 context and predate the equal-threshold comparison-sort fallback in this standalone latest-main PR. The standalone head has not been remeasured, so the table is supporting implementation evidence rather than a current-head benchmark claim.

Testing

• cargo test -p lance-index vector::bq::builder::tests --lib --no-fail-fast (13 passed)
• cargo fmt --all -- --check
• cargo clippy --all --tests --benches -- -D warnings

The added regression tests verify that radix sorting matches floating-point threshold ordering and that the selected rescale factor is bit-for-bit identical to the comparison-sort reference for RQ1 through RQ8, including zero, duplicate, NaN, and infinite inputs. A targeted equal-threshold regression also verifies the exact rescale factor from the previous comparison-sort behavior.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The equal-threshold correctness issue is fixed: duplicate keys now reproduce the prior comparison-sort behavior, while unique keys retain the radix path. The remaining risk is that the benchmark predates this fallback; duplicate-heavy inputs perform both sorting paths and use additional temporary memory, so the target-workload speedup is not yet demonstrated on this head.