Widen expected-MI cache keys to the full integer range - #141
Conversation
mi_key() narrowed each block size to uint16_t, so partitions whose block sizes differed by a multiple of 65536 shared a .ExpectedMICache entry and the second was served the first one's expected mutual information. uint32_t spans the whole of int, so the encoding is now injective and the collision is impossible rather than unlikely. The two sort invariances the key relies on hold: expected_mi() agrees to 2.8e-13 when the blocks of ni are swapped and to 1.5e-15 when nj is permuted, over 2000 random partitions. Fixes #132 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Doubling the hex width made mi_key() measurably slower in isolation: at 2e6 calls per arm, appending to a reserved string costs 250 ns against the 16-bit original's 215 ns at three blocks, rising to 318 vs 239 at eight. Sizing the string once and writing through a pointer drops the per-character capacity check, and is 1-4% faster than the 16-bit original at every block count from 2 to 16 despite emitting twice the characters. The emitted key is unchanged; a new assertion pins it to fixed-width hex of the sorted values, using 0x12345678 so a mis-shifted nibble fails. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Performance of the wider keyReview raised that the hand-unrolled hex emit looked profile-tuned, and that Timed in C++ (
The shift loop first pushed was 15–53% slower than the original. Unrolling it The emitted key is byte-identical across all three 32-bit variants; a new Why this was invisible end-to-end
There is no hotter caller to measure instead: nothing in |
…cache-aliasing # Conflicts: # NEWS.md
Fixes #132
mi_key()narrowed every block size touint16_tbefore hex-encoding it, sotwo partitions whose block sizes differed by a multiple of 65536 received the
same
.ExpectedMICachekey, and the second was served the first one's expectedmutual information.
38 000x wrong, with no diagnostic.
ClusteringConcordance(normalize = TRUE)would have subtracted that as its chance correction.
The fix
uint32_tspans the whole ofint, and theint->uint32_tconversion isinjective across that whole range, so distinct block sizes now always give
distinct keys — the collision is impossible rather than merely improbable.
Values are encoded in 8 hex characters instead of 4; the key roughly doubles in
length, which costs nothing in a hashed environment.
mi_key()'s signature is unchanged, soRcpp::compileAttributes()on this treeleaves
src/RcppExports.cppandR/RcppExports.Rbyte-identical (verified via.claude/tools/compile-attrs.R) andsrc/ts_rcpp.cppandsrc/TreeSearch-init.care untouched. No collision with #122 or #56.The two sort invariances are sound, and stay
mi_key()canonicalises by sortingniandnj, which is only legitimate ifexpected_mi()is invariant under both. It is. Over 2000 random partitions withNin 6..400 and 2-5 blocks:ninjBoth are floating-point noise, so sorting is not a second, size-independent
aliasing bug. That is worth stating explicitly, because had either invariance
failed the key would have been wrong at every tree size rather than only above
65535.
How reachable was it?
Remote, and worth saying so plainly rather than overselling the fix.
A block of 65536 requires a character scored across at least 65536 tips, so it
needs a tree of at least that many tips.
ClusteringConcordance()— the onlycaller of the cache — first evaluates
as.logical(as.Splits(tree)), which at65536 tips is a 65533 x 65536 logical matrix: 16 GB, which
apply()thencopies. Measured scaling on this build (5 characters) is close to quadratic:
which extrapolates to roughly 40 minutes and ~32 GB at the threshold. So it is
not unreachable — a large-memory machine could do it — but nobody is hitting it
today, and no published result is affected. The case for fixing it is that the
failure is silent and unbounded while the fix is confined to the encoder.
mi_key()is internal (not inNAMESPACE);expected_mi()is exported butuncached, so a direct call was never affected.
Tests
Three assertions added to
tests/testthat/test-expected-mi.R, all Tier 1arithmetic. Verified against a build of
cpp-searchat cd991cb, i.e. thisbranch's parent:
.ExpectedMI()correct after a colliding key is cachedni/njsort invariances holdLocal:
test-expected-mi.Randtest-Concordance.R108/108 pass (2 CRANskips).
spelling::spell_check_package(vignettes = TRUE)clean.