Repository navigation
fix(cms): fold column index in queryFrequencyFast to match insert (non-pow2 cols) - #60
Merged
Merged
Conversation
The Count-Min Sketch INSERT path folds the per-row column index into range via `c %= s.Cols` (CountMinSketch.go:229-231), but the fast point-query path `queryFrequencyFast` (reached via QueryFrequency and FastEstimateWithHash) indexed `RowSlice(r)[c]` WITHOUT the same fold. For a non-power-of-two `cols`, the column mask spans the next power of two, so a masked index can exceed `cols`. Without the fold the query either panics (index out of range) or reads a different cell than insert wrote, breaking the CMS one-sided over-estimate guarantee. Insert and query disagreed on column placement. This applies the identical fold to queryFrequencyFast so insert and query always land on the same cell, matching the Rust asap_sketchlib CMS (which folds on both insert and query) and the already-correct estimateMatrixHash / QuerySum / QuerySum2 paths. Adds a regression test using non-power-of-two cols (17 and 2000) that inserts keys with known counts and asserts no panic and estimates >= true counts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In
sketches/CountMinSketch/CountMinSketch.go, the insert path folds the per-row column index back into range whencolsis not a power of two:but the query fast path
queryFrequencyFast(~L354-357) indexedRowSlice(r)[c]with no fold. BecausehashLayoutForColsrounds the mask up to the next power of two, a masked index can exceedcols, so querying a non-power-of-two CMS could panic (index out of range) and/or read the wrong cell.queryFrequencyFastwas the only unfolded read site (estimateMatrixHash,QuerySum,QuerySum2already fold).This also matters for cross-language parity: the Rust
asap_sketchlibCMS folds on both insert and query, so Go was the odd one out for non-pow2cols.Fix
Apply the identical fold in
queryFrequencyFastso insert and query always land on the same cell.Test
TestCMS_NonPowerOfTwoCols_QueryFold(cols = 17 and 2000, known counts):panic: index out of range [26] with length 17atCountMinSketch.go:357.QueryWithHash==FastEstimateWithHash, estimates ≥ true counts.go build ./...clean;go test ./...passes (pre-existingCocoSketch/cross_language-needs-XTEST_DIRfailures are unrelated, confirmed on untouchedmain).🤖 Generated with Claude Code