Skip to content

feat(countsketch)!: ASAPv1 codec with counter type and mode; drop the protobuf, msgpack and delta formats - #89

Draft
GordonYuanyc wants to merge 2 commits into
asapv1-only/infrafrom
asapv1-only/cs
Draft

GordonYuanyc wants to merge 2 commits into
asapv1-only/infrafrom
asapv1-only/cs

Conversation

@GordonYuanyc

Copy link
Copy Markdown
Collaborator

Stacked on #85 (asapv1-only/infra). Part of the ASAPv1-only series: Go's Count Sketch gets an ASAPv1 codec and loses its protobuf, msgpack and delta formats.

Draft because of the open questions below.

Added

  • CounterType (CounterFloat64 default, CounterInt32, CounterInt64) and Mode (ModeFast default, ModeRegular) on CountSketch, and MarshalASAPv1 / UnmarshalASAPv1 (kind 0x04 0x00). All three cs_* fixtures pass CheckGolden.
  • Regular mode is real behaviour: every method that has the key bytes (Update*, UpdateString*, the sampled/at-rows/GOS variants, Estimate*, ProcessInput/ColForRow/SignForRow/UpdateCell with key bytes, and the top-k rebuilds) derives cells through one mode-aware path that matches Rust's regular path or Rust's fast path on every geometry (packed-64, packed-128, per-row). Confirmed byte-identical to Rust for both modes.
  • The encoder refuses sketches whose bytes would lie (decided during review; easy to revisit): methods that only get a precomputed 64-bit hash can't reproduce regular mode or fast mode outside packed-64, so a write through them on such a sketch sets a sticky flag (kept by Merge, cleared by Reset) and MarshalASAPv1 refuses.
  • Float storage with integer counter types: encode refuses non-integral or out-of-range cells (float64-counter sketches can't be encoded at all); decode refuses i64 values float64 can't hold exactly.

Changed

  • In fast mode, key-based methods now place keys like Update and Rust on matrices outside the packed-64 layout (they used Go-only mix64 hashing there).

Removed

  • SerializePortable, SerializeProtoBytes, DeserializeCountSketchFromProtoBytes, SerializeMsgpack / DeserializeMsgpack, SerializeDelta / DeserializeDelta; asapmsgpack CS full, cell-delta and sparse-delta codecs; proto CountSketchDelta, CountSketchCell, and count_sketch in both envelopes (reserved 11; reserved "count_sketch";). CountSketchState stays for UnivMon and Hydra (Hydra's cell now builds it locally).

Open questions

  1. Deltas: ASAPCollector sends Count Sketch as proto full state plus CountSketchDelta; both are removed and ASAPv1 has no delta.
  2. f64 counters: the collector's sketches are float64 (weighted / 1/p-sampled); they can't be encoded until they move to integer counters or the spec gains an "f64" Count Sketch counter type.
  3. Power-of-two cols: Go decodes only power-of-two widths, so a valid Rust sketch with e.g. cols = 3 is rejected (documented on UnmarshalASAPv1, tested with real Rust bytes).
  4. Gaps: cell-level writes (IncrCell, SetCell, MergeDelta) don't set the refusal flag; the OctoSketch per-cell path now hashes key bytes on every call (not benchmarked).

Verification (Go 1.24.9)

go build, go vet, gofmt, go test ./.... Known failures also on main: CocoSketch tests, TestKLL_Reset_SubsequentInserts (flaky), tests/cross_language without XTEST_DIR. Implemented and reviewed by separate agents; review findings are addressed in the second commit.

🤖 Generated with Claude Code

GordonYuanyc and others added 2 commits October 2, 2026 01:06
… formats

CountSketch gains CounterType (float64, i32, i64) and Mode (fast,
regular). Update, UpdateWeight and Estimate follow the regular derivation
when Mode is ModeRegular. MarshalASAPv1 / UnmarshalASAPv1 carry kind
0x04 0x00; a float64-counter sketch has no ASAPv1 encoding and fails to
marshal.

Removed: SerializePortable, SerializeProtoBytes,
DeserializeCountSketchFromProtoBytes, SerializeMsgpack, DeserializeMsgpack,
SerializeDelta, DeserializeDelta, the asapmsgpack plain, sparse-delta and
cell-delta Count Sketch codecs, the CountSketchDelta / CountSketchCell
protos and the count_sketch fields of both proto envelopes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UpdateString, UpdateStringGOS, the sampled and pre-admitted row updates,
EstimateStringCount, ProcessInput, ColForRow / SignForRow and the top-k
rebuilds in Merge, ApplyDelta and the wire heap now derive rows from the key
bytes under the sketch's Mode, using the key's matrix hash on the fast path
for every geometry. A write from a precomputed hash alone that does not
follow Mode marks the sketch, Merge carries the mark, Reset clears it, and
MarshalASAPv1 refuses a marked sketch.

Also: document the power-of-two cols limit on UnmarshalASAPv1, reserve the
count_sketch name in both envelopes, and drop stale CountSketchDelta
references from the Count-Min proto and delta docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant