Skip to content

refactor!: ASAPv1 as the only serialization — integrate the series and remove proto, portable and MessagePackCodec - #178

Draft
GordonYuanyc wants to merge 49 commits into
mainfrom
asapv1-only/cleanup
Draft

GordonYuanyc wants to merge 49 commits into
mainfrom
asapv1-only/cleanup

Conversation

@GordonYuanyc

Copy link
Copy Markdown
Collaborator

Final PR of the ASAPv1-only series. It contains all 13 per-sketch PRs (#165–#177, merged --no-ff at their exact tips, conflicts resolved) and then removes the shared legacy plumbing. After it, ASAPv1 is the only serialization in asap_sketchlib.

Draft: it inherits the open questions of the per-sketch drafts, plus the design questions below.

Integration

  • Merge order: kll, hll, cms, cs, cmsheap, csheap, ddsketch, hydra, univmon, coco, elastic, countl2hh, setagg. Every reserved line kept; generated proto regenerated, never hand-merged; CHANGELOG entries all kept; fixture-coverage sentences defer to the golden README.
  • tests/asapv1_golden.rs: 40 tests, exactly the union of every branch's; every fixture in the asapv1_golden submodule (at golden main) is referenced.
  • An independent reviewer checked that each merge kept every line its branch added.

Removed (shared plumbing)

  • MessagePackCodec, its Error, message_pack_format::{native, portable} including the sampling helpers (effective_sample_p, sample_p_or_default, rescale_count, rescale_count_with_env, is_quantile_scale_invariant).
  • The protobuf layer: proto/, src/proto.rs, src/proto/generated/, tools/gen-proto/, pub mod proto, the prost and bytes dependencies, the CI proto-drift job and docs path triggers.
  • xxhash-rust (no users left) and the unused public common::hashspec module.
  • tests/msgpack_compat.rs, tests/xtest_consumer.rs, tests/sketches_go_parity_probe.rs (each emptied by the merges), docs/message_pack_format.md.

Moved

  • The ASAPv1 framing (envelope.rs, wire_key.rs) now lives in a private src/asapv1/ module; the public message_pack_format module is gone.

Design questions

  1. Dispatch on kind: with SketchEnvelope's oneof gone there is no public way to read an ASAPv1 buffer's kind_id; a consumer holding bytes of unknown kind must try decoders in turn. Add a small public peek_kind_id(&[u8]) / kind registry?
  2. Sampling probability: ASAPv1 has no field for p, and the rescale helpers are gone. Accept, or add a metadata key in a follow-up?
  3. Landing order: merging this PR alone instead of the 13 avoids a rebase if those were squash-merged first.

Verification (rustc 1.99)

cargo fmt --check; clippy -D warnings for all four CI feature sets; RUSTDOCFLAGS="-D warnings" cargo doc; tests: --all-features --lib 991, --test asapv1_golden 40, --test pbt 432, --test e2e 599, --doc 21, default-features --lib 858 — all passing. Implemented and reviewed by separate agents; review findings are addressed in the last two commits.

🤖 Generated with Claude Code

GordonYuanyc and others added 30 commits October 2, 2026 00:40
KLL and KLLDynamic serialize only as ASAPv1 (kind_ids 0x06 0x00 / 0x06 0x01).

- proto: reserve SketchEnvelope field 13 ("kll"); KLLState and CoinState stay
  in kll.proto because hydra.proto's HydraCell still embeds them.
- portable: remove KllSketch/KllSketchData from the crate root, the
  sketchlib_kll_* helpers, SketchlibKll, the value-offset codec and
  KllProtoItems; portable::kll becomes a crate-private module holding only
  the per-cell type HydraKllSketch uses.
- native: remove the MessagePackCodec shims for KLL<T> and KLLDynamic<T>.
- KLL::from_portable_state is removed.
- Tests: the portable KLL PBT laws and the e2e merge-and-wire rank test now
  run on KLL<f64> through serialize_to_bytes; proto/portable-only tests go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Count Sketch serializes as ASAPv1 only. Removes the portable CountSketch,
CountSketchDelta and COUNT_SKETCH_TOPK_CAPACITY, the native
MessagePackCodec shim for Count, the proto CountSketchDelta and
CountSketchCell messages, and SketchEnvelope's count_sketch field (11,
now reserved). CountSketchState stays: UnivMon and Hydra embed it.

Tests that exercised Count Sketch through the removed formats are dropped
(proto parity probe, msgpack round trip, xtest consumer case, portable e2e
bound) or ported to the native Count (the four portable property laws).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CountMin serializes to ASAPv1 only. Removes the CountMinDelta and
CountMinCell proto messages and SketchEnvelope's count_min field (10,
now reserved), portable::countminsketch (CountMinSketch,
CountMinSketchDelta, CountMinSketchWire, SketchlibCms and the
sketchlib_cms_* helpers), the native MessagePackCodec shim for CountMin,
and the cms_envelope_golden.hex proto golden. CountMinState stays as the
nested Count-Min state of HydraCell and ElasticState.

The Count-Min property laws that ran through the portable type now run
on CountMin<Vector2D<f64>, FastPath> over string keys, with the ASAPv1
round trip and CountMin::apply_delta in the sequence law.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HyperLogLog serializes only as ASAPv1. Remove the portable HllSketch,
HllSketchDelta and HllVariant with the sparse-register helpers, the native
MessagePackCodec shims, the HLLDelta proto message and the SketchEnvelope
hll field (12, now reserved). HyperLogLogState stays for HydraCell.

Add HyperLogLogImpl::from_storage and HyperLogLogHIPImpl::from_storage, and
check the three HLL ASAPv1 goldens through the native types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Count<Vector2D<i64>, FastPath> fed the 25 k-a..k-e inserts must hold the
15 signed cells Go's CountSketch wrote for the same stream. Also rewords
the CSHeap portable module doc and the features.md proto list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CSHeap serializes through ASAPv1 only. Removes CountSketchWithHeap,
CsHeapItem and the rest of message_pack_format::portable::countsketch_topk,
with the e2e test and docs rows that covered them, and adds the CSHeap
golden fixture csheap_i64_regular_2x4_strkeys (kind_id 0x0a 0x00).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CMSHeap serializes through ASAPv1 only. Removes
message_pack_format::portable::countminsketch_topk (CountMinSketchWithHeap,
CmsHeapItem, its wire DTOs and sketchlib_cms_heap_* helpers) and the
crate-root re-exports. The e2e merge-and-wire test runs on CMSHeap over
ASAPv1, and two CMSHeap golden fixtures are added (submodule bump).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DDSketch serializes through ASAPv1 only. Removes proto/ddsketch
(DDSketchState, DDSketchDelta, DDSketchBucketDelta) and reserves
SketchEnvelope field 14; removes message_pack_format::portable::ddsketch
(DdSketch, DdSketchDelta, DDSKETCH_GROW_CHUNK,
MAX_APPLY_DELTA_SPAN_BUCKETS) with its crate-root re-exports, and the
native MessagePackCodec shim. Tests that exercised only the removed
formats are deleted; the rest run against DDSketch and ASAPv1.

Adds ASAPv1 golden fixtures ddsketch_positive_a001 (metadata version 1)
and ddsketch_signed_a001 (metadata version 2) via the asapv1_golden
submodule bump.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Make the HLL wire DTOs, kind_id constants and metadata builder private to
wire.rs, state only what the remaining HLL proto messages are, shorten the
HIP registers_of doc, and drop a stray proptest regression file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- asapv1_golden -> 13797db: kll_dynamic_f64_k200 and kll_dynamic_i64_k200
  (kind_id 0x06 0x01, seed 42, no compaction; i64 items cross every msgpack
  integer width in both signs).
- tests/asapv1_golden.rs: all four KLL cases read the golden payload by hand
  and compare levels, items and coin to the known state; compact decodes
  compare wire_levels/wire_items/wire_coin to the source; each KLL kind_id is
  rejected by the other variant's decoder.
- KLLDynamic has seeded constructors: correct the wire.rs comment and the
  Q-KLL-SEED wording; fixture counts in the docs say seven kind_ids.
- CHANGELOG: HydraKllSketch's cell type is reachable but unnameable; Added
  entry for the fixtures.
- Re-wrap kll/wire.rs and e2e_testing_harness.md; drop kll.proto from the
  sketchlib.proto header list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds cm_f64_fast_path_places_keys_in_go_cells: the 40 non-zero cells of
sketchlib-go's 4x2048 Count-Min after 50 inserts of flow-{i % 10}, listed
inline, reproduced by CountMin<Vector2D<f64>, FastPath, DefaultXxHasher>.
Records in the CHANGELOG that Count-Min deltas, hh_keys and the envelope
sample_p have no ASAPv1 carrier, and drops stale CountMin mentions from
the sample_p proto doc and xtest_consumer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hydra serializes as ASAPv1 only. Remove the portable HydraKllSketch and
HydraKllSketchWire, hydra/hydra.proto (HydraState, HydraCell,
HydraCounterType) and SketchEnvelope's hydra field (16, now reserved), with
the tests, example section and docs that used them. The two portable
property laws give way to a KLL-counter ASAPv1 round-trip law on the
native Hydra.

Add ASAPv1 golden fixtures for all five Hydra counter variants
(0x07 0x00-0x07 0x04), bumping asapv1_golden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delete proto/cocosketch/cocosketch.proto (CocoSketchState), reserve the
SketchEnvelope oneof field 17 ("coco"), regenerate, and remove the Coco
section of the cross-language proto consumer. Coco serializes only as ASAPv1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump asapv1_golden to the commit adding coco_3x8.hex, a 3x8 bucket table set
directly: values at both ends of every msgpack uint width, empty, 31-byte,
32-byte and multi-byte UTF-8 keys, and unoccupied buckets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delete proto/elasticsketch/elasticsketch.proto (ElasticState) and reserve
SketchEnvelope field 18 ("elastic"); regenerate src/proto/generated. Drop
the Elastic check from tests/xtest_consumer.rs. ASAPv1 (0x0b 0x00) is
Elastic's only serialization.

Add the elastic_4b_2x4 and elastic_4b_2x4_stale golden fixtures (bump
asapv1_golden) and their cases in tests/asapv1_golden.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spec states the key comparison: signed keys by their two's-complement
bit pattern read unsigned, floats by their bits, strings and bytes
byte-wise with a proper prefix first. Three new golden fixtures pin i64
ties across zero and i64::MIN, string ties across case, length and
prefix, and the empty-heap form (submodule bump). The docs defer the
fixture list to the golden README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UnivMon serializes only as ASAPv1 (kind_id 0x10 0x00).

- proto: reserve SketchEnvelope field 15 ("univmon"); UnivMonState and
  UnivMonLayer stay in univmon.proto because hydra.proto's HydraCell still
  embeds them. Regenerated src/proto/generated.
- tests/xtest_consumer.rs: drop the univmon.pb check and its g-sum helpers.
- asapv1_golden -> e2086f0: univmon_str_l2_2x4_h2, univmon_i64_l2_2x4_h2 and
  univmon_empty_l2_2x4_h2, built through L2HH::apply_delta and
  HHHeap::update (no hashing). tests/asapv1_golden.rs checks serialize ==
  golden, the hand-read metadata and payload, the decoded state and the
  re-encode.
- docs: fixture counts say seven kind_ids; features.md and tests.md follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
State no fixture count; point at asapv1_golden/README.md's Coverage
section. The cross-language proto test reads eight envelope files. The
CHANGELOG entry notes that native Hydra routes subkeys differently from
HydraKllSketch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SAPv1 golden

Removes `message_pack_format::native::countsketch_topk`, CountL2HH's
`MessagePackCodec` pass-through; `serialize_to_bytes` /
`deserialize_from_bytes` emit the same ASAPv1 bytes. CountL2HH has no proto
message and no portable type.

Bumps `asapv1_golden` to the commit adding `count_l2hh_2x4_seed7`, built from
the sketch's serde form with signed cells across the msgpack integer widths,
`l2` accumulators set apart from the cells, and seed index 7, and adds the
golden test. Docs defer the fixture coverage to the golden README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… to the golden README

A width that is not a power of two separates the spec's % cols fold from a
mask fold. The fixture adds a 4-byte UTF-8 key. Docs no longer state how many
kind_ids have fixtures; they point at the golden README's Coverage section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o the README

Set the eviction flag on the Elastic fixtures' free bucket, a state a
vacated bucket keeps, and bump asapv1_golden to the regenerated bytes.
The spec and message_pack_format.md point to the golden README's Coverage
section instead of counting fixtures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Add the empty-state DDSketch golden (ddsketch_empty_a001) and its test;
  build the golden mirror state with named fields.
- Restore a core-only small-n probe that separates ceil(q*n) from
  floor(q*(n-1)).
- Replace the one-variant DdRankConvention with dd_rank_index /
  dd_order_statistic.
- Spec: alpha is carried as given, never re-derived from gamma.
- Docs state current facts only; fixture counts defer to the golden
  README's Coverage section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump asapv1_golden to the commit adding hll_classic_p14, hll_ertl_mle_p14
and hll_hip_p14, built from fixed registers holding the maximum P14 rank,
and check each in both directions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bumps `asapv1_golden` to the commit regenerating `count_l2hh_2x4_seed7` with
an int16 cell and a uint32 `l2` accumulator, so the cells and accumulators
reach all ten integer families. Drops the claim that `sketchlib-go` mirrors
every fixture from the spec.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e docs

- asapv1_golden -> 946e5c5: univmon_{str,i64,empty}_l3_2x4_h5 replace the
  l2_2x4_h2 files. 3 layers, 2 rows, 4 columns, heap 5, heap lengths
  [3, 1, 2]; i64 keys add int16 and uint64.
- Fixture docs point at the golden README's Coverage section instead of a
  count; spec 3.15 states the empty pyramid's key_type is "u64".
- features.md: SketchEnvelope's sketches, with UnivMonState only in HydraCell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ult; drop the portable format

SetAggregator (0x08 0x00) and DeltaResult (0x09 0x00) get ASAPv1 payloads
(spec §3.21, §3.22): metadata is {metadata_version: 1}; the payloads are
[values] and [added, removed], each an array of str in strictly ascending
UTF-8 byte order. DeltaResult refuses a key in both sets on both sides.

Both types move to sketches::set_aggregator with serialize_to_bytes /
deserialize_from_bytes; the crate-root re-exports are kept. The portable
modules, their MessagePackCodec impls and DeltaResult's serde derives are
removed. Adds four golden fixtures and bumps asapv1_golden.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…8, widen the string fixture

Adds SetAggregator and DeltaResult to the Q-NOHASH and Q-ORDER decisions,
requires every member to be valid UTF-8, states the encoder's one encoding
per set, and numbers §3.22's rules in full. The DeltaResult doc states the
disjointness serializing enforces, and read_canonical checks neighbours
before collecting. set_aggregator_strings gains a 31-byte fixstr and a
proper-prefix chain; asapv1_golden is bumped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/features.md
#	docs/message_pack_format.md
#	docs/tests.md
#	examples/accuracy_probe.rs
#	proto/sketchlib.proto
#	src/lib.rs
#	src/message_pack_format/native/mod.rs
#	src/message_pack_format/portable/mod.rs
#	src/proto/generated/sketchlib.v1.rs
#	tests/asapv1_golden.rs
#	tests/msgpack_compat.rs
GordonYuanyc and others added 19 commits October 2, 2026 01:32
…y/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/features.md
#	docs/library_map.md
#	docs/message_pack_format.md
#	docs/tests.md
#	examples/accuracy_probe.rs
#	proto/sketchlib.proto
#	src/lib.rs
#	src/message_pack_format/portable/mod.rs
#	src/proto/generated/sketchlib.v1.rs
#	tests/msgpack_compat.rs
#	tests/sketches_go_parity_probe.rs
#	tests/xtest_consumer.rs
…/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/features.md
#	docs/message_pack_format.md
#	docs/tests.md
#	proto/countminsketch/countminsketch.proto
#	proto/sketchlib.proto
#	src/common/hashspec.rs
#	src/lib.rs
#	src/message_pack_format/native/mod.rs
#	src/message_pack_format/portable/countminsketch.rs
#	src/message_pack_format/portable/countsketch.rs
#	src/message_pack_format/portable/mod.rs
#	src/proto/generated/sketchlib.v1.rs
#	tests/e2e/frequency.rs
#	tests/msgpack_compat.rs
#	tests/sketches_go_parity_probe.rs
#	tests/xtest_consumer.rs
…-only/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/message_pack_format.md
#	docs/tests.md
#	src/lib.rs
#	src/message_pack_format/portable/countminsketch_topk.rs
#	src/message_pack_format/portable/countsketch_topk.rs
#	src/message_pack_format/portable/mod.rs
#	tests/asapv1_golden.rs
#	tests/e2e/composition.rs
#	tests/msgpack_compat.rs
…only/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/message_pack_format.md
#	docs/tests.md
#	src/lib.rs
#	src/message_pack_format/portable/countsketch_topk.rs
#	src/message_pack_format/portable/mod.rs
#	tests/asapv1_golden.rs
#	tests/e2e/composition.rs
…1-only/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/features.md
#	docs/message_pack_format.md
#	docs/tests.md
#	examples/accuracy_probe.rs
#	proto/sketchlib.proto
#	src/lib.rs
#	src/message_pack_format/native/mod.rs
#	src/message_pack_format/portable/mod.rs
#	src/proto/generated/sketchlib.v1.rs
#	tests/asapv1_golden.rs
#	tests/conformance_kit.rs
#	tests/msgpack_compat.rs
#	tests/sketches_go_parity_probe.rs
#	tests/xtest_consumer.rs
…nly/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/features.md
#	docs/message_pack_format.md
#	docs/tests.md
#	examples/accuracy_probe.rs
#	proto/sketchlib.proto
#	src/lib.rs
#	src/message_pack_format/portable/kll.rs
#	src/message_pack_format/portable/mod.rs
#	src/proto/generated/sketchlib.v1.rs
#	tests/PBT_COVERAGE.md
#	tests/asapv1_golden.rs
#	tests/e2e/quantiles.rs
#	tests/msgpack_compat.rs
#	tests/xtest_consumer.rs
…-only/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/features.md
#	docs/message_pack_format.md
#	docs/tests.md
#	proto/sketchlib.proto
#	src/proto/generated/sketchlib.v1.rs
#	tests/asapv1_golden.rs
#	tests/xtest_consumer.rs
…ly/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/features.md
#	docs/message_pack_format.md
#	docs/tests.md
#	proto/sketchlib.proto
#	src/proto/generated/sketchlib.v1.rs
#	tests/asapv1_golden.rs
#	tests/xtest_consumer.rs
#	tools/gen-proto/src/main.rs
…-only/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/features.md
#	docs/message_pack_format.md
#	docs/tests.md
#	proto/sketchlib.proto
#	src/proto/generated/sketchlib.v1.rs
#	tests/asapv1_golden.rs
#	tests/xtest_consumer.rs
#	tools/gen-proto/src/main.rs
…v1-only/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/message_pack_format.md
#	docs/tests.md
#	src/message_pack_format/native/mod.rs
#	tests/asapv1_golden.rs
…only/cleanup

# Conflicts:
#	CHANGELOG.md
#	docs/asapv1_wire_format.md
#	docs/tests.md
#	src/lib.rs
#	src/message_pack_format/portable/mod.rs
#	tests/PBT_COVERAGE.md
#	tests/asapv1_golden.rs
#	tests/msgpack_compat.rs
Drop what no merged branch still uses: the empty portable probe in
accuracy_probe, the unused MessagePackCodec import in e2e/composition, the
removed sketch_state field in the sampling test, and the docs/tests.md
section for the deleted parity probe. Share one envelope splitter between
the KLL and UnivMon golden checks, and recount the property-test totals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every sketch serializes through serialize_to_bytes / deserialize_from_bytes,
so the codec trait, its error type, the last native shim (KMV) and the
portable sampling helpers have no ASAPv1 role left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No sketch has a SketchEnvelope field any more, so the proto sources, the
vendored prost output, the gen-proto tool, its CI drift job and the prost
and bytes dependencies go with them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
message_pack_format held only the crate-private envelope and wire-key code;
it now lives in src/asapv1/ and the public module is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delete the message_pack_format page (library_map covers the code layout),
drop the portable and proto sections from the wire-format spec, the library
map and the e2e harness notes, and keep only what the Unreleased changelog
entries still remove.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant