Repository navigation
refactor!: ASAPv1 as the only serialization — integrate the series and remove proto, portable and MessagePackCodec - #178
Draft
GordonYuanyc wants to merge 49 commits into
Draft
GordonYuanyc wants to merge 49 commits into
GordonYuanyc wants to merge 49 commits into
Conversation
KLL and KLLDynamic serialize only as ASAPv1 (kind_ids 0x06 0x00 / 0x06 0x01).
- proto: reserve SketchEnvelope field 13 ("kll"); KLLState and CoinState stay
in kll.proto because hydra.proto's HydraCell still embeds them.
- portable: remove KllSketch/KllSketchData from the crate root, the
sketchlib_kll_* helpers, SketchlibKll, the value-offset codec and
KllProtoItems; portable::kll becomes a crate-private module holding only
the per-cell type HydraKllSketch uses.
- native: remove the MessagePackCodec shims for KLL<T> and KLLDynamic<T>.
- KLL::from_portable_state is removed.
- Tests: the portable KLL PBT laws and the e2e merge-and-wire rank test now
run on KLL<f64> through serialize_to_bytes; proto/portable-only tests go.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Count Sketch serializes as ASAPv1 only. Removes the portable CountSketch, CountSketchDelta and COUNT_SKETCH_TOPK_CAPACITY, the native MessagePackCodec shim for Count, the proto CountSketchDelta and CountSketchCell messages, and SketchEnvelope's count_sketch field (11, now reserved). CountSketchState stays: UnivMon and Hydra embed it. Tests that exercised Count Sketch through the removed formats are dropped (proto parity probe, msgpack round trip, xtest consumer case, portable e2e bound) or ported to the native Count (the four portable property laws). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CountMin serializes to ASAPv1 only. Removes the CountMinDelta and CountMinCell proto messages and SketchEnvelope's count_min field (10, now reserved), portable::countminsketch (CountMinSketch, CountMinSketchDelta, CountMinSketchWire, SketchlibCms and the sketchlib_cms_* helpers), the native MessagePackCodec shim for CountMin, and the cms_envelope_golden.hex proto golden. CountMinState stays as the nested Count-Min state of HydraCell and ElasticState. The Count-Min property laws that ran through the portable type now run on CountMin<Vector2D<f64>, FastPath> over string keys, with the ASAPv1 round trip and CountMin::apply_delta in the sequence law. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HyperLogLog serializes only as ASAPv1. Remove the portable HllSketch, HllSketchDelta and HllVariant with the sparse-register helpers, the native MessagePackCodec shims, the HLLDelta proto message and the SketchEnvelope hll field (12, now reserved). HyperLogLogState stays for HydraCell. Add HyperLogLogImpl::from_storage and HyperLogLogHIPImpl::from_storage, and check the three HLL ASAPv1 goldens through the native types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Count<Vector2D<i64>, FastPath> fed the 25 k-a..k-e inserts must hold the 15 signed cells Go's CountSketch wrote for the same stream. Also rewords the CSHeap portable module doc and the features.md proto list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CSHeap serializes through ASAPv1 only. Removes CountSketchWithHeap, CsHeapItem and the rest of message_pack_format::portable::countsketch_topk, with the e2e test and docs rows that covered them, and adds the CSHeap golden fixture csheap_i64_regular_2x4_strkeys (kind_id 0x0a 0x00). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CMSHeap serializes through ASAPv1 only. Removes message_pack_format::portable::countminsketch_topk (CountMinSketchWithHeap, CmsHeapItem, its wire DTOs and sketchlib_cms_heap_* helpers) and the crate-root re-exports. The e2e merge-and-wire test runs on CMSHeap over ASAPv1, and two CMSHeap golden fixtures are added (submodule bump). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DDSketch serializes through ASAPv1 only. Removes proto/ddsketch (DDSketchState, DDSketchDelta, DDSketchBucketDelta) and reserves SketchEnvelope field 14; removes message_pack_format::portable::ddsketch (DdSketch, DdSketchDelta, DDSKETCH_GROW_CHUNK, MAX_APPLY_DELTA_SPAN_BUCKETS) with its crate-root re-exports, and the native MessagePackCodec shim. Tests that exercised only the removed formats are deleted; the rest run against DDSketch and ASAPv1. Adds ASAPv1 golden fixtures ddsketch_positive_a001 (metadata version 1) and ddsketch_signed_a001 (metadata version 2) via the asapv1_golden submodule bump. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Make the HLL wire DTOs, kind_id constants and metadata builder private to wire.rs, state only what the remaining HLL proto messages are, shorten the HIP registers_of doc, and drop a stray proptest regression file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- asapv1_golden -> 13797db: kll_dynamic_f64_k200 and kll_dynamic_i64_k200 (kind_id 0x06 0x01, seed 42, no compaction; i64 items cross every msgpack integer width in both signs). - tests/asapv1_golden.rs: all four KLL cases read the golden payload by hand and compare levels, items and coin to the known state; compact decodes compare wire_levels/wire_items/wire_coin to the source; each KLL kind_id is rejected by the other variant's decoder. - KLLDynamic has seeded constructors: correct the wire.rs comment and the Q-KLL-SEED wording; fixture counts in the docs say seven kind_ids. - CHANGELOG: HydraKllSketch's cell type is reachable but unnameable; Added entry for the fixtures. - Re-wrap kll/wire.rs and e2e_testing_harness.md; drop kll.proto from the sketchlib.proto header list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds cm_f64_fast_path_places_keys_in_go_cells: the 40 non-zero cells of
sketchlib-go's 4x2048 Count-Min after 50 inserts of flow-{i % 10}, listed
inline, reproduced by CountMin<Vector2D<f64>, FastPath, DefaultXxHasher>.
Records in the CHANGELOG that Count-Min deltas, hh_keys and the envelope
sample_p have no ASAPv1 carrier, and drops stale CountMin mentions from
the sample_p proto doc and xtest_consumer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hydra serializes as ASAPv1 only. Remove the portable HydraKllSketch and HydraKllSketchWire, hydra/hydra.proto (HydraState, HydraCell, HydraCounterType) and SketchEnvelope's hydra field (16, now reserved), with the tests, example section and docs that used them. The two portable property laws give way to a KLL-counter ASAPv1 round-trip law on the native Hydra. Add ASAPv1 golden fixtures for all five Hydra counter variants (0x07 0x00-0x07 0x04), bumping asapv1_golden. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delete proto/cocosketch/cocosketch.proto (CocoSketchState), reserve the
SketchEnvelope oneof field 17 ("coco"), regenerate, and remove the Coco
section of the cross-language proto consumer. Coco serializes only as ASAPv1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump asapv1_golden to the commit adding coco_3x8.hex, a 3x8 bucket table set directly: values at both ends of every msgpack uint width, empty, 31-byte, 32-byte and multi-byte UTF-8 keys, and unoccupied buckets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delete proto/elasticsketch/elasticsketch.proto (ElasticState) and reserve
SketchEnvelope field 18 ("elastic"); regenerate src/proto/generated. Drop
the Elastic check from tests/xtest_consumer.rs. ASAPv1 (0x0b 0x00) is
Elastic's only serialization.
Add the elastic_4b_2x4 and elastic_4b_2x4_stale golden fixtures (bump
asapv1_golden) and their cases in tests/asapv1_golden.rs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The spec states the key comparison: signed keys by their two's-complement bit pattern read unsigned, floats by their bits, strings and bytes byte-wise with a proper prefix first. Three new golden fixtures pin i64 ties across zero and i64::MIN, string ties across case, length and prefix, and the empty-heap form (submodule bump). The docs defer the fixture list to the golden README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UnivMon serializes only as ASAPv1 (kind_id 0x10 0x00).
- proto: reserve SketchEnvelope field 15 ("univmon"); UnivMonState and
UnivMonLayer stay in univmon.proto because hydra.proto's HydraCell still
embeds them. Regenerated src/proto/generated.
- tests/xtest_consumer.rs: drop the univmon.pb check and its g-sum helpers.
- asapv1_golden -> e2086f0: univmon_str_l2_2x4_h2, univmon_i64_l2_2x4_h2 and
univmon_empty_l2_2x4_h2, built through L2HH::apply_delta and
HHHeap::update (no hashing). tests/asapv1_golden.rs checks serialize ==
golden, the hand-read metadata and payload, the decoded state and the
re-encode.
- docs: fixture counts say seven kind_ids; features.md and tests.md follow.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
State no fixture count; point at asapv1_golden/README.md's Coverage section. The cross-language proto test reads eight envelope files. The CHANGELOG entry notes that native Hydra routes subkeys differently from HydraKllSketch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…SAPv1 golden Removes `message_pack_format::native::countsketch_topk`, CountL2HH's `MessagePackCodec` pass-through; `serialize_to_bytes` / `deserialize_from_bytes` emit the same ASAPv1 bytes. CountL2HH has no proto message and no portable type. Bumps `asapv1_golden` to the commit adding `count_l2hh_2x4_seed7`, built from the sketch's serde form with signed cells across the msgpack integer widths, `l2` accumulators set apart from the cells, and seed index 7, and adds the golden test. Docs defer the fixture coverage to the golden README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… to the golden README A width that is not a power of two separates the spec's % cols fold from a mask fold. The fixture adds a 4-byte UTF-8 key. Docs no longer state how many kind_ids have fixtures; they point at the golden README's Coverage section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o the README Set the eviction flag on the Elastic fixtures' free bucket, a state a vacated bucket keeps, and bump asapv1_golden to the regenerated bytes. The spec and message_pack_format.md point to the golden README's Coverage section instead of counting fixtures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Add the empty-state DDSketch golden (ddsketch_empty_a001) and its test; build the golden mirror state with named fields. - Restore a core-only small-n probe that separates ceil(q*n) from floor(q*(n-1)). - Replace the one-variant DdRankConvention with dd_rank_index / dd_order_statistic. - Spec: alpha is carried as given, never re-derived from gamma. - Docs state current facts only; fixture counts defer to the golden README's Coverage section. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bump asapv1_golden to the commit adding hll_classic_p14, hll_ertl_mle_p14 and hll_hip_p14, built from fixed registers holding the maximum P14 rank, and check each in both directions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bumps `asapv1_golden` to the commit regenerating `count_l2hh_2x4_seed7` with an int16 cell and a uint32 `l2` accumulator, so the cells and accumulators reach all ten integer families. Drops the claim that `sketchlib-go` mirrors every fixture from the spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e docs
- asapv1_golden -> 946e5c5: univmon_{str,i64,empty}_l3_2x4_h5 replace the
l2_2x4_h2 files. 3 layers, 2 rows, 4 columns, heap 5, heap lengths
[3, 1, 2]; i64 keys add int16 and uint64.
- Fixture docs point at the golden README's Coverage section instead of a
count; spec 3.15 states the empty pyramid's key_type is "u64".
- features.md: SketchEnvelope's sketches, with UnivMonState only in HydraCell.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ult; drop the portable format
SetAggregator (0x08 0x00) and DeltaResult (0x09 0x00) get ASAPv1 payloads
(spec §3.21, §3.22): metadata is {metadata_version: 1}; the payloads are
[values] and [added, removed], each an array of str in strictly ascending
UTF-8 byte order. DeltaResult refuses a key in both sets on both sides.
Both types move to sketches::set_aggregator with serialize_to_bytes /
deserialize_from_bytes; the crate-root re-exports are kept. The portable
modules, their MessagePackCodec impls and DeltaResult's serde derives are
removed. Adds four golden fixtures and bumps asapv1_golden.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…8, widen the string fixture Adds SetAggregator and DeltaResult to the Q-NOHASH and Q-ORDER decisions, requires every member to be valid UTF-8, states the encoder's one encoding per set, and numbers §3.22's rules in full. The DeltaResult doc states the disjointness serializing enforces, and read_canonical checks neighbours before collecting. set_aggregator_strings gains a 31-byte fixstr and a proper-prefix chain; asapv1_golden is bumped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/features.md # docs/message_pack_format.md # docs/tests.md # examples/accuracy_probe.rs # proto/sketchlib.proto # src/lib.rs # src/message_pack_format/native/mod.rs # src/message_pack_format/portable/mod.rs # src/proto/generated/sketchlib.v1.rs # tests/asapv1_golden.rs # tests/msgpack_compat.rs
…y/cleanup # Conflicts: # CHANGELOG.md # docs/features.md # docs/library_map.md # docs/message_pack_format.md # docs/tests.md # examples/accuracy_probe.rs # proto/sketchlib.proto # src/lib.rs # src/message_pack_format/portable/mod.rs # src/proto/generated/sketchlib.v1.rs # tests/msgpack_compat.rs # tests/sketches_go_parity_probe.rs # tests/xtest_consumer.rs
…/cleanup # Conflicts: # CHANGELOG.md # docs/features.md # docs/message_pack_format.md # docs/tests.md # proto/countminsketch/countminsketch.proto # proto/sketchlib.proto # src/common/hashspec.rs # src/lib.rs # src/message_pack_format/native/mod.rs # src/message_pack_format/portable/countminsketch.rs # src/message_pack_format/portable/countsketch.rs # src/message_pack_format/portable/mod.rs # src/proto/generated/sketchlib.v1.rs # tests/e2e/frequency.rs # tests/msgpack_compat.rs # tests/sketches_go_parity_probe.rs # tests/xtest_consumer.rs
…-only/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/message_pack_format.md # docs/tests.md # src/lib.rs # src/message_pack_format/portable/countminsketch_topk.rs # src/message_pack_format/portable/countsketch_topk.rs # src/message_pack_format/portable/mod.rs # tests/asapv1_golden.rs # tests/e2e/composition.rs # tests/msgpack_compat.rs
…only/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/message_pack_format.md # docs/tests.md # src/lib.rs # src/message_pack_format/portable/countsketch_topk.rs # src/message_pack_format/portable/mod.rs # tests/asapv1_golden.rs # tests/e2e/composition.rs
…1-only/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/features.md # docs/message_pack_format.md # docs/tests.md # examples/accuracy_probe.rs # proto/sketchlib.proto # src/lib.rs # src/message_pack_format/native/mod.rs # src/message_pack_format/portable/mod.rs # src/proto/generated/sketchlib.v1.rs # tests/asapv1_golden.rs # tests/conformance_kit.rs # tests/msgpack_compat.rs # tests/sketches_go_parity_probe.rs # tests/xtest_consumer.rs
…nly/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/features.md # docs/message_pack_format.md # docs/tests.md # examples/accuracy_probe.rs # proto/sketchlib.proto # src/lib.rs # src/message_pack_format/portable/kll.rs # src/message_pack_format/portable/mod.rs # src/proto/generated/sketchlib.v1.rs # tests/PBT_COVERAGE.md # tests/asapv1_golden.rs # tests/e2e/quantiles.rs # tests/msgpack_compat.rs # tests/xtest_consumer.rs
…-only/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/features.md # docs/message_pack_format.md # docs/tests.md # proto/sketchlib.proto # src/proto/generated/sketchlib.v1.rs # tests/asapv1_golden.rs # tests/xtest_consumer.rs
…ly/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/features.md # docs/message_pack_format.md # docs/tests.md # proto/sketchlib.proto # src/proto/generated/sketchlib.v1.rs # tests/asapv1_golden.rs # tests/xtest_consumer.rs # tools/gen-proto/src/main.rs
…-only/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/features.md # docs/message_pack_format.md # docs/tests.md # proto/sketchlib.proto # src/proto/generated/sketchlib.v1.rs # tests/asapv1_golden.rs # tests/xtest_consumer.rs # tools/gen-proto/src/main.rs
…v1-only/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/message_pack_format.md # docs/tests.md # src/message_pack_format/native/mod.rs # tests/asapv1_golden.rs
…only/cleanup # Conflicts: # CHANGELOG.md # docs/asapv1_wire_format.md # docs/tests.md # src/lib.rs # src/message_pack_format/portable/mod.rs # tests/PBT_COVERAGE.md # tests/asapv1_golden.rs # tests/msgpack_compat.rs
Drop what no merged branch still uses: the empty portable probe in accuracy_probe, the unused MessagePackCodec import in e2e/composition, the removed sketch_state field in the sampling test, and the docs/tests.md section for the deleted parity probe. Share one envelope splitter between the KLL and UnivMon golden checks, and recount the property-test totals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every sketch serializes through serialize_to_bytes / deserialize_from_bytes, so the codec trait, its error type, the last native shim (KMV) and the portable sampling helpers have no ASAPv1 role left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No sketch has a SketchEnvelope field any more, so the proto sources, the vendored prost output, the gen-proto tool, its CI drift job and the prost and bytes dependencies go with them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
message_pack_format held only the crate-private envelope and wire-key code; it now lives in src/asapv1/ and the public module is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Delete the message_pack_format page (library_map covers the code layout), drop the portable and proto sections from the wire-format spec, the library map and the e2e harness notes, and keep only what the Unreleased changelog entries still remove. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final PR of the ASAPv1-only series. It contains all 13 per-sketch PRs (#165–#177, merged
--no-ffat their exact tips, conflicts resolved) and then removes the shared legacy plumbing. After it, ASAPv1 is the only serialization inasap_sketchlib.Draft: it inherits the open questions of the per-sketch drafts, plus the design questions below.
Integration
reservedline kept; generated proto regenerated, never hand-merged; CHANGELOG entries all kept; fixture-coverage sentences defer to the golden README.tests/asapv1_golden.rs: 40 tests, exactly the union of every branch's; every fixture in theasapv1_goldensubmodule (at goldenmain) is referenced.Removed (shared plumbing)
MessagePackCodec, itsError,message_pack_format::{native, portable}including the sampling helpers (effective_sample_p,sample_p_or_default,rescale_count,rescale_count_with_env,is_quantile_scale_invariant).proto/,src/proto.rs,src/proto/generated/,tools/gen-proto/,pub mod proto, theprostandbytesdependencies, the CI proto-drift job and docs path triggers.xxhash-rust(no users left) and the unused publiccommon::hashspecmodule.tests/msgpack_compat.rs,tests/xtest_consumer.rs,tests/sketches_go_parity_probe.rs(each emptied by the merges),docs/message_pack_format.md.Moved
envelope.rs,wire_key.rs) now lives in a privatesrc/asapv1/module; the publicmessage_pack_formatmodule is gone.Design questions
SketchEnvelope's oneof gone there is no public way to read an ASAPv1 buffer'skind_id; a consumer holding bytes of unknown kind must try decoders in turn. Add a small publicpeek_kind_id(&[u8])/ kind registry?Verification (rustc 1.99)
cargo fmt --check; clippy-D warningsfor all four CI feature sets;RUSTDOCFLAGS="-D warnings" cargo doc; tests:--all-features --lib991,--test asapv1_golden40,--test pbt432,--test e2e599,--doc21, default-features--lib858 — all passing. Implemented and reviewed by separate agents; review findings are addressed in the last two commits.🤖 Generated with Claude Code