Repository navigation
feat(hydra)!: port Rust's Hydra with ASAPv1 codecs; remove proto/ and the protobuf harness - #95
Open
GordonYuanyc wants to merge 2 commits into
Open
GordonYuanyc wants to merge 2 commits into
GordonYuanyc wants to merge 2 commits into
Conversation
Hydra is a rows x cols grid of counters over a schema of named key columns. Update takes one value per column and fans the record out into its 2^D - 1 subpopulations, each encoded as escaped label:value pairs joined by ";" and placed by the matrix hash at seed index 6; QueryKey constrains any non-empty subset of the columns and returns the median of the rows. Grids with equal dimensions, counter variant and schema merge cell by cell. Counters are Count-Min, Count Sketch (i32, fast), HyperLogLog (P14 Ertl-MLE), KLL (f64) and UnivMon[K]. MarshalASAPv1 / UnmarshalASAPv1 encode the grid as kinds 0x07 0x00 to 0x07 0x04 and match the hydra_kll, hydra_cm, hydra_cs, hydra_hll and hydra_univmon fixtures. Cells encode and decode through their counters' own codecs and payload hooks. KLL gains K and M accessors. A UnivMon cell keyed by string takes a value with no bytes by its Hash in hex. Removes HydraConfig, the global counter, TopK / SetTopKEnabled, UpdateValue / UpdateWithInput / UpdateWithHash, ParallelUpdate, MultiHeadHydra, the gob SerializeToBytes / DeserializeHydraFromBytes pair, portable.go, the whole proto/ tree (no package imports it any more) and tests/cross_language, whose only producer was hydra.pb. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UnmarshalASAPv1 into a UnivMon grid keeps that grid's key type: an idle grid, whose counter_key_type is "u64", decodes into it, and a grid whose heap keys are of another type fails with an error naming both. A fresh Hydra still takes the type counter_key_type names. Merge names both Go key types when UnivMon grids differ in them. A Hydra Count Sketch grid whose counter_cols is not a power of two fails to decode with an error saying so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #94 (
asapv1-only/univmon). Part of the ASAPv1-only series: Go's Hydra is replaced by a port of Rust's and gets its ASAPv1 codecs for all five counter kinds; as the last importer of the protobuf code, it also deletesproto/and the protobuf cross-language harness.Changed (algorithm and API — breaking)
Go's Hydra was a different algorithm (unlabelled
;subkeys, an xorshift router, a global counter). It now follows Rust's: a key schema of up to 16 labels,label:valuesubkeys with\,:,;escaped, the matrix hash at seed index 6, median over rows, Rust's merge rules, and a closedHydraCounterset (KLL, Count-Min, Count Sketch, HLL, UnivMon). An independent reviewer's scratch Rust + Go programs produced byte-identical envelopes after realUpdatestreams for Count-Min, Count Sketch, HLL and UnivMon across packed-64, packed-128, per-row and non-power-of-two grids; queries match.HydraConfig,HydraDimension,MultiHeadHydraand its types,Pair,UpdateJob/ParallelUpdate,NewHydraWithDimensions,UpdateValue/UpdateWithHash/UpdateWithInput,GetCardinality/GetEntropy,TopK/SetTopKEnabled, the gob and portable formats.NewHydra(rows, cols, schema, counter)and the query methods changed signature.Added
MarshalASAPv1/UnmarshalASAPv1(kinds0x07 0x00–0x07 0x04); cells go through each counter's own payload hooks. All five Hydra fixtures passCheckGolden.Removed
proto/tree (no importers left) andtests/cross_language/(it only exercised Hydra's protobuf path);go mod tidy(glog dropped; protobuf stays indirect via prometheus).Notes / open questions
counter_colsare rejected (Go CountSketch limitation; documented, tested).CDF.Quantilein Go returns the fraction strictly below x, Rust at-or-below; tracked separately.Verification (Go 1.24.9)
go build,go vet,gofmt,go test ./...,go mod tidy -diffclean. Known flake:TestKLL_Reset_SubsequentInserts. Implemented and reviewed by separate agents; review findings are addressed in the second commit.🤖 Generated with Claude Code