Conversation
The store now holds Planner's summary kernels directly. asap_summary_state keeps only their stored tags and byte encodings (stored_state::codec), edge wire decoding (stored_state::decoders), delta reconstruction and readout binding, and no second kernel implementation. - Delete the copied summary_kernels, the backend updater factory, physical.rs boundary conversions and the legacy per-statistic trait surface. - Current stored formats are unchanged; retired exact tags and the native SumAccumulatorV1 codec fail with a named rejection. - Edge-sampled frames are rejected: Planner kernels carry no sample_p. - OTLP SketchEnvelope attributes decode into Planner kernels. - Unkeyed Planner exact MIN/MAX panes feed the MIN/MAX rollup again. - UnivMon keeps a marked shim until Planner exposes its sketch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A late correction, replacement or gap was dropped from the rollup, and the next contiguous pane re-validated it, so the rollup could serve an extremum that ignored the correction. Such a pane now invalidates the series' rollup permanently and readers fall back to the exact panes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
changed the base branch from
fix/topk-by-heap-groups
to
feat/per-state-mixed-placement
September 30, 2026 14:38
zzylol
force-pushed
the
refactor/codecs-over-planner-kernels
branch
from
September 30, 2026 14:38
06236d7 to
1737348
Compare
This was referenced Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #807.
Why
#793 copied Planner's old summary kernels into
crates/asap_summary_stateso the store could keep their byte formats. That left two implementations of every kernel (Sum/Min/Max/keyed exact states, KLL, DDSketch, HLL, CMS/CountSketch with and without heaps, UnivMon, Hydra, the updater factory) plusphysical.rsconverting between them wherever stored state entered or left a Planner DAG. The approved layering is that Planner owns computation and the backend owns storage formats, decoding and readout binding.What
asap_physical_operators::AggregateCore) directly.stored_state::codecis the one backend view of those states. TheStoredStateextension trait gives each one a persisted tag, anAggregationTypeand a byte encoding.decode(tag, bytes)returns a Planner kernel.empty_likereplaces the old window-reset path.check_storableguards the store's append boundary, so a state without a codec is refused instead of being written as empty bytes.stored_state::decodersholds the edge wire formats (protobuf envelopes, msgpack frames, delta frames, the Sum payload) as functions over sketchlib types. The ingest and read paths now share these functions. Before, CMS, CountSketch and heap frames each had two decoders.summary_kernels/*(23 files),physical.rs, the backend updater factory, the legacyquery_statistic/aux_stats/SingleSubpopulationAggregate/MultipleSubpopulationAggregatesurface, andasap_types::traits. Tests read through Planner'sestimate/readout.SummaryStatenow wraps Planner kernels and merges with Planner'smerge_with. Exact readouts delegate to Planner'sreadout::exact_readout.Before this PR
A DDSketch frame decoded into the backend's
DDSketchAccumulator. On the way into a Planner DAG it went throughphysical::to_physical, which checkedsample_pand copiedinnerinto Planner'sDDSketchAccumulator. Outputs came back throughfrom_physical.After this PR
The same frame decodes straight into Planner's
DDSketchAccumulator, and the store keeps that value. DAG inputs areArc::clones of it.check_storablerejects states that have no codec, the case wherefrom_physicalused to fail.Lines: 68 files, +2,651 / −13,932 (
asap_summary_statealone: +1,911 / −12,809). New golden, decoder and rollup tests are included in the added lines.Compatibility
These current formats round-trip unchanged. Each family is tested against golden bytes written by the pre-change code, checking
encode(decode(x)) == xand the readouts:PlannerExactAccumulatorV1WeightedFrequencyV1UnivMonAccumulatorNativePhysicalOutputV1batchesEdge wire formats are unchanged.
Dropped, as the owner approved for this development stage. Each now fails with a named, versioned rejection instead of being misread:
SumAccumulator,IncreaseAccumulator,MinAccumulator,MaxAccumulator,KeyedSumCountAccumulator,KeyedCounterState(andKeyedMinState/KeyedMaxState).decodereturns "stored format X is retired". Immutable maintenance inputs fail with that reason. The best-effort durable query union logs it and skips the entry, the same way it already skips unreadable parts.SumAccumulatorV1.Behavior changes:
0 < sample_p < 1fail to decode. Planner kernels have no place to carrysample_p. Before, the stored-bytes readout silently ignored it, and only the legacy in-memory readout rescaled counts.SketchEnvelopeattribute payloads decode into the matching Planner kernel. Before, they were an opaque accumulator whose merge kept the left side.Min/MaxAccumulator, which exact states stopped producing. A pane that does not extend the series contiguously (a late correction, a replacement or a gap) now permanently invalidates that series' rollup, so readers fall back to the exact panes instead of serving an uncorrected extremum. Regression testsplanner_exact_min_panes_feed_the_min_rollupandmin_rollup_stops_answering_after_a_late_correctionfail on the previous logic.sampled_ddsketch_envelope_on_delta_channel_is_rejected.DdSketch::from_proto), so negative and zero buckets are kept. Before, ingest usedfrom_raw, which kept positive buckets only.Remaining
Two shims are kept, marked in the code:
asap_summary_state::univmon, because Planner's UnivMon has no public sketch or byte form.frequency_kernel/decode_exact.A small Planner PR would let both go by adding:
UnivMonAccumulator::from_sketch(UnivMon) -> Result<Self>andUnivMonAccumulator::sketch(&self) -> &UnivMon(or a byte codec). This deletes the UnivMon shim.WeightedFrequency::algorithm(),shape()and kernel byte access (to_bytes/from_bytes). This removes the msgpack round trip infrequency_kernel/frequency_state.ExactAccumulatordecode (or a public constructor from populations). This removes the serde mirror indecode_exact.AggregateCore::as_any_mut(orinto_any). This lets ingest delta frames mutate a cached base in place instead of clone-and-replace, and letsSummaryState::merge_same_familyavoid a second copy after Planner'smerge_with.sample_pon the DDSketch/HLL/CMS kernels (or a scaled readout), to re-admit sampled edge frames.Validation
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --locked -- -D warningscargo test --workspace --locked --libcargo test -p control_plane --locked --testscargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1🤖 Generated with Claude Code