Skip to content

chore: repin Planner integration and take over storage formats and panes - #793

Open
zzylol wants to merge 2 commits into
feat/query-time-raw-sourcefrom
refactor/repin-planner-integration
Open

zzylol wants to merge 2 commits into
feat/query-time-raw-sourcefrom
refactor/repin-planner-integration

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Stacked on #792.

Why

The backend must build against the Planner integration revision
c98281a (integration/planner-for-backend: #473→#474→#475 plus #477, #479,
#483, #484, #485). That revision keeps only in-memory physical computation in
Planner. Storage formats, stored-state decoding, the stored-definition identity
and Rate placement variants are no longer Planner's. This PR (step D1) moves
them into the backend without changing backend behaviour. Candidate selection
is unchanged; redesigning it is D2.

What

  • Repin every ASAPPlanner dependency from 76fbbf16 to c98281a. In
    Cargo.lock, only the Planner source lines, Planner's dropped dependencies
    and the new path crates change. No unrelated crate moved.
  • crates/asap_summary_state (new): the stored summary kernels (Sum, Min/Max,
    keyed states, sketches with sample_p, envelope, …), AggregationType,
    their byte encodings, stored_state (decoders, delta reconstruction, native
    batch frames, readouts), and physical: conversion between stored and
    Planner physical states where stored state enters or leaves a physical DAG.
    Code is taken from Planner 76fbbf16, with its tests.
  • crates/asap_sketch_codec: moved into the workspace unchanged.
  • asap_types::physical_plan_codec: the versioned JSON envelope of persisted
    physical plans and candidates (Planner now exposes only serde). The bytes
    match what Planner wrote.
  • asap_types::semantic_fragment: SummarySemanticFragment and
    LogicalDatasetIdentity, moved from Planner.
  • control_plane::physical::compiler::rate_placement: the fixed-window and
    query-time Rate→heap/Sum placement variants that fix(precompute): preserve deadline corrections #485 removed from
    SketchAlgorithmStrategy. They are derived from the same direct candidates,
    and compiled through compile_post_asap_dag and the new
    compile_fixed_window_rate_aggregation(&PostAsapDag).
  • The backend never used the pane API removed in fix(precompute): separate event and closure watermarks #483 (PaneInput,
    compile_temporal_pane_candidate), so nothing was ported. No backend fixture
    embeds docs: define ASAPQuery compatibility profile #479 helper IDs, so none needed regeneration.

Before this PR

backend ──► Planner 76fbbf16
            ├─ physical DAG runtime + kernels
            ├─ stored kernels, byte formats, delta decoding (stored_state)
            ├─ CompiledPhysicalDag::encode/decode, SummarySemanticFragment
            └─ SketchAlgorithmStrategy::{fixed_window,query_time}_rate_*

After this PR

backend ──► Planner c98281a: physical DAG runtime + in-memory kernels
   ├─ asap_summary_state: stored kernels, formats, delta decoding,
   │                      stored ⇄ physical state conversion
   ├─ asap_sketch_codec, asap_types::{physical_plan_codec, semantic_fragment}
   └─ control_plane rate_placement

Example: a fixed-window sum by (job) (rate(requests_total[1m])) precompute
binds stored counter windows. They convert to Planner exact states, run in the
Planner DAG, and publish outputs converted back to stored kernels. The
revision records and native batch bytes are the same as before.

Behaviour differences

  • When a stored SumAccumulator is bound as a physical input, or decoded from
    a native SumAccumulatorV1 cell, it becomes Planner's exact Sum state, the
    only exact state the new Planner accepts. Sum readouts are unchanged. If such
    a cell is re-encoded, it is written as ExactAccumulatorV1, and its
    observation count is not carried over.
  • The rules for binding sketches are unchanged: an edge-sampled DDSketch/HLL
    (sample_p ≠ 1) is still rejected as a physical input.
  • These come from the new Planner revision, not from this PR: docs: define ASAPQuery compatibility profile #479 helper
    operator IDs, readout operators serialized as ReadoutQuery, and the wider
    compile coverage from feat(precompute): separate idle and deadline closure #484. Physical plans persisted by a backend built on
    76fbbf16 may not deserialize under the new operator serde, although their
    envelope is unchanged.
  • Each DAG boundary crossing clones the sketch state once during conversion.

Validation

  • cargo fmt --all -- --check: clean
  • cargo clippy --workspace --all-targets --locked -- -D warnings: clean
  • cargo test --workspace --locked --lib: all pass (asap_summary_state 239,
    asap_types 132, control_plane 443, data_plane 898)
  • cargo test -p control_plane --locked --tests: all pass
  • cargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1:
    23 passed, 1 failed. The failing test is
    issue_workloads_execute_warm_at_successive_evaluations, which exceeds its
    30 s readiness budget because of startup candidate search. This failure is
    already known on this branch. The timeout was not raised.
  • New tests: stored ⇄ physical conversion (sketches, exact/Sum, weighted
    frequency, sampled rejection), plan/candidate envelope round trips, the
    stored Sum native cell, key/measurement codecs, and both Rate placement
    variants.

🤖 Generated with Claude Code

zzylol and others added 2 commits September 30, 2026 04:11
Repin ASAPPlanner to c98281a (integration/planner-for-backend). That
revision keeps only in-memory physical kernels, so the backend now owns
what Planner dropped, unchanged in behaviour and byte format:

- crates/asap_summary_state: the stored summary kernels, AggregationType,
  stored-state decoders, delta reconstruction, native batch frames and
  conversion to Planner physical states at DAG boundaries.
- crates/asap_sketch_codec: the SketchEnvelope codec.
- asap_types::physical_plan_codec: the versioned plan/candidate envelope.
- asap_types::semantic_fragment: SummarySemanticFragment and
  LogicalDatasetIdentity.
- control_plane rate_placement: the fixed-window and query-time Rate
  placement variants Planner no longer lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant