Skip to content

feat: share identical summary producers across queries after Pass 1 - #515

Merged
zzylol merged 3 commits into
mainfrom
feat/share-identical-summaries
Oct 2, 2026
Merged

zzylol merged 3 commits into
mainfrom
feat/share-identical-summaries

Conversation

@zzylol

@zzylol zzylol commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Why

#509, logical Pass 2 (ASAP-aware CSE): structurally identical summary producers chosen by different queries should be built and maintained once, and selection should cost them once.

On main, pre-ASAP CSE already shares identical subtrees that have a unique key. Two cases were not shared: different targets whose Pass-1 candidates contain the same SummaryAgg, and identical ungrouped queries. Each paid full maintenance and was deployed separately.

What

This wires the existing share_common_summary_subtrees (crates/types/src/post_asap/cse.rs) into planning. No new types.

It covers the identical-expression rule and the equal-params subset of the summary-capability rule. Example: quantile_over_time(0.5, lat[5m]) and quantile_over_time(0.99, lat[5m]) read one KLL.

  • Costing (summary_maintenance_lifecycle.rs):
    • Summary candidates from different targets whose outermost SummaryAgg interns to the same node form a class.
    • If the shared state still has a selectable lifecycle against the union of the class's entries, it is costed once. The cost is split equally across members through CandidateCostOverrides. Independent candidates stay.
    • If any member selects something else, the class reverts to independent costs and selection runs once more.
  • Assembly (major.rs): MajorPass assembles all roots, runs CSE once, then plans lifecycles. A shared SummaryAgg is therefore Rc::ptr_eq across QueryLifecyclePlans.
  • PlanOutput documents that consumers dedupe deployments by Rc pointer.

Tests

New file crates/planner/tests/summary_sharing.rs, run through e2e_plan / MajorPass:

  • p50 and p99 over the same window and ε share one state, deployed once, with one lifecycle.
  • Negative controls do not share: lat[10m], an ε that changes the KLL params, and {job="a"} vs {job="b"}.
  • Two identical sum(rate(x[5m])) share both their states.
  • Two identical filtered SQL approx_percentile_cont queries share.
  • Sharing alone can beat raw recompute: state 100 vs raw 60 when the query is alone, 50 each when shared. This test fails with shared costing disabled.

Revert path: unit test sharing_class_reverts_when_a_member_selects_elsewhere. It fails with the revert step disabled.

Known limitations

  • SQL p50/p99 do not share yet. The SQL frontend names the state column after the output column, which includes the quantile. Fixed in a follow-up PR.
  • For a shared root, assembly's raw-vs-summary check compares that root's raw cost over the union's reads with the whole shared state. This is approximate unless the members cost about the same per read. It will be resolved when per-root plans become one workload DAG.
  • The revert drops the whole class, not only the member that left.
  • Cross-family sharing (one UnivMon feeding distinct, entropy and L2) and window composition are later Pass 2 PRs.

Validation

Rust 1.99: fmt, clippy -D warnings, the MetricsQL baseline and cargo test --workspace (1499 passed on the pre-rebase base). The rerun on current main is in progress locally and in CI.

🤖 Generated with Claude Code

zzylol and others added 3 commits October 2, 2026 00:02
Two queries can select structurally identical summary producers that
pre-ASAP CSE cannot merge: identical ungrouped expressions (no unique
key) and readouts over one producer with equal parameters (p50 and p99
of one window and accuracy reading one KLL). Each was costed and
deployed as its own state.

Selection now groups finalized summary candidates of different targets
whose outermost SummaryAgg interns to the same node, costs that state
once against the union of the targets' entries, and offers each member
an equal split through CandidateCostOverrides. A class whose shared
state has no selectable lifecycle stays independent; if selection leaves
any member elsewhere, that class reverts to independent costs and
selection runs once more.

MajorPass assembles every root, runs share_common_summary_subtrees once
across them, and plans each root's lifecycle with the entries of every
root that reaches its shared states, so a shared SummaryAgg is one Rc
with one lifecycle across QueryLifecyclePlans. PlanOutput is not
deduplicated; consumers dedupe deployments by Rc pointer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant