Skip to content

executor: support tumbling pane merges (per-entity TimeShift panes, temporal merge timestamp) - #591

Draft
zzylol wants to merge 3 commits into
stack/509-w3-relative-coveragefrom
stack/509-w2-executor-panes
Draft

zzylol wants to merge 3 commits into
stack/509-w3-relative-coveragefrom
stack/509-w2-executor-panes

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Wave 1 chain: #594 → #593 → #592 → #591 → #595 → #596 → #597 → #598 → #599 (on #589)

Why

#580 will plan a tumbling window as N pane SummaryAggs merged by SummaryMerge (#509 Example 3/4, Pattern B). Pane i reads TimeRange(w) over TimeShift(i·w) over Scan. For per-series (PromQL) queries, the executor could not run that shape. This PR ports the executor parts of the parked #566 and #569 onto the renamed crates/executor.

What

  1. Per-entity build over TimeShift (from Generate exact window composition and materialization candidates #566). The per-entity SummaryAgg path required Scan directly under TimeRange. It now looks through one TimeShift. The deployment supplies each pane's shifted raw rows.
  2. Temporal pane merge keeps the evaluation timestamp (from Preserve evaluation timestamps in native temporal pane merges #569). A SummaryMerge whose output has a time_index now lowers to [union] → summary_merge (group by series, no ts) → scope_timestamp, the same pattern as per-series SummaryAgg. Pane build timestamps are grouped away, and the run scope supplies the time column. Non-temporal merges are unchanged.

How it is tested

crates/executor/tests/tumbling_pane_merge.rs takes the selected single-build plan for sum_over_time(m[5m]) (exact) and quantile_over_time(0.99, m[5m]) (KLL, k=52). It rewrites the plan into 5 one-minute panes under a SummaryMerge and runs both versions over 2 series × 20 samples, at evaluation time 300 000 ms. Pane coverage uses absolute regions [T−(i+1)·60 000, T−i·60 000). The pane plan runs two ways:

  • B2: panes rebuilt from raw rows at query time.
  • B1: panes built in Scope::Ingestion for their own window, so they have distinct build timestamps (60 000 … 300 000), then retained and merged at query time.

Before this PR:

exact_sum_over_tumbling_panes_matches_single_build ... FAILED
  Invalid("per-entity summary requires a resolved source")
# with only fix 1 applied:
  Invalid("node 20: invalid DAG: native output type differs from Planner output")

After this PR: both tests pass. Each path returns one row per series, ts = 300 000, and the output is identical to the single 5-minute build: Sum [190, 2190], p99 [19, 119]. 20 samples < k, so the KLL result is exact.

Notes for the planner side (#580)

  • The tests set timing = QueryTime on each node by hand. apply_materialization_timings (types) still returns UnimplementedOperator { "SummaryMerge" }, so a planner-emitted pane plan cannot be exported until that rule is added.
  • Coverage regions are absolute here. The relative-time coverage (W2) is being added to types separately.

Gate: cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -D warnings, and cargo test --workspace pass: 1,519 passed / 10 ignored, compared with 1,517 / 10 on #589. The +2 are the new tests.

Origin: #566, #569. Part of #580 (item O), #509.

🤖 Generated with Claude Code

zzylol and others added 3 commits October 5, 2026 04:48
A tumbling pane (#580) reads TimeRange(w) over TimeShift(i*w) over Scan.
The per-entity build path required the Scan directly under the TimeRange
and rejected panes with "per-entity summary requires a resolved source".
The deployment supplies the shifted raw rows, so look through one
TimeShift to find the source schema. Ported from #566.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A per-series SummaryMerge grouped away the time column, so its native
output disagreed with the Planner schema ("native output type differs
from Planner output"). Merge by series identity, then attach the run
scope's timestamp with scope_timestamp, as per-series SummaryAgg does.
Pane build timestamps never reach the merged state. Ported from #569.

Regression tests merge 5 one-minute panes for an exact Sum and a KLL
quantile, rebuilt at query time and retained with distinct build
timestamps, and compare them with one 5-minute build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pane coverage time is a CoverageTime since #592; the tumbling pane-merge test
keeps absolute time ranges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-w2-executor-panes branch from 630efac to 7fa00bd Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 0f9db83 to 6b31d2d Compare October 5, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant