Skip to content

feat: execute mixed raw and stored inputs with bounded staleness - #803

Open
zzylol wants to merge 3 commits into
fix/topk-by-heap-groupsfrom
feat/bounded-lag-mixed-inputs
Open

zzylol wants to merge 3 commits into
fix/topk-by-heap-groupsfrom
feat/bounded-lag-mixed-inputs

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Stacked on #802.

Why

#792 added query-time raw-series inputs but rejected any physical plan that also read stored state ("no common snapshot"). The owner decided that such mixes are allowed under a bounded-staleness rule, which lets one plan combine fresh raw series with maintained state.

What

  • asap_types: the reject rule is replaced by an admission rule. Raw inputs may share a program with stored native batches, the only stored inputs whose lag can be checked. New MaterializationBinding::slide_ms / latest_window_end_at_or_before give the stored window grid.
  • data_plane: execute_stored binds raw inputs at t_q. For a mixed plan, each stored input takes the newest complete window at or before t_q (t_s). If a window is incomplete it steps back one slide, but only while t_q - t_s <= max_lag. Beyond the bound it returns a capability miss, which is the existing exact fallback, before any raw read.
  • Config: --max-stored-input-lag-ms (ASAPQueryEngine::with_max_stored_input_lag_ms). Unset means one slide of each stored output. I used deployment config rather than a plan field because it needs no plan-format or control-plane change. It sits next to --remote-write-revision-freshness-ms. A per-query field can follow when the Planner emits mixed plans.
  • Reporting: debug trace records t_q, t_s, lag and bound. Execution stats carry the lag; for range queries, the largest lag across steps. The HTTP response gets x-asap-stored-input-lag-ms, and the internal warning is stripped.

Before this PR

sum_over_time(m[5m]) + <stored per-series Sum> with the raw leaf bound to Prometheus fails validation: query-time raw inputs cannot be mixed with installed state.

After this PR

At t_q = 1000s, with a 300s window sliding every 60s:

  • The stored window (660s, 960s] is admitted (lag 40s ≤ 60s). The result is the raw sum_over_time over (700s, 1000s] plus that stored Sum. The response carries x-asap-stored-input-lag-ms: 40000.
  • If only (600s, 900s] is complete (lag 100s > 60s), the query falls back to exact and sends no raw request.

Semantics

  • Raw inputs are read at the query evaluation time t_q.
  • Each stored input uses its latest complete window, with watermark t_s.
  • The mix is admitted iff t_q - t_s <= max_lag for every stored input; otherwise the query fails closed to exact. A mix is never silently stale.
  • Admitted stored families (Sum, heap sketches) read out independently of evaluation time, so a lagged batch is read as of its own window.
  • Stored-only and raw-only plans are unchanged: stored-only still reads exactly the window ending at t_q.

Not in scope

  • Control-plane per-state mixed placement (follows; placement.rs and candidate selection are untouched).
  • Revision-pinned stores (--remote-write-revision-dir) pin [t_q - lookback, t_q]. There, a mixed plan is served only when t_q is on the window grid; otherwise it falls back to exact, which is safe but less available.

Validation

  • cargo fmt --all --check
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test --workspace --locked --lib
  • cargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1 (28 passed)
  • New hand-built-plan tests in raw_source.rs mixed_inputs: within bound equals the exact reference; configured bound admits or rejects an older window; beyond bound falls back with no raw read; a terminal store error is not stepped past; stored-only reads the t_q window; raw inputs are rejected beside exact readouts. The three mixed-execution tests fail with the old reject rule.
  • Plus grid, lag-warning, header and raw-only no-lag checks.
  • An independent reviewer agent reviewed the diff. Fixes applied: one shared raw-input predicate, underlying unavailability reasons kept in miss details, doc for the global override, grid tests at more origins.

🤖 Generated with Claude Code

zzylol and others added 3 commits September 30, 2026 12:36
Replace the no-common-snapshot reject with a bounded-lag admission:
query-time raw inputs may share a physical program with stored native
batches, whose lag the executor checks. Add the stored window grid
helpers used to find the newest complete window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Raw inputs are read at t_q; each stored input uses its newest complete
window t_s and is admitted only when t_q - t_s <= max_lag (default one
slide of that output, override --max-stored-input-lag-ms). Beyond the
bound the query misses to the exact fallback. The observed lag is
traced and reported in execution stats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant