Skip to content

feat: bind query-time raw series inputs to the Prometheus endpoint - #792

Open
zzylol wants to merge 2 commits into
refactor/planner-physical-handofffrom
feat/query-time-raw-source
Open

zzylol wants to merge 2 commits into
refactor/planner-physical-handofffrom
feat/query-time-raw-source

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Why

Planner can select candidates with an Ephemeral summary lifecycle, where the summary is built at query time. The physical DAG for such a candidate starts from a raw-series InputContract, and the deployment has to bind that input. The backend has no local raw samples. It could only forward whole PromQL subqueries to Prometheus, and it rejected any raw Scan in a deployed plan ("local raw Scan is forbidden").

What

  • data_plane/.../raw_source.rs: a deployment RawSource that reads one range selector ({__name__="m",job="api"}[300000ms], instant query at at - offset) from the configured Prometheus-compatible endpoint (exact_subquery_endpoint, subject to the query-forwarding policy). It is bound through Planner's DataSources/Scan, which gives lazy open and schema-drift checks.
    • The contract check accepts only raw PromQL series: series identity, a Timestamp time index, a non-null Float64 value and Utf8 label projections, and the input must be Bounded.
    • Rows are ordered by label set, then by time. Samples fall in the left-open (end - range, end] window, and anything outside it is an error.
    • Wire, decode and decoded rows are charged to the run's RunContext. The request goes through request::wait, so caller cancellation reaches it.
    • HTTP errors, error status, warnings, non-matrix results and timeouts become Error::Operator, never an empty scan.
  • Binding hook (native_values::execute_stored, engine.rs): a Physical root whose inputs are raw Scan nodes binds each one as a lazy physical source. It no longer tries to prepare them as exact leaves. The summary store is required only for stored inputs.
  • asap_types validation accepts raw Scan vector inputs and rejects plans that mix them with installed state (no common snapshot).

Before this PR

An installed Physical program over a raw Scan input for quantile_over_time(0.5, m{job="api"}[5m]) fails in preparation with local raw Scan is forbidden in deployed plans.

After this PR

The same installed plan, holding Planner's compiled KLL SummaryAgg → SummaryEstimate(q=0.5) over the raw input, runs end to end. It makes one Prometheus request and returns the exact per-series median (instance=a → 3, instance=b → 30).

Not in scope

  • Candidate selection. The backend still passes supports_ephemeral: false, and turning it on plus emitting these plans in the control plane is a follow-up.
  • ClickHouse raw rows for SQL, and Prometheus remote-read. Neither is implemented here.
  • Output labels keep __name__ from the raw identity. PromQL name-dropping belongs to the readout.

Validation

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test --workspace --locked --lib (data_plane: 898 passed)
  • New tests in raw_source.rs:
    • a mock matrix binds as a bounded, contract-typed, ordered batch
    • HTTP, protocol, timeout and cancellation failures become execution errors
    • the memory budget is enforced
    • the installed KLL + quantile over raw Prometheus input matches the exact quantile
    • endpoint failure or a missing endpoint propagates as a query error

🤖 Generated with Claude Code

zzylol and others added 2 commits September 30, 2026 03:03
Raw range-selector inputs are validated like other vector inputs and may
not be mixed with installed summary state, which shares no snapshot with
an external endpoint.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Planner raw-series input contract of an installed physical program is
bound to a RawSource that reads the range selector from the configured
Prometheus-compatible endpoint inside the run: lazily, under the request
budget and cancellation, with bounded, deterministically ordered rows.
Endpoint failures are execution errors. Candidate selection is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant