Skip to content

feat: price lifecycle placement from installed windows and all consumers - #804

Open
zzylol wants to merge 2 commits into
feat/bounded-lag-mixed-inputsfrom
feat/placement-pricing
Open

zzylol wants to merge 2 commits into
feat/bounded-lag-mixed-inputsfrom
feat/placement-pricing

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #803.

Why

#794 prices each summary state's lifecycle, but three inputs were proxies. Pane counts came from the evaluation interval, not from the window layout that gets installed. A shared state used its first consumer's unit costs and was priced as one install. Ephemeral was priced as one state rebuild per read, which ignored the raw samples the query-time program actually reads.

What

  • Installed windows (a): retention charges retained_state_count of the window implementation that compilation would install for the state. It reuses the install loop's per-state window narrowing (state_query) and the cost model's window choice (cheapest_window_implementation), both extracted in a no-behaviour-change refactor commit.
  • All consumers (b): consumers share one priced install only when compilation groups them (same layout, cadence, and phase modulo cadence and window). Each install is priced with its own consumers' demand and unit costs.
  • Ephemeral (c): each consumer's program costs, per evaluation, Planner's per-read build + read + retirement plus a fold of the raw samples its Scans cover (ingestion rate × Σ scan range × maintenance_per_update).
  • Group decision: states linked through a query move to query time together, and only when the group's summed program cost is below its summed retained cost. This replaces the per-state decision and demotion loop.
  • Trace: each lifecycle decision now reports retained_states.

Before this PR

Fixture quantile_over_time(0.99, m[1m]), evaluated every 10 s over a 300 s horizon (installs 10 s panes, 7 retained):

  • retention was priced for 6 panes, and Ephemeral for 30 × (10 + 0.1 + 1) = 333;
  • the store price that flips placement was 2.447e-6 per byte-second.

After this PR

  • Retention is priced for 7 panes, equal to the installed num_aggregates_to_retain.
  • Ephemeral costs 30 × (10 + 100/s × 60 s × 0.001 + 0.1 + 1) = 513.
  • The flip price is (513 − 44.3) / (300 × 7 × 65536) = 3.406e-6.
  • Two consumers on different cadences, or out of phase, now cost the sum of each priced alone. Before, they cost less than that sum.

Behaviour differences

  • No existing fixture changes its selection. That includes the default config, the o11y corpus, the compat demo, and the e2e process tests.
  • The change only raises the Ephemeral price and adds retained panes where the old count was low. The break-even store price rises accordingly.
  • A first version that priced Ephemeral as the fold alone flipped instant sums and window ≈ cadence queries to query time. That price charged nothing for fetching from Prometheus and nothing for per-state setup, so it was rejected.
  • Known limits, documented in code:
    • The pane count is for the state's own window. A derived state reading a longer window retains more, so the count is a lower bound there.
    • Native-candidate timing (time_native_candidate) still estimates panes, because its windows are prepared after its timing is chosen.
    • Each Scan is charged the workload's ingestion rate, the same rate maintenance is charged.

Validation

  • cargo fmt --all -- --check, cargo clippy --workspace --all-targets --locked -- -D warnings: clean.
  • cargo test --workspace --locked --lib and cargo test -p control_plane --locked --tests: pass.
  • cargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1: 28/28 pass.
  • New tests in control_plane/tests/lifecycle_placement.rs:
    • retention_prices_the_installed_window_panes, ephemeral_cost_is_the_raw_program_over_its_scanned_samples and shared_state_combines_every_consumers_demand_and_costs failed on the feat: place summary state by backend-priced lifecycle choice #794 code and pass now.
    • store_price_flips_placement_at_the_break_even_price pins the new threshold.
    • out_of_phase_consumers_are_priced_as_separate_installs was added after review.
  • An independent reviewer agent reviewed the change. Its findings were fixed: the group decision, grouping consumers by cadence and phase, the Ephemeral price, and the documented lower bound.

🤖 Generated with Claude Code

zzylol and others added 2 commits September 30, 2026 12:50
Placement needs the window implementation compilation would install for a
retained state. Extract the per-state query narrowing from the install loop
and the cheapest-window rule from the complete-candidate estimate so both
paths use one definition. No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Placement priced retention from window / evaluation interval, priced a
shared state with its first consumer's unit costs as one install, and
priced Ephemeral as a state rebuild per read that ignored the raw samples
the query-time program actually reads.

- Retention charges the panes the window implementation compilation would
  install keeps (retained_state_count of the chosen layout).
- Consumers share one priced install only when compilation groups them
  (same layout, cadence and phase); each install uses its own consumers'
  demand and unit costs.
- Ephemeral adds, per evaluation, the fold of every raw sample the
  program's Scans cover at the per-update cost.
- States linked through a query move to query time together, decided on
  the group's summed costs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol changed the base branch from fix/topk-by-heap-groups to feat/bounded-lag-mixed-inputs September 30, 2026 13:04
@zzylol
zzylol force-pushed the feat/placement-pricing branch from 603b10f to ea98dc0 Compare September 30, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant