Skip to content

docs: propose workload-wide planning, summary sharing, and materialization - #509

Merged
zzylol merged 56 commits into
mainfrom
docs/planner-layering-contract
Oct 2, 2026
Merged

zzylol merged 56 commits into
mainfrom
docs/planner-layering-contract

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Propose ASAPPlanner's planning stages, from language-specific frontends through logical and physical optimization to selection of one physical plan for the whole workload. The deployment supplies empirical cost and accuracy models and execution capabilities, then executes the selected plan.

Add docs/design_docs/proposals/planner-layering.md and link it from the proposals index.

Before this PR

The planning boundary needs an explicit contract for workload-wide candidates, summary sharing, materialization, and plan selection.

After this PR

  • Frontends produce CandidateLogicalDAGs; logical optimization produces CandidateLogicalASAPDAGs; physical optimization produces CandidatePhysicalASAPDAGs; selection returns one PhysicalASAPDAG.
  • Logical optimization generates local exact and summary alternatives, then applies ASAP-aware common-subexpression elimination across computations and windows.
  • Physical optimization explores materialization, retention and operator implementations. Outputs may be maintained at ingestion time, retained at query time, or rebuilt without materialization.
  • Selection evaluates complete workload candidates using deployment cost and accuracy models, rejects candidates that fail requirements or capabilities, and chooses the cheapest remaining plan.

Four worked examples cover candidate growth (3 local, 54 logical and 156 physical alternatives), UnivMon sharing across three computations, KLL sharing across windows, and ingestion/query-time materialization choices.

Scope and validation

Documentation-only proposal; no runtime or API changes. Deployment-input types, summary subtract/delete design, and physical parallelism, partitioning and resource management remain TODO.

The branch's file contents have been restored to commit 8c29c2e376800ad0823d4917e121eed3b60ed3ab and verified identical with git diff. No runtime tests run.

zzylol and others added 3 commits September 30, 2026 17:59
Add the layering proposal, its proposals-index entry, and an Output layers
section in the input/output/workflow design. Every Planner layer outputs all
legal candidates; the deployment keeps every summary-family candidate and
selects with its own costs. Design DAG names are marked as the target API with
the current main type alongside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split physical planning into physical design (summary materialization,
workload-level, like materialized-view selection) and physical
implementation (per-node lowering and cutting). The annotated graph stays a
Post-ASAP DAG; materialization is decided in design and realized by the cut.
State node-by-node correspondence and the Fallback exception that the
operator-flattening proposal removes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 5 commits September 30, 2026 18:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe current Planner behaviour (no family pruning) and the backend gap,
and replace the Binary 'exception' with the general rule that a timing-
sensitive node needs one compilation per distinct timing, with an example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng proposal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Selvomega
Selvomega self-requested a review September 30, 2026 18:36
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/design_docs/architecture/input-output-workflow.md Outdated
Comment thread docs/design_docs/architecture/input-output-workflow.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Planner takes the query and data workloads plus the deployment's cost
model, accuracy requirements and capabilities, and returns one optimal
PhysicalDAG; candidate sets stay internal. Name the annotated stage
MaterializedPostASAPDAG, tabulate what each DAG encodes, fold timing and
selection into the layer descriptions, rename physical design to summary
materialization, and make the example trace one query through every DAG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated
…g row

A lifecycle fixes materialization, timing, maintenance, retention and
window framework together, so the stage and its DAG are named after the
lifecycle (LifecyclePostASAPDAG) rather than one of those aspects. The
deployment row no longer mentions ranking or selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
Describe the two plain candidate collections, the view-based compile, the
caller-typed physical candidate errors, lifecycle_guarantee, and
PhysicalExecution as an execution handle. Remove APIs the docs said were
removed but never existed (compile_timed_candidates, PhysicalDAGCandidate),
the agent instructions in the alignment proposal's baseline, and the
ingestion-time Binary exception from design docs, where it is an
implementation detail (the developer migration guide keeps it).

Restore #485's statement that candidates do not choose placement and #508's
CandidatePostASAPDAGs<Id> names in input-output-workflow.md, rejoin the
split test table in physical-planning-and-deployment.md, and take
planner-backend-layering.md verbatim from #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
Copies docs/design_docs/proposals/planner-backend-layering.md from #509's
head 3607270 so this file merges cleanly after #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
The layering proposal (#509) names a single annotated DAG,
LifecyclePostASAPDAG: a PostASAPDAG plus its lifecycle assignment. The
code kept a SummaryMaintenanceLifecyclePlan beside the DAG and named the
timed collection CandidatePostASAPDAGsWithTiming.

- Rename SummaryMaintenanceLifecyclePlan to LifecyclePostASAPDAG and its
  error to LifecyclePostASAPDAGError. The type already held the root and
  each state's lifecycle, retention and window framework; per-node timing
  stays derived by execution_assignment as the existing
  PostASAPDAGAssignment overlay, so no timed graph is stored beside it and
  no new type is added.
- Rename CandidatePostASAPDAGsWithTiming to CandidateLifecyclePostASAPDAGs;
  CandidatePostASAPDAGs stays the logical collection.
- Call layer 2 "summary lifecycle planning" and reword comments that had
  the deployment choose or rank; selection is Planner's, over the
  deployment's cost model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
Several docs still said the deployment or "downstream" selects, ranks or
chooses the plan, and the glossary said physical plans are owned by
downstream systems. Per the layering proposal (#509), Planner compiles
and selects the physical plan using the deployment's cost model; the
deployment supplies prices and executes the selected plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
#509 carries the proposal; #480's copy now matches its head a095ce5
exactly, so merging either PR first leaves no conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol requested a review from Selvomega September 30, 2026 20:37
Selvomega
Selvomega previously approved these changes Sep 30, 2026
Comment thread docs/design_docs/proposals/planner-layering.md Outdated
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every stage after the frontend is a Post-ASAP DAG; the prefix says how
far planning has gone: Logical, Lifecycle, Physical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 1, 2026
#477)

Logical Pass 1 (#509): expose TopK heap alternatives over current series as candidates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 1, 2026
#482, #479, #485, #491)

Stage 2 materialization (#509): enumerate summary-maintenance lifecycle candidates, derive execution timing from the chosen lifecycle, compile once and cut per lifecycle, keep placement in the lifecycle layer, and plan maintained populations.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 2, 2026
- PlanSpace -> CandidateLogicalASAPDAGs (stage 1 output)
- PhysicalCandidate -> PhysicalASAPDAG (element of the stage 2 candidate
  set; one is chosen by selection)
- UncheckedCandidate -> UncheckedPhysicalASAPDAG (private serde helper)

Identifiers embedding these names, doc comments and docs/ follow.
No behavior or serialized-output change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 2, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace "executable", "lower", "price" and "costed" wording, note that
materialized output may live on disk or in memory, keep Hydra's per-job
heap explicit, mark SQL entropy/L2 recognition as frontend TODO, and fill
in the sliding-window merge citation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol merged commit 20c22d1 into main Oct 2, 2026
1 of 2 checks passed
zzylol added a commit that referenced this pull request Oct 2, 2026
Resolve the conflict in docs/design_docs/proposals/operator-sharing.md by
keeping this PR's version. Since this branch was cut, main changed two lines
in sections that this PR rewrites:
- the MaintainPopulation/ReadPopulation timing row (lifecycle-set population
  timing, from the lifecycle stack), and
- PlanSpace -> CandidateLogicalASAPDAGs (#514).
Neither line exists in this PR's rewritten §2, which already assigns timing
to physical planning under #509.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol

zzylol commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Self-reflection: This PR raises the need for complete design documents explaining how strategies, query workloads, and ASAP primitives fit into the architecture.

  1. Calling for design documentation explaining each replacement strategy in ASAPPlanner, how it works within this architecture, and an end-to-end example for each strategy, in Document every ASAPPlanner replacement strategy with architecture mapping and an end-to-end example #546.
  2. Calling for design documentation explaining how to add a new query workload, with end-to-end examples showing how it can use existing strategies and when an extension is needed, in Document adding new query workloads and reusing existing strategies with end-to-end examples #547.
  3. Calling for design documentation explaining how to add a new ASAP primitive, sketch algorithm, or other summary algorithm, in Design the extension contract for new ASAP primitives and sketch algorithms #548. The assumption is that the algorithm and its strategy are supplied by an algorithm developer, not invented by ASAPPlanner. The design should separate: (a) supported aggregation intents, including statistics and window/time behavior; (b) summary operations it can use in the logical ASAP DAG, or genuinely new operations it requires; (c) materialization capabilities and execution-timing constraints, including whether an ingestion-time-only restriction is real or query-time computation is merely expensive; and (d) physical operator implementations. Include a complete end-to-end extension example.

The schema, physical data, and summary-state contract is separately requested in #545, following the discussion on #535.

FYI @milindsrivastava1997 @Selvomega @vyassekar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants