docs: propose workload-wide planning, summary sharing, and materialization - #509
Merged
Merged
Conversation
Add the layering proposal, its proposals-index entry, and an Output layers section in the input/output/workflow design. Every Planner layer outputs all legal candidates; the deployment keeps every summary-family candidate and selects with its own costs. Design DAG names are marked as the target API with the current main type alongside. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Split physical planning into physical design (summary materialization, workload-level, like materialized-view selection) and physical implementation (per-node lowering and cutting). The annotated graph stays a Post-ASAP DAG; materialization is decided in design and realized by the cut. State node-by-node correspondence and the Fallback exception that the operator-flattening proposal removes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Describe current Planner behaviour (no family pruning) and the backend gap, and replace the Binary 'exception' with the general rule that a timing- sensitive node needs one compilation per distinct timing, with an example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng proposal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
self-requested a review
September 30, 2026 18:36
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
requested changes
Sep 30, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
requested changes
Sep 30, 2026
Planner takes the query and data workloads plus the deployment's cost model, accuracy requirements and capabilities, and returns one optimal PhysicalDAG; candidate sets stay internal. Name the annotated stage MaterializedPostASAPDAG, tabulate what each DAG encodes, fold timing and selection into the layer descriptions, rename physical design to summary materialization, and make the example trace one query through every DAG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
requested changes
Sep 30, 2026
…g row A lifecycle fixes materialization, timing, maintenance, retention and window framework together, so the stage and its DAG are named after the lifecycle (LifecyclePostASAPDAG) rather than one of those aspects. The deployment row no longer mentions ranking or selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Sep 30, 2026
Describe the two plain candidate collections, the view-based compile, the caller-typed physical candidate errors, lifecycle_guarantee, and PhysicalExecution as an execution handle. Remove APIs the docs said were removed but never existed (compile_timed_candidates, PhysicalDAGCandidate), the agent instructions in the alignment proposal's baseline, and the ingestion-time Binary exception from design docs, where it is an implementation detail (the developer migration guide keeps it). Restore #485's statement that candidates do not choose placement and #508's CandidatePostASAPDAGs<Id> names in input-output-workflow.md, rejoin the split test table in physical-planning-and-deployment.md, and take planner-backend-layering.md verbatim from #509. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Sep 30, 2026
zzylol
added a commit
that referenced
this pull request
Sep 30, 2026
The layering proposal (#509) names a single annotated DAG, LifecyclePostASAPDAG: a PostASAPDAG plus its lifecycle assignment. The code kept a SummaryMaintenanceLifecyclePlan beside the DAG and named the timed collection CandidatePostASAPDAGsWithTiming. - Rename SummaryMaintenanceLifecyclePlan to LifecyclePostASAPDAG and its error to LifecyclePostASAPDAGError. The type already held the root and each state's lifecycle, retention and window framework; per-node timing stays derived by execution_assignment as the existing PostASAPDAGAssignment overlay, so no timed graph is stored beside it and no new type is added. - Rename CandidatePostASAPDAGsWithTiming to CandidateLifecyclePostASAPDAGs; CandidatePostASAPDAGs stays the logical collection. - Call layer 2 "summary lifecycle planning" and reword comments that had the deployment choose or rank; selection is Planner's, over the deployment's cost model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Sep 30, 2026
Several docs still said the deployment or "downstream" selects, ranks or chooses the plan, and the glossary said physical plans are owned by downstream systems. Per the layering proposal (#509), Planner compiles and selects the physical plan using the deployment's cost model; the deployment supplies prices and executes the selected plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Selvomega
previously approved these changes
Sep 30, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Sep 30, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every stage after the frontend is a Post-ASAP DAG; the prefix says how far planning has gone: Logical, Lifecycle, Physical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Oct 1, 2026
#482, #479, #485, #491) Stage 2 materialization (#509): enumerate summary-maintenance lifecycle candidates, derive execution timing from the chosen lifecycle, compile once and cut per lifecycle, keep placement in the lifecycle layer, and plan maintained populations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 1, 2026
milindsrivastava1997
self-requested a review
October 2, 2026 00:25
milindsrivastava1997
previously approved these changes
Oct 2, 2026
zzylol
added a commit
that referenced
this pull request
Oct 2, 2026
- PlanSpace -> CandidateLogicalASAPDAGs (stage 1 output) - PhysicalCandidate -> PhysicalASAPDAG (element of the stage 2 candidate set; one is chosen by selection) - UncheckedCandidate -> UncheckedPhysicalASAPDAG (private serde helper) Identifiers embedding these names, doc comments and docs/ follow. No behavior or serialized-output change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Oct 2, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace "executable", "lower", "price" and "costed" wording, note that materialized output may live on disk or in memory, keep Hydra's per-job heap explicit, mark SQL entropy/L2 recognition as frontend TODO, and fill in the sliding-window merge citation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 2, 2026
Open
zzylol
added a commit
that referenced
this pull request
Oct 2, 2026
Resolve the conflict in docs/design_docs/proposals/operator-sharing.md by keeping this PR's version. Since this branch was cut, main changed two lines in sections that this PR rewrites: - the MaintainPopulation/ReadPopulation timing row (lifecycle-set population timing, from the lifecycle stack), and - PlanSpace -> CandidateLogicalASAPDAGs (#514). Neither line exists in this PR's rewritten §2, which already assigns timing to physical planning under #509. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 2, 2026
Open
Contributor
Author
|
Self-reflection: This PR raises the need for complete design documents explaining how strategies, query workloads, and ASAP primitives fit into the architecture.
The schema, physical data, and summary-state contract is separately requested in #545, following the discussion on #535. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Propose ASAPPlanner's planning stages, from language-specific frontends through logical and physical optimization to selection of one physical plan for the whole workload. The deployment supplies empirical cost and accuracy models and execution capabilities, then executes the selected plan.
Add
docs/design_docs/proposals/planner-layering.mdand link it from the proposals index.Before this PR
The planning boundary needs an explicit contract for workload-wide candidates, summary sharing, materialization, and plan selection.
After this PR
CandidateLogicalDAGs; logical optimization producesCandidateLogicalASAPDAGs; physical optimization producesCandidatePhysicalASAPDAGs; selection returns onePhysicalASAPDAG.Four worked examples cover candidate growth (3 local, 54 logical and 156 physical alternatives), UnivMon sharing across three computations, KLL sharing across windows, and ingestion/query-time materialization choices.
Scope and validation
Documentation-only proposal; no runtime or API changes. Deployment-input types, summary subtract/delete design, and physical parallelism, partitioning and resource management remain TODO.
The branch's file contents have been restored to commit
8c29c2e376800ad0823d4917e121eed3b60ed3aband verified identical withgit diff. No runtime tests run.