Conversation
This was referenced Oct 2, 2026
zzylol
force-pushed
the
stack/528-02-ir
branch
from
October 2, 2026 18:32
a59aff8 to
8f384f7
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 2, 2026 18:32
9369217 to
9870df6
Compare
zzylol
force-pushed
the
stack/528-02-ir
branch
from
October 2, 2026 19:40
8f384f7 to
d8f0b1a
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 2, 2026 19:40
9870df6 to
88c2931
Compare
zzylol
force-pushed
the
stack/528-02-ir
branch
from
October 2, 2026 21:14
d8f0b1a to
dd71af0
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
2 times, most recently
from
October 2, 2026 21:22
c0638f6 to
d414efd
Compare
zzylol
force-pushed
the
stack/528-02-ir
branch
from
October 2, 2026 21:22
dd71af0 to
7d5c382
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 2, 2026 21:25
d414efd to
ec0417b
Compare
zzylol
force-pushed
the
stack/528-02-ir
branch
from
October 2, 2026 21:25
7d5c382 to
6aba8f2
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
3 times, most recently
from
October 3, 2026 02:39
00a3d27 to
467baad
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 3, 2026 14:57
467baad to
aecd114
Compare
This was referenced Oct 3, 2026
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 16:07
ba26c8b to
7cd4ee0
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
3 times, most recently
from
October 3, 2026 16:33
76b23bc to
c30977a
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 17:02
2defbaf to
e83edd6
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 3, 2026 17:02
c30977a to
d02dd8e
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 17:11
e83edd6 to
8497e53
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
2 times, most recently
from
October 3, 2026 17:23
725db8e to
7be9e1a
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 17:23
8497e53 to
3ab6be7
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 3, 2026 17:29
7be9e1a to
2d5b627
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 17:29
3ab6be7 to
77628c8
Compare
This was referenced Oct 3, 2026
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 3, 2026 17:40
2d5b627 to
2cfb683
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 17:40
77628c8 to
02a6a1b
Compare
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 3, 2026 19:32
2cfb683 to
ec425f9
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 19:32
02a6a1b to
60a9e4f
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
force-pushed
the
stack/528-03-graph
branch
from
October 3, 2026 19:45
ec425f9 to
4280239
Compare
zzylol
force-pushed
the
stack/528-02b-merge-structure
branch
from
October 3, 2026 19:46
60a9e4f to
cb00197
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
marked this pull request as draft
October 3, 2026 20:02
A workload DAG has one root per batch query; queries that share a sub-DAG reference the same exported nodes. Single-query export is a batch of one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem: the unified operator IR has no canonicalization, no sub-DAG sharing, and no logical export without execution timing
#509 §Stages says logical planning outputs
CandidateLogicalASAPDAGs: what to compute, with no materialization or execution placement ("Materialization and execution placement are decided in later stages", §1. Logical ASAP-aware optimization). #509 §Pass 2: ASAP-aware common-subexpression elimination starts from traditional CSE:Its first rule is the identical-expression rule: "The input and computation semantics are identical" → "One common computation node serving multiple consumers."
#511 adds two constraints on the same graph:
operator-sharing.md): "Logical nodes may retainNone" forOperatorNode.timing; timing follows the physical materialization choice.decoupling_op_and_expr.md): "Operator dependencies inside scalar conversions/subqueries remain visible and shared; scalar DAGs remain owned."Before this PR. Canonicalization, CSE and DAG export exist only for the old split IRs:
pre_asap::canonicalize::canonicalize(QueryExpr) -> QueryExprQueryExprpre_asap::cse::share_common_sub_dags(Vec<(Id, QueryExpr)>)QueryExprpost_asap::cseSummaryNodedag_export::export/export_post_asapQueryExpr(+ post-ASAP substitutions)The unified
ir::OperatorNodeDAG from #511 (with #567 coverage and #560SummaryMerge) has none of these. So:Two equivalent spellings stay different shapes. Example:
Filter { EXISTS (s) }andJoin { Semi, true, left, s }mean the same thing but cannot be matched structurally.Two identical sub-DAGs stay two
Rcs. Example: one query with the same grouped p50 on both sides of a comparison:Nothing collapses A and B into one node with two consumers.
There is no flat, serializable logical DAG for the unified IR. The earlier feat(ir): add logical sub-DAG sharing and export without execution timing #537 proposal tied export to mandatory timing, so an untimed logical plan (
timing = None, which docs: define unified operators and SQL/PromQL scalar boundaries #511 §2.3 allows) could not be exported.Scope of this PR. It implements the traditional-CSE base of #509 Pass 2 (the identical-expression rule, as exact structural equality) and a transport for the stage-1 output that carries no execution timing. It also adds the canonicalization pass CSE needs as input. It does not implement the summary-capability rule or the window-composition rule of #509 Pass 2, Pass 1 candidate generation, or any physical-stage decision (materialization, timing assignment, phase splitting).
Proposed method
Three steps, all in the logical stage on bound
OperatorNodeDAGs. Each works on the unified IR incrates/types/src/ir/.1. Canonicalize (
ir::canonicalize). Bottom-up over operator inputs, memoized byRcpointer. At each node it applies local rules until none matches:Limit { n: k, offset: 0, Sort { one descending column key, [passthrough Project] Aggregate(Reduce(by), [Count | Sum]) } }becomesAggregate(Reduce(partition), [TopK { k }])over the unchanged inner aggregate. It is skipped for ascending sorts, offsets, ranking by a group key, a Limit partition that differs from the Sort partition, andSumover a derived childAggregate(rate/increase).Filter, oneEXISTS (s)conjunct becomesJoin { Semi, true },NOT EXISTS (s)becomesJoin { Anti, true }, and a positivex IN (s)becomesJoin { Semi, x = Column(left_width) }. Remaining conjuncts stay in an outerFilter. One conjunct per round.NOT IN, scalar subqueries (a cross join would lose their zero-row NULL and multi-row error semantics), andROW_NUMBER()filters (removing the window column would change the visible schema).Untouched sub-DAGs keep their pointer identity, a rewritten shared sub-DAG stays shared, and the pass is idempotent. A
Concatwhose first branch changes schema drops itsdiscriminator_unique_key.2. Share common sub-DAGs (
ir::cse). Hash-consing over all roots of a workload batch, bottom-up:PromqlScalarFromVector,ScalarSubquery,Exists,InSubquery). Soscalar(v)in two queries can sharev.structural_hashonly picks a bucket. The decision is typed equality (same_node): children by pointer (or memoized value comparison), thenresult_kind,schema,timing,coverage,guarantee, and the operator's own fields. Guarantee and own fields must also serialize identically, so0.0vs-0.0andNaNvsNaNare not shared.schema.has_unique_key(). ASAP nodes have no such gate.coverageis part of both hash and equality, so two summary states over different observations are never shared.PartialEq-equal to its input); only itsRcstructure may now alias.The pass does not decide whether sharing is cheaper. That is selection's job (#509 stage 3).
3. Export a logical DAG with no execution timing assigned (
ir::export).compile_logical_asap_queryfirst runsQueryRoot::validate_structure(). Then it visits the DAG in post-order and gives each distinctRconeLogicalASAPNodeId(0, 1, 2, … in visit order). For each node it emits:WireScalarExpr, with operator references replaced by IDs;result_kind,output_schema,guarantee,coverage;Input/Left/Right) and oneScalarRefedge per operator node read by its scalar expressions (and by aSummaryAgg.filter).timingis not exported. A standalone scalar root (QueryRoot::Scalar) stays a scalar root; it gets no synthetic operator node.LogicalASAPDAGDocument::validate()checks a received DAG: version, unique IDs, edge endpoints, edge schema = producer schema, summary family/grouping in the schema, coverage rules, no cycles, every node reachable from the root.Key code interfaces
crates/types/src/ir/canonicalize.rscrates/types/src/ir/cse.rscrates/types/src/ir/export.rscrates/types/src/ir/wire.rs(private module; these are re-exported fromir::export)Usage:
Also added, listed by name only:
NonASAPOpKind(the child-free,WireScalarExpr-based mirror of everyNonASAPOpvariant;kindfields renamedjoin_kind,set_kind,range_kind,sample_kind),WirePredicate,WireProjectItem,WireSortKey,NonASAPOpKind::from_op. Doc:docs/develop_docs/logical-asap-dag.md.Fields
Canonicalize / CSE
canonicalize(root)Rc<OperatorNode>→Result<Rc<OperatorNode>, SchemaDerivationError>root. Returns the sameRcif nothing changes. Errors only when a rebuilt node fails schema derivation.share_common_sub_dags(roots)Vec<(Id, Rc<OperatorNode>)>Idis any caller key (query name, index); it is returned unchanged, in the same order. A batch of one still deduplicates repeats inside that query. Input should be canonicalized.structural_hash(node, cache)&OperatorNode,&mut HashCache→u64result_kind,schema,guarantee,timing,coverage) plus each child's hash. Bucket key only, never the sharing decision.HashCacheHashMap<*const OperatorNode, u64>dag_node_count(root)&Rc<OperatorNode>→usizerootby pointer (operator inputs and scalar-referenced nodes). Measures sharing.LogicalASAPDAGDocumentschema_versionu32LOGICAL_ASAP_DAG_WIRE_VERSION(1). Set bynew. Version 1 is separate from the older phase-assigned format.dagLogicalASAPDAGLogicalASAPDAGnodesVec<LogicalASAPDAGNode>nodes[i].id == LogicalASAPNodeId(i);validateonly requires unique IDs.edgesVec<LogicalASAPDAGEdge>rootLogicalASAPQueryRootLogicalASAPDAGNodeidLogicalASAPNodeIdpayloadLogicalASAPOperatorPayloadresult_kindOperatorResultKindRelation,InstantVector,RangeVectororState(unfinalized summary/accumulator state). Copied from the node.output_schemaSchemaschema(#535).guaranteeOption<ResultGuarantee>Nonemeans not assessed, not exact.coverageOption<SummaryCoverage>Someonly onStatenodes. Required on everySummaryAgg. On aSummaryMergeit must equalSummaryCoverage::merge_disjointof its input nodes' coverage. Missing in JSON reads asNone.LogicalASAPDAGEdgeproducerLogicalASAPNodeIdconsumerLogicalASAPNodeIdroleEdgeRoleInput: single or n-ary input (child,Concat/SummaryMergechildren,summary_input).Left/Right: two-sided operators (Join,SetOp,BinaryOp,SummarySubtract,SummaryJoinouter/inner).ScalarRef: the consumer reads the producer inside a scalar expression (scalar(v), scalar subquery,EXISTS,IN, or aSummaryAggfilter).intermediate_schemaSchemavalidaterequires equality with the producer'soutput_schema.groupingGroupingEdgeCompatibilitySummaryAgg→SummaryAggedges, from theirreductions.Identical: equal reductions.ConsumerCoarsensProducer:PerEntity→Reduce(_), orReduce(p)→Reduce(c)with both notwithoutand every key ofcinp.Incompatible: any other pair ofSummaryAggs.NotApplicable: every other edge.LogicalASAPQueryRootOperator(LogicalASAPNodeId)Scalar(WireScalarExpr)42.0) produces zero nodes; a scalar subquery produces the referenced producer DAG.operator_refs()Operator, all referenced IDs forScalar. Reachability starts here.LogicalASAPOperatorPayload(each ASAP variant is the matchingASAPOpwithout children)Relational { operator: NonASAPOpKind }SummaryAgg.familyFieldDataType: committed summary family and parameters (e.g.Sketch(KLL{k=200}, …)).validaterequires a schema field with this type.SummaryAgg.inputSummaryUpdate: what each row contributes.SummaryAgg.reductionReduction: grouping (PerEntityorReduce(keys)).SummaryAgg.groupingGroupingStrategy.validaterequires everySketchfield in the schema to carry this strategy.SummaryAgg.filterOption<WirePredicate>: pre-aggregation filter; its operator references becomeScalarRefedges.SummaryEstimate.querySketchStatisticread from the state (e.g.Quantile { q }).FinalizeExactAccumulatorMaintainPopulation.populationMaintainedPopulation<OperatorNode>, cloned as is.EvaluatePopulation.evaluationPopulationStatistic.SummaryMergeInputedges.SummarySubtractSummaryDelete.key,SummaryJoin.keyColumnIdof the key column.SummaryJoin.familyFieldDataTypeof the joined summary.Extension.nameLogicalASAPDAGValidationErrorUnsupportedVersion(v)schema_version != 1(document only).DuplicateNode(id)MissingNode(id)EdgeSchemaMismatch(producer)intermediate_schemadiffers from the producer'soutput_schema.SummarySchemaMismatch(id)SummaryAggschema lacks a field offamily, or has aSketchfield with a different grouping strategy.InvalidCoverage(id)Statenode; coverage failsSummaryCoverage::validate;SummaryAggwithout coverage; aSummaryMergeinput without coverage, an input union thatmerge_disjointrejects, or a stored merge coverage that differs from the union.CycleUnreachableNode(id)Compilation and identity
compile_logical_asap_dag(root)compile_logical_asap_query(&QueryRoot::Operator(root)), without the map.compile_logical_asap_query(root)root.validate_structure()(returns itsSchemaDerivationError), then exports. Does not callLogicalASAPDAG::validate.*_with_node_idsLogicalASAPDAGCompilation.dagLogicalASAPDAGCompilation.node_idsLogicalASAPNodeIdentityMap: in-memory only, not serialized.node_id(&rc)Rc(pointer equality), orNone.operator_node(id)Rc<OperatorNode>exported asid, orNone.Wire types
LogicalASAPNodeId(pub u32)WireScalarExprScalarExprwith each operator reference (PromqlScalarFromVector,ScalarSubquery,Exists.subquery,InSubquery.subquery) replaced by the producer's ID. Every other variant keeps its fields, includingExprSemantics.WireScalarExpr::operator_refs()WireScalarExpr::from_expr(expr, id_of)expr, resolving each referenced node throughid_of.Examples
Example 1: summary merge export (
crates/types/tests/logical_export.rs,merged_summary_preserves_typed_state)Input: one
Valuesnode, one KLL (k = 200)SummaryAggover it, cloned twice with #567 coveragetime_ms = 0..1and1..2(samesource,input,reduction), and aSummaryMergeof the two. No node hastiming.compile_logical_asap_dag(&root)produces (post-order IDs):Relational { Values }RelationNoneSummaryAgg { family: KLL{k=200}, … }State[0,1)SummaryAgg { … }State[1,2)SummaryMergeStateroot.coverageEdges:
0→1,0→2,1→3,2→3, allrole: Input,grouping: NotApplicable. The test checks 4 nodes,dag.validate()passes, and the merge node keepsresult_kind,output_schemaandcoverage. Then it setscoverage = Noneon oneSummaryAggandvalidate()returnsInvalidCoverage(<that id>).Example 2: CSE on the identical-expression rule (
crates/types/src/ir/cse.rstests)The
BinaryOpfrom the Problem section, aftershare_common_sub_dags(vec![("q", root)]):lhsandrhsare the sameRc(single_query_shares_its_own_repeated_sub_dag). The grouped aggregate has unique keyby [1], so it is legal to share.Quantile q=0.5 col=2in two roots (median_and_explicit_half_percentile_merge)col=2vscol=3(distinct_column_quantiles_do_not_merge)no_unique_keys_means_no_merge_even_when_structurally_identical)without(..)aggregate (group_keys_gate_still_prevented_when_partition_by_without_used)Dedup { cols: [1] }(dedup_gates_sharing_the_same_as_aggregate)Dedupadds a unique keyvector(scalar(sum by (service)(up)))twice (scalar_referenced_vector_is_shared_across_queries)SummaryAgg,alpha=0.01, same guarantee (asap_nodes_share_without_a_unique_key)alpha=0.01vs0.001; exact guarantee vsNone(asap_nodes_with_distinct_parameters_or_guarantees_are_not_shared)SummaryEstimatep95 and p99 over equal producers (evaluations_share_their_producer_but_not_each_other)q0.0vs-0.0,+infvs-inf,NaNvsNaN(signed_zero_and_nonfinite_values_remain_distinct)+infvs+infis sharedcoveragecoverageis in hash and equality (no dedicated test)Example 3: canonicalization (
crates/types/src/ir/canonicalize.rstests)Limit 5 { Sort desc col 1 { Aggregate(by [1], [Count]) } }Aggregate(by [], [TopK{k:5}]) { Aggregate(by [1], [Count]) }(promotes_count_ranked_limit_sort); also through a passthroughProjectFilter { EXISTS (s) }Join { Semi, true, left, s }(exists_filter_becomes_semi_join)Filter { NOT EXISTS (s) }Join { Anti, true, left, s }Filter { service IN (s) }over a 3-column leftJoin { Semi, service = Column(3) }Filter { value > 1 AND EXISTS (s) }Filter { value > 1 } { Join { Semi } }Filter { NOT IN (s) }Project [ScalarSubquery(s) AS a, ScalarSubquery(s) AS b]s(scalar_subqueries_remain_explicit_and_shared)Filter { rn <= 5 } { ROW_NUMBER window }Every rewrite test also checks
canonicalize(out)returns the sameRc.Example 4: transport rejections (
malformed_transport_is_rejected,standalone_scalar_roots_roundtrip_without_synthetic_operators)schema_version = 99UnsupportedVersion(99)DuplicateNodeLogicalASAPNodeId(9)MissingNodeUnreachableNode"output_state": {"timing": "ingestion_time"}deny_unknown_fields)QueryRoot::Scalar(literal 42.0)QueryRoot::Scalar(ScalarSubquery(values))scalar_dependencies_share_one_exported_producer: aProjectwhosechildandScalarSubqueryare the sameValuesexports 2 nodes and 2 edges (oneInput, oneScalarRef), and the JSON contains none ofoutput_state,data_state,timing,retention,window.Out of scope
timingbut has no evaluation-context field to compare.OperatorNode.timingstays common IR metadata and is not exported.pre_asap::canonicalize,pre_asap::cse,post_asap::cseanddag_exportto the new modules. Downstream physical PRs keep their legacy base until that refactor.Reviewer notes:
compile_logical_asap_*does not runLogicalASAPDAG::validate. ASummaryAggwithcoverage = Nonecompiles but then failsvalidate()withInvalidCoverage.canonicalize.rsstill describeROW_NUMBERrewriting and scalar-subquery cross-join lowering (apply_local_rules,filter). The code does neither, and the tests assert the shapes stay unchanged.Stack and validation
Revised logical foundation 3/6 · Previous: #560 · Next: #539 · Tracker: #528
Order: #567 → #560 → #537 → #539 → #540 → #561.
logical_export_accepts_unassigned_timing) failed on the original timing requirement and passes now.🤖 Generated with Claude Code