feat: add unified operators and owned scalar expressions - #536
Merged
Merged
Conversation
This was referenced Oct 2, 2026
zzylol
force-pushed
the
stack/528-02-ir
branch
2 times, most recently
from
October 2, 2026 19:40
8f384f7 to
d8f0b1a
Compare
zzylol
force-pushed
the
stack/528-01-schema
branch
from
October 2, 2026 19:40
f57b64b to
93ed497
Compare
Selvomega
previously approved these changes
Oct 2, 2026
zzylol
force-pushed
the
stack/528-02-ir
branch
from
October 2, 2026 21:14
d8f0b1a to
dd71af0
Compare
zzylol
force-pushed
the
stack/528-02-ir
branch
from
October 2, 2026 21:22
dd71af0 to
7d5c382
Compare
zzylol
force-pushed
the
stack/528-02-ir
branch
from
October 2, 2026 21:25
7d5c382 to
6aba8f2
Compare
zzylol
commented
Oct 3, 2026
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2/8 of #528's review stack, based on
main, following the merged schema changes in #535 and terminology changes in #549. Implements the core data structures in both #511 documents.Before this PR: the workspace has a common schema but still only the legacy query/summary graph types.
After this PR:
OperatorNodecombinesNonASAPOpandASAPOpwith shared schema, result kind, guarantee, and timing. Operators ownScalarExpr; standalone scalar queries useQueryRoot::Scalar. Structural validation checks scopes, operand kinds, state families, and derived schemas independently of phase assignment.Node construction is category-independent:
new_shared(Operator)derives the schema for either operator category and returnsResult<Rc<OperatorNode>, SchemaDerivationError>. Explicit output names and qualifiers use the commonwith_schemabuilder; guarantees usewith_guarantee.map_childrenre-derives structural metadata for both categories, retaining only naming overrides and clearing assessed guarantee/timing. Validation allows those naming overrides but requires the complete structural schema (including unique keys, time index, and open/closed status) to match derivation.Existing production callers stay on their legacy graph until the cutover layer. During that transition, operator parameters reuse their existing definitions and maintained-population inputs are generic over the graph node type; the cleanup layer removes this temporary coexistence. The
SketchQuery→SketchStatisticandPopulationReadout→PopulationStatisticrenames are isolated in #549 and are absent from this PR’s incremental diff. This layer introducesSchemaDerivationError. The scalar type-checking module is namedpre_asap::scalar_type_rules(formerlyscalar_signature) to reflect its type/nullability rules for map, element, struct, and PromQL functions.Validation: all workspace targets compile; types unit, structure-contract, and doctests pass; formatting and workspace/all-target/all-feature clippy pass. Graph algorithms and frontend lowering are intentionally reviewed in the next layers.
Code interface design: LogicalDAG → LogicalASAPDAG
The two stages use the same
ir::OperatorNodeandir::ScalarExprrepresentation. In this layer,LogicalDAGandLogicalASAPDAGdescribe planning stages, not separate Rust graph types. Frontend logical plans containNonASAPOp; ASAP-aware logical plans can mixNonASAPOpandASAPOp, connected through the sameRc<OperatorNode>edges. Optimization changes the operations and their inputs rather than converting the surrounding graph to another representation.Shared operator interface
The core declarations are:
Both operator categories expose
children(),map_children(...),output_schema(),output_kind(), andvalidate_inputs(). Ordinary operators can consume summary readouts, and summary producers can consume ordinary relational subplans.Stateidentifies an unfinalized summary or accumulator; an identical column schema does not make it interchangeable with a relation or vector.Construction and rewriting use these
OperatorNodemethods:newderives the schema and result kind.with_schemasupports output naming and qualification overrides; structural validation still requires derived types, nullability, unique keys, time index, and open/closed schema status to agree. Rewriting throughmap_childrenderives structural metadata again, preserves explicit naming overrides when arity permits, and clears the old accuracy guarantee and execution timing. Shared nodes useRcidentity; equal-looking nodes are not automatically interned.Owned scalar expression interface
ScalarExpris a value-computation tree owned by an operator field, such asPredicate,ProjectItem.expr,SortKey.expr, aggregate filters, orHAVING. Recursive scalar operands useBox<ScalarExpr>orVec<ScalarExpr>. A standalone scalar query isQueryRoot::Scalar; it does not need a synthetic operator node.Columns are positional
ColumnIds interpreted against the owning operator's scope. Expressions include literals, arithmetic, comparisons, Boolean operations, casts, function calls,CASE, and evaluation timestamps. Arithmetic, comparison, and negation carryExprSemantics::{Sql, Promql}so language behavior survives lowering even when both languages useFloat64.The scalar traversal and typing APIs are:
The Boolean in
scalar_typeis nullability. Scope is operator-specific: projection and row filters use the input schema, join predicates use the combined input scope, andHAVINGuses the aggregate output scope. Standalone scalar roots are typed against an empty column scope.Four scalar variants explicitly read operator results:
PromqlScalarFromVector,ScalarSubquery,Exists, andInSubquery. They holdRc<OperatorNode>references.operator_refs()finds these dependencies recursively, andOperatorNode::children()includes them alongside ordinary operator inputs. Rewriters must also visit references insideQueryRoot::Scalar, so subqueries and scalar/vector conversions are not missed.What changes across the logical-stage boundary
A representative quantile replacement has this shape (schematic, not executable Rust):
The projection and its scalar expression keep the same interfaces.
SummaryAggcommits the summary family, parameters, update inputs, reduction, and grouping and producesState;SummaryEstimateexposes a query result for downstream ordinary operators. Exact alternatives useFinalizeExactAccumulatoror maintained-population readouts. A rewrite must preserve the downstream column contract or explicitly remap positional references.Structure is validated independently of lifecycle assignment.
guarantee: Nonemeans accuracy is unassessed, not exact.timing: Nonemeans no execution phase has been assigned; timing validation requires assigned phases and rejects ingestion-time nodes that depend on query-time or unassigned inputs.Scope of this PR
This PR establishes the representation and structural contracts. Graph algorithms, frontend lowering, planner cutover, and executable export are reviewed in subsequent stack layers. In #536 specifically,
SummaryMerge,SummarySubtract,SummaryDelete,SummaryJoin, andExtensionremain reserved and fail closed; their enum presence does not imply executable support.Source:
node.rs,scalar.rs,non_asap.rs,asap.rs, andquery.rs.Stack 2/8 · Previous: #535 and #549 (merged) · Next: #537 · Reference/tracker: #528