Skip to content

feat: add unified operators and owned scalar expressions - #536

Merged
zzylol merged 3 commits into
mainfrom
stack/528-02-ir
Oct 3, 2026
Merged

zzylol merged 3 commits into
mainfrom
stack/528-02-ir

Conversation

@zzylol

@zzylol zzylol commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Part 2/8 of #528's review stack, based on main, following the merged schema changes in #535 and terminology changes in #549. Implements the core data structures in both #511 documents.

Before this PR: the workspace has a common schema but still only the legacy query/summary graph types.

After this PR: OperatorNode combines NonASAPOp and ASAPOp with shared schema, result kind, guarantee, and timing. Operators own ScalarExpr; standalone scalar queries use QueryRoot::Scalar. Structural validation checks scopes, operand kinds, state families, and derived schemas independently of phase assignment.

Node construction is category-independent: new_shared(Operator) derives the schema for either operator category and returns Result<Rc<OperatorNode>, SchemaDerivationError>. Explicit output names and qualifiers use the common with_schema builder; guarantees use with_guarantee. map_children re-derives structural metadata for both categories, retaining only naming overrides and clearing assessed guarantee/timing. Validation allows those naming overrides but requires the complete structural schema (including unique keys, time index, and open/closed status) to match derivation.

Existing production callers stay on their legacy graph until the cutover layer. During that transition, operator parameters reuse their existing definitions and maintained-population inputs are generic over the graph node type; the cleanup layer removes this temporary coexistence. The SketchQuery → SketchStatistic and PopulationReadout → PopulationStatistic renames are isolated in #549 and are absent from this PR’s incremental diff. This layer introduces SchemaDerivationError. The scalar type-checking module is named pre_asap::scalar_type_rules (formerly scalar_signature) to reflect its type/nullability rules for map, element, struct, and PromQL functions.

Validation: all workspace targets compile; types unit, structure-contract, and doctests pass; formatting and workspace/all-target/all-feature clippy pass. Graph algorithms and frontend lowering are intentionally reviewed in the next layers.

Code interface design: LogicalDAG → LogicalASAPDAG

The two stages use the same ir::OperatorNode and ir::ScalarExpr representation. In this layer, LogicalDAG and LogicalASAPDAG describe planning stages, not separate Rust graph types. Frontend logical plans contain NonASAPOp; ASAP-aware logical plans can mix NonASAPOp and ASAPOp, connected through the same Rc<OperatorNode> edges. Optimization changes the operations and their inputs rather than converting the surrounding graph to another representation.

Shared operator interface

The core declarations are:

pub enum Operator {
    NonASAP(NonASAPOp),
    ASAP(ASAPOp),
}

pub enum OperatorResultKind {
    Relation,
    InstantVector,
    RangeVector,
    State,
}

pub struct OperatorNode {
    pub operator: Operator,
    pub result_kind: OperatorResultKind,
    pub schema: Schema,
    pub guarantee: Option<ResultGuarantee>,
    pub timing: Option<ExecutionTiming>,
}

pub enum QueryRoot {
    Operator(Rc<OperatorNode>),
    Scalar(ScalarExpr),
}

Both operator categories expose children(), map_children(...), output_schema(), output_kind(), and validate_inputs(). Ordinary operators can consume summary readouts, and summary producers can consume ordinary relational subplans. State identifies an unfinalized summary or accumulator; an identical column schema does not make it interchangeable with a relation or vector.

Construction and rewriting use these OperatorNode methods:

pub fn new(operator: Operator) -> Result<Self, SchemaDerivationError>;
pub fn new_shared(operator: Operator)
    -> Result<Rc<Self>, SchemaDerivationError>;
pub fn with_schema(operator: Operator, schema: Schema) -> Self;
pub fn map_children(
    &self,
    f: impl FnMut(&Rc<OperatorNode>) -> Rc<OperatorNode>,
) -> Result<Self, SchemaDerivationError>;
pub fn validate_structure(self: &Rc<Self>)
    -> Result<(), SchemaDerivationError>;

new derives the schema and result kind. with_schema supports output naming and qualification overrides; structural validation still requires derived types, nullability, unique keys, time index, and open/closed schema status to agree. Rewriting through map_children derives structural metadata again, preserves explicit naming overrides when arity permits, and clears the old accuracy guarantee and execution timing. Shared nodes use Rc identity; equal-looking nodes are not automatically interned.

Owned scalar expression interface

ScalarExpr is a value-computation tree owned by an operator field, such as Predicate, ProjectItem.expr, SortKey.expr, aggregate filters, or HAVING. Recursive scalar operands use Box<ScalarExpr> or Vec<ScalarExpr>. A standalone scalar query is QueryRoot::Scalar; it does not need a synthetic operator node.

Columns are positional ColumnIds interpreted against the owning operator's scope. Expressions include literals, arithmetic, comparisons, Boolean operations, casts, function calls, CASE, and evaluation timestamps. Arithmetic, comparison, and negation carry ExprSemantics::{Sql, Promql} so language behavior survives lowering even when both languages use Float64.

The scalar traversal and typing APIs are:

pub fn children(&self) -> Vec<&ScalarExpr>;
pub fn operator_refs(&self) -> Vec<&Rc<OperatorNode>>;
pub fn map_operator_refs(
    &self,
    f: &mut impl FnMut(&Rc<OperatorNode>) -> Rc<OperatorNode>,
) -> ScalarExpr;
pub fn columns_referenced(&self) -> Vec<ColumnId>;
pub fn scalar_type(&self, schema: &Schema)
    -> Result<(DataType, bool), SchemaDerivationError>;

The Boolean in scalar_type is nullability. Scope is operator-specific: projection and row filters use the input schema, join predicates use the combined input scope, and HAVING uses the aggregate output scope. Standalone scalar roots are typed against an empty column scope.

Four scalar variants explicitly read operator results: PromqlScalarFromVector, ScalarSubquery, Exists, and InSubquery. They hold Rc<OperatorNode> references. operator_refs() finds these dependencies recursively, and OperatorNode::children() includes them alongside ordinary operator inputs. Rewriters must also visit references inside QueryRoot::Scalar, so subqueries and scalar/vector conversions are not missed.

What changes across the logical-stage boundary

A representative quantile replacement has this shape (schematic, not executable Rust):

LogicalDAG:
  NonASAP::Project { expr: Column(result), child:
    NonASAP::Aggregate { measures: [Quantile(...)], child: input }
  }

LogicalASAPDAG candidate:
  NonASAP::Project { expr: Column(result), child:
    ASAP::SummaryEstimate { query: Quantile(...), summary_input:
      ASAP::SummaryAgg { family: KLL(...), input: ..., child: input, ... }
    }
  }

The projection and its scalar expression keep the same interfaces. SummaryAgg commits the summary family, parameters, update inputs, reduction, and grouping and produces State; SummaryEstimate exposes a query result for downstream ordinary operators. Exact alternatives use FinalizeExactAccumulator or maintained-population readouts. A rewrite must preserve the downstream column contract or explicitly remap positional references.

Structure is validated independently of lifecycle assignment. guarantee: None means accuracy is unassessed, not exact. timing: None means no execution phase has been assigned; timing validation requires assigned phases and rejects ingestion-time nodes that depend on query-time or unassigned inputs.

Scope of this PR

This PR establishes the representation and structural contracts. Graph algorithms, frontend lowering, planner cutover, and executable export are reviewed in subsequent stack layers. In #536 specifically, SummaryMerge, SummarySubtract, SummaryDelete, SummaryJoin, and Extension remain reserved and fail closed; their enum presence does not imply executable support.

Source: node.rs, scalar.rs, non_asap.rs, asap.rs, and query.rs.


Stack 2/8 · Previous: #535 and #549 (merged) · Next: #537 · Reference/tracker: #528

Selvomega
Selvomega previously approved these changes Oct 2, 2026
Comment thread crates/asap-aware-mapping/src/cost_model.rs
Comment thread crates/asap-aware-mapping/src/maintained_population.rs
Comment thread crates/types/src/ir/node.rs Outdated
@zzylol
zzylol changed the base branch from main to refactor/summary-statistic-names October 2, 2026 21:14
@zzylol
zzylol changed the base branch from refactor/summary-statistic-names to main October 2, 2026 21:25
Comment thread crates/types/src/ir/scalar.rs Outdated
Comment thread crates/types/src/ir/scalar.rs
Comment thread crates/types/src/ir/query.rs
@zzylol
zzylol merged commit a718a09 into main Oct 3, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants