Skip to content

feat(executor): execute the exact SQL entropy fallback; Example 2 status - #599

Draft
zzylol wants to merge 2 commits into
stack/509-w7b-2-sql-frequency-recognitionfrom
stack/509-w7b-sql-frequency
Draft

zzylol wants to merge 2 commits into
stack/509-w7b-2-sql-frequency-recognitionfrom
stack/509-w7b-sql-frequency

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Wave 1 chain: #594 → #593 → #592 → #591 → #595 → #596 → #597 → #598 → #599 (on #589)

Why

#598 recognizes the SQL entropy idiom only under narrow conditions. Any other form, such as a nullable src_ip, stays relational, and its native binding failed:

Piece Before
SUM(COUNT(*)) OVER () value operation has no native implementation
LN(p) expression validate rejects ln

The parked #565 added both on the old stack. This PR ports it and records where #509 Example 2 stands on the current Stage 1 to Stage 3 pipeline. Part of #509 / #580 (W8).

What

  • Operator::sql_window_sum(input, column, name) appends one nullable total to every row (SUM semantics, NULL for empty or all-NULL input). It is memory-accounted and cooperative, and survives the UncheckedOperator round trip. Binding accepts only SUM(col) OVER () with the unbounded frame. Partitioned, ordered and finite frames return native SQL window SUM requires the complete unordered relation.
  • SQL ln uses the same numeric path as sqrt.
  • integration-tests/tests/planner_layering_example2.rs records Example 2's status.

After: the original Q2 SQL and its recognized form return the same values on every fixture: NULL, -0.0, ln 2, and -0.75 ln 0.75 - 0.25 ln 0.25.

Example 2 status (asserted by planner_layering_example2)

Q1 distinct Q2 entropy Q3 L2
Lowers to the intent Cardinality FrequencyEntropy (the doc's text) FrequencyL2 only with a Float64 product. The doc's SUM(c * c) is Int64 and stays relational
Time filter WHERE ts >= now() - INTERVAL '1 minute' becomes a Scan predicate (CAST(ts) >= CAST(now() - 60 s)). It is not a per-measure FILTER and not a TimeRange, and Pass 1 accepts it same same
plan_stages plans; Stage 1 offers PassThrough + UnivMon (+ HLL) plans; PassThrough + UnivMon plans; PassThrough + UnivMon
Exact execution 2 entropy in nats √5

Remaining blockers for the shared UnivMon plan:

  1. No accuracy model for UnivMon, so Stage 3 rejects every UnivMon candidate (no accuracy model for UnivMon). Another agent is adding this.
  2. No summary-capability rule in Pass 2, so the three UnivMon builds are never merged. Another agent is adding this.
  3. The time filter carries no lookback that Stage 3 or coverage can see: it is a scan predicate, not a TimeRange. The runtime also has no now(): binding the windowed queries fails with unsupported scalar expression: CurrentTimestamp, so the exact runs above drop the filter.
  4. The design's integer Q3 is not recognized (overflow).
  5. Unrelated to sharing: Stage 1 also offers an exact Count accumulator for each population guard, and it fails Stage 2 over SQL rows (ColumnRef::SampleValue has no value column).

Tests

Ported from #565: complete_sql_sum_window_preserves_rows_and_nulls, sql_ln_executes_numeric_and_null_arguments, complete_window_sum_enforces_workspace_budget, the original-vs-rewrite entropy comparison, and native_sql_sum_window_rejects_other_frames. Before the port, both entropy integration tests fail. The developer-doc additions from #565 are not ported.

Gate: fmt and clippy (-D warnings) pass. cargo test --workspace: 1,546 passed, 10 ignored (#589: 1,517 / 10).

Ports #565. Stacked on #598.

🤖 Generated with Claude Code

zzylol and others added 2 commits October 5, 2026 04:48
Port of the parked #565: SQL LN and a complete, unordered
`SUM(column) OVER ()` window (SQLWindowSum) run natively, so the original
Q2 SQL of #509 Example 2 executes and matches its recognized entropy form.
Partitioned, ordered and finite frames stay refused.

planner_layering_example2 records Example 2's status: the three queries
lower to Cardinality / FrequencyEntropy / FrequencyL2 (the design's integer
Q3 does not), the time filter is a scan predicate, plan_stages plans the
workload with UnivMon offered to each statistic, and the exact candidates
execute (without the time filter: the runtime has no now() yet).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
plan_stages takes per-root demand since #594, and DataWorkload declares metric
types since #593; Example 2's pipeline test runs each query once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-w7b-2-sql-frequency-recognition branch from 66bc1a6 to e211fba Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-w7b-sql-frequency branch from 2798c38 to 0b2194b Compare October 5, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant