Skip to content

Expose accumulator state to allow prefix scanning - #24035

Merged
avantgardnerio merged 5 commits into
apache:mainfrom
avantgardnerio:brent/bwag-finalized-state-observer
Aug 11, 2026
Merged

Expose accumulator state to allow prefix scanning#24035
avantgardnerio merged 5 commits into
apache:mainfrom
avantgardnerio:brent/bwag-finalized-state-observer

Conversation

@avantgardnerio

@avantgardnerio avantgardnerio commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Expose state of aggregate streams within BWAG so downstream prefix scanning can take place.

API

// physical-plan/src/windows/bounded_window_agg_exec.rs
pub type FinalizedWindowStateObserver = Arc<
    dyn Fn(usize, &PartitionKey, &[Option<Vec<ScalarValue>>]) -> Result<()>
        + Send + Sync,
>;

impl BoundedWindowAggExec {
    pub fn with_finalized_state_observer(mut self, obs: FinalizedWindowStateObserver) -> Self {}
}

// physical-expr/src/window/window_expr.rs
impl WindowState {
    /// `Accumulator::state()` if this is an aggregate window function, `None` otherwise.
    pub fn aggregate_state(&mut self) -> Result<Option<Vec<ScalarValue>>> {}
}

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate labels Jul 31, 2026
@avantgardnerio
avantgardnerio marked this pull request as draft July 31, 2026 18:55
@avantgardnerio avantgardnerio changed the title feat(physical-plan): FinalizedWindowStateObserver on BoundedWindowAggExec Expose accumulator state to allow prefix scanning Jul 31, 2026
@avantgardnerio
avantgardnerio force-pushed the brent/bwag-finalized-state-observer branch from ae37c0b to 65014d5 Compare July 31, 2026 18:58
@avantgardnerio
avantgardnerio requested a review from Dandandan July 31, 2026 18:58
@codecov-commenter

codecov-commenter commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.00000% with 108 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.01%. Comparing base (33ad1cc) to head (aa4d196).
⚠️ Report is 18 commits behind head on main.

Files with missing lines Patch % Lines
...ysical-plan/src/windows/bounded_window_agg_exec.rs 85.01% 21 Missing and 49 partials ⚠️
datafusion/physical-plan/src/windows/mod.rs 72.72% 13 Missing and 5 partials ⚠️
datafusion/physical-expr/src/window/window_expr.rs 58.97% 13 Missing and 3 partials ⚠️
...zer/src/ensure_requirements/enforce_sorting/mod.rs 84.00% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24035      +/-   ##
==========================================
+ Coverage   80.99%   81.01%   +0.02%     
==========================================
  Files        1106     1106              
  Lines      383352   384686    +1334     
  Branches   383352   384686    +1334     
==========================================
+ Hits       310488   311665    +1177     
- Misses      54544    54620      +76     
- Partials    18320    18401      +81     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@avantgardnerio
avantgardnerio marked this pull request as ready for review August 1, 2026 16:24
@neilconway

Copy link
Copy Markdown
Contributor

Thanks for this contribution @avantgardnerio ! I'd find it helpful if you'd elaborate a little bit more about the motivation for this change in the PR description. For example, some intended use-cases, what kind of performance improvement this unlocks, etc.

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

@neilconway I'm trying to speed up window functions using parallel prefix scans. I am presently incubating this in Ballista, and this is the minimum API exposure that I need to do it for non-decomposable operations like approx_distinct() (vs others like AVG, STDDEV via Welford / Chan, etc). Though, the work is certainly not limited to Ballista, it has shown improvement in DataFusion as well.

The jury is still out about re-partition cost vs performance benefit, but the signs are hopeful:

image

And at least from a big-O time perspective (table 1) it should be optimal for some queries (select my_agg() over unbounded preceding...)

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

@gene-bordegaray and @JSOD11 you guys might be interested as well.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @avantgardnerio and @neilconway -- I left some comments

Comment thread datafusion/physical-expr/src/window/window_expr.rs Outdated
Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs Outdated
Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs Outdated
Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs Outdated
Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs Outdated

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may not fully understand prefix scanning, but it seems to me like this API will only give you access to the window state for the single last row in each partition.

Don't you potentially need access to the window state for the last N rows in a partition (e.g the HALO rows) 🤔

@alamb

alamb commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

FWIW claude claims this doesn't get run with windows like

UNBOUNDED PRECEDING → CURRENT ROW

@alamb

alamb commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

I think it would also be super useful to add some sort of example / test that shows how you intend to use this API (for exmple some simple example for computing a window function in parallel or something 🤔

that way we could see the API in action

@avantgardnerio

Copy link
Copy Markdown
Contributor Author

this doesn't get run with windows like

Thanks @alamb ! That was a critical bug that would have defeated the whole point. It is now fixed and asserted in test_finalized_state_observer_fires_on_causal_frame()

@avantgardnerio

avantgardnerio commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

access to the window state for the single last row

Yes, this is exactly what is required.

need access to the window state for the last N rows

No, not for prefix scanning. (answer below)

e.g the HALO rows

The HPC "halo" term is a good fit for bounded preceding/following (surrounding cells, in 1D) but doesn't extend cleanly to "last row of every other partition." Regardless of the name, this PR doesn't take that approach - because although it works for SUM, and decomposes for AVG (sum+count), it fails by the time you get to arbitrary accumulators like approx_distinct.

add some sort of example / test

Which is exactly where (the newly added) test_prefix_scan_across_tasks_matches_single_bwag() comes in. It shows that with only the accumulator state of the very last row of the entire (DF) partition, parallel partitioned prefix-scanned sums produce exactly the same results as a single (DF) partition BWAG. Given the observer exposes Accumulator::state() directly, any function that supports Accumulator::merge_batch - including approx_distinct - can be prefix-scanned the same way, with the downstream consumer merging sketches instead of adding scalars.

Edit: added the qualifier (DF) partition to distinguish between the ambiguously named (SQL window) partition.

@avantgardnerio
avantgardnerio requested a review from alamb August 6, 2026 17:05
@avantgardnerio

avantgardnerio commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Pseudo code, stripped directly from the test_prefix_scan_across_tasks_matches_single_bwag test, for those who value brevity:

        // Two tasks under range partition on sn:
        let (task1_out, task1_total) = run_running_sum_task(&[1, 1, 2, 2, 3, 3, 4, 4]);
        let (task2_out, task2_total) = run_running_sum_task(&[5, 5, 6, 6, 7, 7, 8, 8]);

        // Local (uncorrected) outputs and totals — first pass.
        assert_eq!(task1_out, vec![1, 2, 4, 6, 9, 12, 16, 20]);
        assert_eq!(task2_out, vec![5, 10, 16, 22, 29, 36, 44, 52]);

        // Prefix scan over per-task totals → carry-in for each task. Task 0's
        // carry-in is 0; task N's carry-in is the sum of tasks [0, N).
        let carry_ins = [0u64, task1_total];

        // Second pass: shift each task's local values by its carry-in.
        let task1_final: Vec<u64> = task1_out.iter().map(|v| v + carry_ins[0]).collect();
        let task2_final: Vec<u64> = task2_out.iter().map(|v| v + carry_ins[1]).collect();
        let parallel_result: Vec<u64> = task1_final.iter().chain(task2_final.iter());

        // Oracle: single BWAG over the full concatenated input.
        let (single_result, single_total) = run_running_sum_task(
            &[1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8]);

        assert_eq!(parallel_result, single_result);
        assert_eq!(single_result,
            vec![1, 2, 4, 6, 9, 12, 16, 20, 25, 30, 36, 42, 49, 56, 64, 72]
        );

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs
Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs Outdated
Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs Outdated
Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs Outdated
Comment thread datafusion/physical-plan/src/windows/bounded_window_agg_exec.rs
avantgardnerio added a commit to avantgardnerio/arrow-datafusion that referenced this pull request Aug 8, 2026
Match the field's type so with_new_children collapses to a single chained
call and the setter can also clear a previously-installed observer.

Addresses apache#24035 review comment 3738803158.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
avantgardnerio added a commit to avantgardnerio/arrow-datafusion that referenced this pull request Aug 8, 2026
The method now always mutates when called and takes the observer as a
required argument; the "is observer installed?" check moves to the caller
in `compute_aggregates`. Removes the "&mut self that only mutates when
observer is set" shape.

Addresses apache#24035 review comment 3738822701.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
avantgardnerio added a commit to avantgardnerio/arrow-datafusion that referenced this pull request Aug 8, 2026
Rename the trait method to `finalize_window_aggregate` and split its
signature so the callback fires once per aggregate window expression per
closing PARTITION BY group, receiving that expression's Arc and its own
`Accumulator::state` directly. Non-aggregate window functions no longer
fire the callback at all.

Removes the per-partition-key `Vec<Option<Vec<ScalarValue>>>` wrapper
allocation, and gives the observer the window-expression context needed
to disambiguate calls when the exec carries multiple window expressions.

Leaves room to add a peer `finalize_window_function` later for built-in
(non-aggregate) window functions.

Addresses apache#24035 review comments 3738790091 and 3738816488.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
avantgardnerio added a commit to avantgardnerio/arrow-datafusion that referenced this pull request Aug 8, 2026
…close tests

The two `test_finalized_state_observer_*` tests were structurally
identical apart from the window frame. Fold their common setup and
assertions into a single async helper that takes the frame, so each test
body is now just the frame construction + a one-line comment explaining
which causality regime it exercises.

Addresses apache#24035 review comment 3738845164.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@avantgardnerio
avantgardnerio requested a review from alamb August 8, 2026 22:40
@github-actions github-actions Bot added the optimizer Optimizer rules label Aug 10, 2026
@avantgardnerio

avantgardnerio commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

@timsaucer thank you for the thorough review. These are all legitimate concerns:

  1. I was not able to reproduce this, but I added a test anyway, as well as your suggested mitigation
  2. Following the belt-and-suspenders theme, I made this case hard-error in a central place
  3. I've updated all the construction sites to include the callback during clones
  4. It now prevents installation of a callback if any frames are sliding.

I think all of the above stem from this trying to fire per-partition-key, rather than once, at the end, when the motivating case is prefix scanning. What would you think about a getter shape where a downstream node reads Some(state) / None from the BWAG it already holds via input()? Scope is single-group-per-output-partition (matches distributed prefix-scan; wouldn't cover per-key PARTITION BY windows). Concerns 1/2/3 collapse to not-applicable, and 4 is a one-line gate at the write site.

@timsaucer timsaucer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for all the work on this!

@avantgardnerio
avantgardnerio added this pull request to the merge queue Aug 11, 2026
Merged via the queue into apache:main with commit a253a6a Aug 11, 2026
40 checks passed
@avantgardnerio
avantgardnerio deleted the brent/bwag-finalized-state-observer branch August 11, 2026 16:03
@alamb

alamb commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Thank you @avantgardnerio and @timsaucer

@alamb

alamb commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

run benchmark bounded_window

@alamb

alamb commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

run benchmark window_query_sql

@alamb

alamb commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

run benchmark h2o_small_window

@alamb

alamb commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

(just running some benchmarks to make sure this doesn't change performance noticably)

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5257786963-1548-jvh8r 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark bounded_window

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5257789157-1550-mwtd6 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark h2o_small_window

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5257788057-1549-cxjlj 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark window_query_sql

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark bounded_window
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                                       HEAD                                   brent_bwag-finalized-state-observer
-----                                                       ----                                   -----------------------------------
bounded_window_partitions/linear 100 partitions             1.00     83.8±0.09ms        ? ?/sec    1.00     83.9±0.05ms        ? ?/sec
bounded_window_partitions/linear 10000 partitions           1.00    457.1±1.67ms        ? ?/sec    1.00    456.5±5.82ms        ? ?/sec
bounded_window_partitions/linear multi 10000 partitions     1.00    709.8±2.96ms        ? ?/sec    1.02    724.1±7.03ms        ? ?/sec
bounded_window_partitions/linear rows 10000 partitions      1.00    409.9±4.19ms        ? ?/sec    1.04    427.0±5.99ms        ? ?/sec
bounded_window_partitions/linear sparse 32768 partitions    1.00    452.1±4.61ms        ? ?/sec    1.00    450.5±2.64ms        ? ?/sec
bounded_window_partitions/sorted 10000 partitions           1.00     70.7±0.21ms        ? ?/sec    1.00     70.9±0.17ms        ? ?/sec

Resource Usage

bounded_window — base (merge-base)

Metric Value
Wall time 475.1s
Peak memory 88.9 MiB
Avg memory 9.8 MiB
CPU user 84.6s
CPU sys 0.2s
Peak spill 0 B

bounded_window — branch

Metric Value
Wall time 470.1s
Peak memory 91.2 MiB
Avg memory 10.8 MiB
CPU user 90.0s
CPU sys 0.1s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark h2o_small_window
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

Comparing HEAD and brent_bwag-finalized-state-observer
--------------------
Benchmark h2o_window.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ brent_bwag-finalized-state-observer ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │  140.09 ms │                           139.61 ms │     no change │
│ QQuery 2  │ 1958.73 ms │                          1954.55 ms │     no change │
│ QQuery 3  │ 2519.09 ms │                          2825.21 ms │  1.12x slower │
│ QQuery 4  │  486.56 ms │                           468.42 ms │     no change │
│ QQuery 5  │ 1238.25 ms │                          1250.95 ms │     no change │
│ QQuery 6  │ 2849.53 ms │                          2866.78 ms │     no change │
│ QQuery 7  │ 2626.92 ms │                          2639.56 ms │     no change │
│ QQuery 8  │ 9535.86 ms │                          9547.38 ms │     no change │
│ QQuery 9  │  415.93 ms │                           407.97 ms │     no change │
│ QQuery 10 │  544.19 ms │                           538.88 ms │     no change │
│ QQuery 11 │  527.02 ms │                           519.35 ms │     no change │
│ QQuery 12 │ 1073.63 ms │                          1076.98 ms │     no change │
│ QQuery 13 │  308.01 ms │                           319.21 ms │     no change │
│ QQuery 14 │  407.38 ms │                           366.71 ms │ +1.11x faster │
│ QQuery 15 │  354.05 ms │                           342.45 ms │     no change │
│ QQuery 16 │  320.59 ms │                           328.13 ms │     no change │
│ QQuery 17 │  355.69 ms │                           322.25 ms │ +1.10x faster │
│ QQuery 18 │  413.38 ms │                           430.51 ms │     no change │
│ QQuery 19 │  365.71 ms │                           387.11 ms │  1.06x slower │
│ QQuery 20 │  385.90 ms │                           417.78 ms │  1.08x slower │
│ QQuery 21 │  332.06 ms │                           355.50 ms │  1.07x slower │
│ QQuery 22 │  380.59 ms │                           375.47 ms │     no change │
│ QQuery 23 │  357.97 ms │                           356.96 ms │     no change │
│ QQuery 24 │  436.54 ms │                           436.11 ms │     no change │
│ QQuery 25 │  377.74 ms │                           379.31 ms │     no change │
│ QQuery 26 │  384.04 ms │                           404.51 ms │  1.05x slower │
│ QQuery 27 │  365.12 ms │                           356.72 ms │     no change │
│ QQuery 28 │  387.05 ms │                           372.17 ms │     no change │
│ QQuery 29 │  342.05 ms │                           342.96 ms │     no change │
└───────────┴────────────┴─────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                  ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                  │ 30189.65ms │
│ Total Time (brent_bwag-finalized-state-observer)   │ 30529.51ms │
│ Average Time (HEAD)                                │  1041.02ms │
│ Average Time (brent_bwag-finalized-state-observer) │  1052.74ms │
│ Queries Faster                                     │          2 │
│ Queries Slower                                     │          5 │
│ Queries with No Change                             │         22 │
│ Queries with Failure                               │          0 │
└────────────────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and brent_bwag-finalized-state-observer
--------------------
Benchmark h2o_window.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃   brent_bwag-finalized-state-observer ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │      140.09 / 146.83 ±7.65 / 157.52 ms │     139.61 / 145.68 ±4.44 / 150.12 ms │     no change │
│ QQuery 2  │   1958.73 / 1958.94 ±0.24 / 1959.27 ms │  1954.55 / 1957.96 ±2.75 / 1961.28 ms │     no change │
│ QQuery 3  │ 2519.09 / 2662.55 ±128.21 / 2830.31 ms │ 2825.21 / 2858.15 ±29.98 / 2897.74 ms │  1.07x slower │
│ QQuery 4  │     486.56 / 510.58 ±17.37 / 527.07 ms │    468.42 / 497.35 ±30.67 / 539.80 ms │     no change │
│ QQuery 5  │   1238.25 / 1247.87 ±7.43 / 1256.35 ms │  1250.95 / 1253.26 ±2.31 / 1256.42 ms │     no change │
│ QQuery 6  │   2849.53 / 2857.10 ±6.51 / 2865.43 ms │ 2866.78 / 2876.32 ±12.81 / 2894.42 ms │     no change │
│ QQuery 7  │   2626.92 / 2634.73 ±7.87 / 2645.50 ms │ 2639.56 / 2656.07 ±11.88 / 2667.00 ms │     no change │
│ QQuery 8  │  9535.86 / 9563.66 ±19.68 / 9578.80 ms │  9547.38 / 9552.70 ±3.87 / 9556.48 ms │     no change │
│ QQuery 9  │     415.93 / 458.66 ±30.21 / 480.18 ms │    407.97 / 426.44 ±21.31 / 456.30 ms │ +1.08x faster │
│ QQuery 10 │     544.19 / 569.37 ±31.62 / 613.96 ms │     538.88 / 545.62 ±6.98 / 555.24 ms │     no change │
│ QQuery 11 │     527.02 / 547.59 ±16.98 / 568.60 ms │    519.35 / 536.47 ±15.29 / 556.47 ms │     no change │
│ QQuery 12 │   1073.63 / 1079.93 ±4.47 / 1083.46 ms │  1076.98 / 1086.36 ±9.68 / 1099.69 ms │     no change │
│ QQuery 13 │     308.01 / 339.37 ±35.71 / 389.32 ms │    319.21 / 339.67 ±23.38 / 372.39 ms │     no change │
│ QQuery 14 │      407.38 / 411.50 ±2.92 / 413.85 ms │    366.71 / 409.51 ±34.86 / 452.10 ms │     no change │
│ QQuery 15 │      354.05 / 364.14 ±9.44 / 376.75 ms │    342.45 / 360.91 ±16.66 / 382.82 ms │     no change │
│ QQuery 16 │     320.59 / 339.15 ±18.67 / 364.70 ms │     328.13 / 333.43 ±4.01 / 337.84 ms │     no change │
│ QQuery 17 │      355.69 / 362.10 ±8.14 / 373.59 ms │     322.25 / 330.06 ±5.95 / 336.66 ms │ +1.10x faster │
│ QQuery 18 │     413.38 / 433.03 ±14.10 / 445.78 ms │    430.51 / 442.70 ±13.59 / 461.66 ms │     no change │
│ QQuery 19 │      365.71 / 374.83 ±9.26 / 387.53 ms │     387.11 / 396.63 ±7.21 / 404.54 ms │  1.06x slower │
│ QQuery 20 │      385.90 / 388.98 ±3.06 / 393.16 ms │     417.78 / 425.90 ±5.74 / 429.99 ms │  1.09x slower │
│ QQuery 21 │     332.06 / 365.22 ±40.37 / 422.05 ms │    355.50 / 370.67 ±16.20 / 393.12 ms │     no change │
│ QQuery 22 │      380.59 / 385.21 ±4.55 / 391.40 ms │     375.47 / 382.48 ±5.41 / 388.65 ms │     no change │
│ QQuery 23 │     357.97 / 383.55 ±20.12 / 407.14 ms │    356.96 / 366.96 ±13.12 / 385.49 ms │     no change │
│ QQuery 24 │      436.54 / 448.69 ±8.93 / 457.74 ms │    436.11 / 453.22 ±16.53 / 475.56 ms │     no change │
│ QQuery 25 │      377.74 / 385.56 ±5.92 / 392.07 ms │     379.31 / 383.39 ±3.11 / 386.84 ms │     no change │
│ QQuery 26 │     384.04 / 438.01 ±43.99 / 491.79 ms │    404.51 / 446.59 ±50.41 / 517.47 ms │     no change │
│ QQuery 27 │     365.12 / 382.58 ±24.54 / 417.28 ms │    356.72 / 384.99 ±25.50 / 418.52 ms │     no change │
│ QQuery 28 │      387.05 / 391.35 ±5.60 / 399.26 ms │     372.17 / 380.36 ±8.19 / 391.54 ms │     no change │
│ QQuery 29 │     342.05 / 368.91 ±26.88 / 405.64 ms │    342.96 / 357.57 ±13.74 / 375.97 ms │     no change │
└───────────┴────────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                  ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                  │ 30799.98ms │
│ Total Time (brent_bwag-finalized-state-observer)   │ 30957.42ms │
│ Average Time (HEAD)                                │  1062.07ms │
│ Average Time (brent_bwag-finalized-state-observer) │  1067.50ms │
│ Queries Faster                                     │          2 │
│ Queries Slower                                     │          3 │
│ Queries with No Change                             │         24 │
│ Queries with Failure                               │          0 │
└────────────────────────────────────────────────────┴────────────┘

Resource Usage

h2o_small_window — base (merge-base)

Metric Value
Wall time 95.0s
Peak memory 4.1 GiB
Avg memory 972.2 MiB
CPU user 449.2s
CPU sys 17.1s
Peak spill 0 B

h2o_small_window — branch

Metric Value
Wall time 95.0s
Peak memory 4.1 GiB
Avg memory 1.0 GiB
CPU user 464.4s
CPU sys 16.6s
Peak spill 0 B

File an issue against this benchmark runner

@alamb

alamb commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

run benchmark h2o_small_window

@alamb

alamb commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

run benchmark bounded_window

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5258171918-1551-djbfl 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark h2o_small_window

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5258175342-1552-q6gl5 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark bounded_window

Results will be posted here when complete


File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark bounded_window
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                                       HEAD                                   brent_bwag-finalized-state-observer
-----                                                       ----                                   -----------------------------------
bounded_window_partitions/linear 100 partitions             1.00     84.0±0.72ms        ? ?/sec    1.00     83.9±0.07ms        ? ?/sec
bounded_window_partitions/linear 10000 partitions           1.00    452.0±1.90ms        ? ?/sec    1.02   459.1±15.99ms        ? ?/sec
bounded_window_partitions/linear multi 10000 partitions     1.00    703.7±5.74ms        ? ?/sec    1.00    704.5±4.25ms        ? ?/sec
bounded_window_partitions/linear rows 10000 partitions      1.01    411.6±3.05ms        ? ?/sec    1.00    406.5±1.39ms        ? ?/sec
bounded_window_partitions/linear sparse 32768 partitions    1.02    446.2±8.16ms        ? ?/sec    1.00    436.5±3.50ms        ? ?/sec
bounded_window_partitions/sorted 10000 partitions           1.00     70.6±0.26ms        ? ?/sec    1.01     71.0±0.25ms        ? ?/sec

Resource Usage

bounded_window — base (merge-base)

Metric Value
Wall time 220.1s
Peak memory 89.3 MiB
Avg memory 23.7 MiB
CPU user 87.3s
CPU sys 0.1s
Peak spill 0 B

bounded_window — branch

Metric Value
Wall time 220.0s
Peak memory 89.9 MiB
Avg memory 21.4 MiB
CPU user 87.1s
CPU sys 0.1s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark h2o_small_window
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

Comparing HEAD and brent_bwag-finalized-state-observer
--------------------
Benchmark h2o_window.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃       HEAD ┃ brent_bwag-finalized-state-observer ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │  140.60 ms │                           140.26 ms │     no change │
│ QQuery 2  │ 1971.38 ms │                          1959.88 ms │     no change │
│ QQuery 3  │ 2603.45 ms │                          2494.99 ms │     no change │
│ QQuery 4  │  455.12 ms │                           486.11 ms │  1.07x slower │
│ QQuery 5  │ 1222.26 ms │                          1272.60 ms │     no change │
│ QQuery 6  │ 2828.92 ms │                          2864.17 ms │     no change │
│ QQuery 7  │ 2685.73 ms │                          2659.05 ms │     no change │
│ QQuery 8  │ 9543.31 ms │                          9586.29 ms │     no change │
│ QQuery 9  │  406.62 ms │                           433.07 ms │  1.07x slower │
│ QQuery 10 │  559.17 ms │                           540.33 ms │     no change │
│ QQuery 11 │  526.52 ms │                           524.02 ms │     no change │
│ QQuery 12 │ 1058.94 ms │                          1079.48 ms │     no change │
│ QQuery 13 │  335.76 ms │                           296.53 ms │ +1.13x faster │
│ QQuery 14 │  407.03 ms │                           385.46 ms │ +1.06x faster │
│ QQuery 15 │  368.00 ms │                           345.69 ms │ +1.06x faster │
│ QQuery 16 │  318.85 ms │                           336.23 ms │  1.05x slower │
│ QQuery 17 │  313.77 ms │                           326.09 ms │     no change │
│ QQuery 18 │  404.66 ms │                           417.98 ms │     no change │
│ QQuery 19 │  385.73 ms │                           376.55 ms │     no change │
│ QQuery 20 │  408.55 ms │                           405.82 ms │     no change │
│ QQuery 21 │  346.45 ms │                           334.68 ms │     no change │
│ QQuery 22 │  369.20 ms │                           375.53 ms │     no change │
│ QQuery 23 │  357.87 ms │                           362.95 ms │     no change │
│ QQuery 24 │  450.54 ms │                           405.73 ms │ +1.11x faster │
│ QQuery 25 │  373.97 ms │                           368.18 ms │     no change │
│ QQuery 26 │  387.17 ms │                           417.37 ms │  1.08x slower │
│ QQuery 27 │  337.79 ms │                           365.38 ms │  1.08x slower │
│ QQuery 28 │  386.77 ms │                           369.34 ms │     no change │
│ QQuery 29 │  350.46 ms │                           346.96 ms │     no change │
└───────────┴────────────┴─────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                  ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                  │ 30304.59ms │
│ Total Time (brent_bwag-finalized-state-observer)   │ 30276.71ms │
│ Average Time (HEAD)                                │  1044.99ms │
│ Average Time (brent_bwag-finalized-state-observer) │  1044.02ms │
│ Queries Faster                                     │          4 │
│ Queries Slower                                     │          5 │
│ Queries with No Change                             │         20 │
│ Queries with Failure                               │          0 │
└────────────────────────────────────────────────────┴────────────┘

Distribution per query (min / mean ±stddev / max):

Comparing HEAD and brent_bwag-finalized-state-observer
--------------------
Benchmark h2o_window.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃    brent_bwag-finalized-state-observer ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │      140.60 / 142.87 ±2.08 / 145.63 ms │      140.26 / 143.76 ±4.38 / 149.93 ms │     no change │
│ QQuery 2  │   1971.38 / 1972.34 ±0.73 / 1973.14 ms │   1959.88 / 1961.85 ±2.51 / 1965.39 ms │     no change │
│ QQuery 3  │ 2603.45 / 2713.65 ±128.36 / 2893.68 ms │ 2494.99 / 2866.00 ±316.48 / 3268.32 ms │  1.06x slower │
│ QQuery 4  │     455.12 / 472.84 ±23.57 / 506.15 ms │     486.11 / 496.80 ±11.38 / 512.57 ms │  1.05x slower │
│ QQuery 5  │   1222.26 / 1233.80 ±8.32 / 1241.55 ms │   1272.60 / 1274.27 ±1.29 / 1275.73 ms │     no change │
│ QQuery 6  │   2828.92 / 2836.76 ±5.64 / 2841.92 ms │   2864.17 / 2866.84 ±2.22 / 2869.60 ms │     no change │
│ QQuery 7  │   2685.73 / 2690.67 ±3.51 / 2693.56 ms │   2659.05 / 2665.69 ±4.87 / 2670.61 ms │     no change │
│ QQuery 8  │  9543.31 / 9578.95 ±25.34 / 9599.95 ms │  9586.29 / 9596.14 ±11.79 / 9612.72 ms │     no change │
│ QQuery 9  │     406.62 / 434.46 ±20.86 / 456.84 ms │     433.07 / 450.31 ±12.69 / 463.25 ms │     no change │
│ QQuery 10 │     559.17 / 584.09 ±29.47 / 625.48 ms │     540.33 / 553.34 ±11.01 / 567.25 ms │ +1.06x faster │
│ QQuery 11 │     526.52 / 546.46 ±14.12 / 557.30 ms │      524.02 / 529.74 ±6.43 / 538.72 ms │     no change │
│ QQuery 12 │  1058.94 / 1077.17 ±14.75 / 1095.07 ms │  1079.48 / 1112.79 ±29.45 / 1151.09 ms │     no change │
│ QQuery 13 │     335.76 / 353.96 ±25.19 / 389.58 ms │     296.53 / 330.45 ±27.78 / 364.57 ms │ +1.07x faster │
│ QQuery 14 │     407.03 / 420.68 ±12.41 / 437.06 ms │      385.46 / 391.11 ±4.07 / 394.87 ms │ +1.08x faster │
│ QQuery 15 │     368.00 / 391.27 ±20.06 / 416.96 ms │      345.69 / 352.99 ±5.41 / 358.63 ms │ +1.11x faster │
│ QQuery 16 │      318.85 / 332.68 ±9.87 / 341.26 ms │      336.23 / 343.81 ±6.47 / 352.04 ms │     no change │
│ QQuery 17 │      313.77 / 314.45 ±0.65 / 315.33 ms │     326.09 / 344.71 ±19.05 / 370.87 ms │  1.10x slower │
│ QQuery 18 │     404.66 / 432.69 ±20.55 / 453.37 ms │     417.98 / 438.66 ±15.00 / 453.10 ms │     no change │
│ QQuery 19 │     385.73 / 401.73 ±16.35 / 424.19 ms │     376.55 / 393.80 ±14.94 / 413.00 ms │     no change │
│ QQuery 20 │      408.55 / 418.41 ±6.99 / 423.96 ms │      405.82 / 415.73 ±9.65 / 428.82 ms │     no change │
│ QQuery 21 │      346.45 / 351.05 ±4.25 / 356.69 ms │     334.68 / 344.94 ±11.26 / 360.61 ms │     no change │
│ QQuery 22 │      369.20 / 372.18 ±3.14 / 376.52 ms │     375.53 / 396.31 ±23.14 / 428.59 ms │  1.06x slower │
│ QQuery 23 │     357.87 / 373.98 ±19.38 / 401.23 ms │      362.95 / 366.08 ±3.26 / 370.57 ms │     no change │
│ QQuery 24 │      450.54 / 457.42 ±9.22 / 470.45 ms │     405.73 / 440.50 ±25.05 / 463.74 ms │     no change │
│ QQuery 25 │     373.97 / 393.10 ±23.51 / 426.22 ms │     368.18 / 385.20 ±12.99 / 399.69 ms │     no change │
│ QQuery 26 │     387.17 / 400.98 ±11.48 / 415.27 ms │     417.37 / 430.77 ±17.47 / 455.45 ms │  1.07x slower │
│ QQuery 27 │      337.79 / 343.57 ±6.37 / 352.44 ms │     365.38 / 381.99 ±16.45 / 404.40 ms │  1.11x slower │
│ QQuery 28 │     386.77 / 410.63 ±19.01 / 433.28 ms │      369.34 / 377.21 ±7.36 / 387.05 ms │ +1.09x faster │
│ QQuery 29 │      350.46 / 358.02 ±7.44 / 368.15 ms │      346.96 / 350.81 ±2.73 / 352.88 ms │     no change │
└───────────┴────────────────────────────────────────┴────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                  ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                  │ 30810.84ms │
│ Total Time (brent_bwag-finalized-state-observer)   │ 31002.60ms │
│ Average Time (HEAD)                                │  1062.44ms │
│ Average Time (brent_bwag-finalized-state-observer) │  1069.06ms │
│ Queries Faster                                     │          5 │
│ Queries Slower                                     │          6 │
│ Queries with No Change                             │         18 │
│ Queries with Failure                               │          0 │
└────────────────────────────────────────────────────┴────────────┘

Resource Usage

h2o_small_window — base (merge-base)

Metric Value
Wall time 95.0s
Peak memory 4.2 GiB
Avg memory 1.0 GiB
CPU user 453.3s
CPU sys 15.2s
Peak spill 0 B

h2o_small_window — branch

Metric Value
Wall time 95.0s
Peak memory 4.3 GiB
Avg memory 1.0 GiB
CPU user 464.6s
CPU sys 14.7s
Peak spill 0 B

File an issue against this benchmark runner

@adriangbot

Copy link
Copy Markdown

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

Comparing brent/bwag-finalized-state-observer (aa4d196) to 33ad1cc (merge-base) diff

Run configuration
run benchmark window_query_sql
CPU Details (lscpu)
Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected
Details

group                                                             HEAD                                   brent_bwag-finalized-state-observer
-----                                                             ----                                   -----------------------------------
window empty over, aggregate functions                            1.00     14.3±0.85ms        ? ?/sec    1.10     15.8±0.86ms        ? ?/sec
window empty over, built-in functions                             1.00    170.2±0.87ms        ? ?/sec    1.03    175.8±2.54ms        ? ?/sec
window order by, aggregate functions                              1.00    913.1±3.62ms        ? ?/sec    1.00    914.1±4.13ms        ? ?/sec
window order by, built-in functions                               1.00    848.8±3.41ms        ? ?/sec    1.00    845.4±2.66ms        ? ?/sec
window partition and order by, u64_narrow, aggregate functions    1.01    284.0±1.71ms        ? ?/sec    1.00    282.5±0.88ms        ? ?/sec
window partition and order by, u64_narrow, built-in functions     1.00    204.3±1.18ms        ? ?/sec    1.01    205.4±1.18ms        ? ?/sec
window partition and order by, u64_wide, aggregate functions      1.00  965.7±110.85ms        ? ?/sec    1.05  1016.2±116.52ms        ? ?/sec
window partition and order by, u64_wide, built-in functions       1.03  903.0±141.32ms        ? ?/sec    1.00  875.6±138.12ms        ? ?/sec
window partition by, u64_narrow, aggregate functions              1.00     11.2±0.08ms        ? ?/sec    1.00     11.2±0.09ms        ? ?/sec
window partition by, u64_narrow, built-in functions               1.00     41.8±0.38ms        ? ?/sec    1.00     41.8±0.53ms        ? ?/sec
window partition by, u64_wide, aggregate functions                1.00   588.8±90.93ms        ? ?/sec    1.00   589.6±89.49ms        ? ?/sec
window partition by, u64_wide, built-in functions                 1.00   550.8±60.95ms        ? ?/sec    1.05   577.6±58.83ms        ? ?/sec

Resource Usage

window_query_sql — base (merge-base)

Metric Value
Wall time 1550.3s
Peak memory 1.4 GiB
Avg memory 357.4 MiB
CPU user 4177.1s
CPU sys 34.8s
Peak spill 0 B

window_query_sql — branch

Metric Value
Wall time 1425.3s
Peak memory 1.4 GiB
Avg memory 400.1 MiB
CPU user 4321.5s
CPU sys 38.1s
Peak spill 0 B

File an issue against this benchmark runner

kosiew pushed a commit to kosiew/datafusion that referenced this pull request Aug 12, 2026
## Summary

Expose state of aggregate streams within BWAG so downstream prefix
scanning can take place.

## API

```rust
// physical-plan/src/windows/bounded_window_agg_exec.rs
pub type FinalizedWindowStateObserver = Arc<
    dyn Fn(usize, &PartitionKey, &[Option<Vec<ScalarValue>>]) -> Result<()>
        + Send + Sync,
>;

impl BoundedWindowAggExec {
    pub fn with_finalized_state_observer(mut self, obs: FinalizedWindowStateObserver) -> Self { … }
}

// physical-expr/src/window/window_expr.rs
impl WindowState {
    /// `Accumulator::state()` if this is an aggregate window function, `None` otherwise.
    pub fn aggregate_state(&mut self) -> Result<Option<Vec<ScalarValue>>> { … }
}
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change optimizer Optimizer rules physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants