test: end-to-end regression tests for view array spill compaction (replaces #21325)

[adriangb]

Which issue does this PR close?

• Replaces (and closes) fix: gc StringView/BinaryView arrays before spilling to prevent write amplification #21325.

Rationale for this change

#21325 set out to gc StringViewArray/BinaryViewArray batches in the hash aggregation and sort-merge join spill paths to prevent spill write amplification, but went stale with unaddressed review feedback. Re-deriving it from scratch on today's main: the runtime fix is already fully superseded — #21633 centralized the fix by running gc_view_arrays in InProgressSpillFile::append_batch, which every spill write path (sort, all aggregation streams, sort-merge join, nested-loop join, the repartition spill pool) funnels through, covering both view types plus view arrays nested inside container types. #21750 added unit tests for the compaction at the SpillManager level.

What remains from #21325 — and what the review feedback on it asked for ("Unless we make assertions on the sizes … we won't catch regressions") — is end-to-end coverage: nothing asserts that the operators with the heaviest view-array spill traffic actually keep routing their spills through the compaction. The existing e2e spill tests only assert spilled_bytes < disk_limit, which would not catch a 10x view-buffer amplification.

What changes are included in this PR?

Test-only: a new memory_limit::view_spill_compaction module with two end-to-end regression tests that run spilling queries over Utf8View + BinaryView data — one through the hash aggregation spill path, one through the sort spill path — and assert the total spilled_bytes stays proportional to the logical data size.

Compared to the benchmark in #21325, per the review feedback there:

• deterministic single runs with hard assertions instead of benchmark loops with printouts
• reads spilled_bytes from plan metrics instead of parsing EXPLAIN ANALYZE text
• bounds derived from the input data's logical size rather than magic constants

Measured on main (4.4 MB logical data):

Path	spill_count	spilled_bytes	ratio
sort	7	4,428,968	1.007x
aggregate	2	2,214,224	0.50x (spills part of its input)

With gc_view_arrays turned into a no-op (simulated regression), both tests fail: at these memory limits the un-compacted buffer sizes flow into the merge memory estimates and both queries die with ResourcesExhausted; at looser limits the spill files inflate instead (e.g. sort: 7,553,600 bytes, 1.7x at a 12 MB pool) and grow past the asserted bound as runs get sliced more finely. Either failure mode trips the tests.

Are these changes tested?

They are tests. Verified both directions: green on main, red with compaction disabled.

Are there any user-facing changes?

No.

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.36%. Comparing base (c429919) to head (332671c).
⚠️ Report is 56 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24547      +/-   ##
==========================================
+ Coverage   81.24%   81.36%   +0.12%     
==========================================
  Files        1113     1116       +3     
  Lines      392744   397509    +4765     
  Branches   392744   397509    +4765     
==========================================
+ Hits       319090   323451    +4361     
- Misses      54900    55116     +216     
- Partials    18754    18942     +188     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jayzhan211]

Any chance we could note why these numbers were picked? Not critical, but might help future readers. LGTM either way!

spilled_bytes > logical_bytes / 4,
spilled_bytes < logical_bytes * 3,

[adriangb]

Any chance we could note why these numbers were picked? Not critical, but might help future readers. LGTM either way!

spilled_bytes > logical_bytes / 4, spilled_bytes < logical_bytes * 3,

Thanks for asking. They were arbitrary, and measuring showed the 3x bound could never actually fire (with gc_view_arrays disabled the sort spill files only inflate to 1.73x at batch_size=1024). I retuned the sort test to batch_size=256 / 8 MB pool where compacted measures 1.0152x against 9.3384x regressed, tightened the bound to 2x, and recorded those measurements inline.