Skip to content

fix(ci): stabilize test_scan_slice against page-cache state - #8815

Merged
Xuanwo merged 1 commit into
lance-format:mainfrom
RupertMaiti2005:fix/scan-slice-benchmark-stability
Aug 27, 2026
Merged

fix(ci): stabilize test_scan_slice against page-cache state#8815
Xuanwo merged 1 commit into
lance-format:mainfrom
RupertMaiti2005:fix/scan-slice-benchmark-stability

Conversation

@RupertMaiti2005

Copy link
Copy Markdown
Contributor

Problem

test_scan_slice[tpch] measures a single unwarmed round (rounds=1, iterations=1), so its recorded value tracks whatever OS page-cache state the preceding benchmarks left behind — not the scan itself. The same slice reads ~2 ms warm and >100 ms cold on unchanged code.

The apparent ~20x regression in #8289 began exactly when the merge_insert (#8052) and data-overlay (#7544) suites were added ahead of test_scan in the session (both merged 2026-07-31, hours after the last "good" measured commit). The reader commits the issue names were tested directly and are not at fault:

  • pylance 10.1.0-beta.2 (pre-window) vs 11.0.0-beta.1 (post-window) vs current main, same TPC-H dataset: identical wall time, identical elapsed_compute, identical IO (27 IOPS / 14,378 bytes for the full open+scan)
  • Datasets written by pre- and post-window versions are byte-identical (879,168,372 data bytes; 2,771 column-metadata bytes) and read identically in all writer×reader combinations
  • Reproduced at CI scale (SF-10, 59,986,052 rows): identical, ~2 ms on both sides of the window

Even within test_scan.py alone, the slice shot recorded right after test_full_scan's multi-GB materialization reads ~3x slower than steady state, on every version equally. #8290's cached-search step at the same date is likely the same mechanism.

Fix

Warm up once and record five rounds so the published statistics track the code path deterministically. Adds ~5 extra 2-4 ms reads per run, next to the 10 s test_full_scan beside it.

Validation

Run against TPC-H SF-10 locally: before — one polluted 6.8 ms shot; after — five stable warm rounds (min 3.59 ms, median 4.16 ms). ruff check and ruff format pass.

Fixes #8289

test_scan_slice[tpch] measured a single unwarmed round, so its recorded
value tracked whatever OS page-cache state the preceding benchmarks left
behind rather than the scan itself. The apparent ~20x regression in
issue lance-format#8289 began exactly when the merge_insert (lance-format#8052) and data-overlay
(lance-format#7544) suites were added ahead of it in the session; the same slice
reads ~2 ms warm and >100 ms cold on unchanged code, and the reader
commits blamed in the issue measure identically on both sides of the
window (same wall time, same IOPS, byte-identical written layouts).

Warm up once and record five rounds so the published statistics track
the code path deterministically.

Fixes lance-format#8289
@github-actions github-actions Bot added A-python Python bindings bug Something isn't working labels Aug 26, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

An unmeasured warmup followed by five recorded rounds turns the previously uncontrolled one-shot into an explicit warm slice-latency signal while retaining dataset-open cost. This is the right contract for stable steady-state tracking; cold-read performance should remain a separate, explicitly cache-cleared benchmark if needed.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Aug 27, 2026
@Xuanwo
Xuanwo merged commit 0ac7d22 into lance-format:main Aug 27, 2026
13 checks passed
@RupertMaiti2005
RupertMaiti2005 deleted the fix/scan-slice-benchmark-stability branch August 30, 2026 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-python Python bindings bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance regression: test_scan_slice[tpch] (~20x slower at d4da624)

2 participants