fix(ci): stabilize test_scan_slice against page-cache state - #8815
Merged
Xuanwo merged 1 commit intoAug 27, 2026
Merged
Conversation
test_scan_slice[tpch] measured a single unwarmed round, so its recorded value tracked whatever OS page-cache state the preceding benchmarks left behind rather than the scan itself. The apparent ~20x regression in issue lance-format#8289 began exactly when the merge_insert (lance-format#8052) and data-overlay (lance-format#7544) suites were added ahead of it in the session; the same slice reads ~2 ms warm and >100 ms cold on unchanged code, and the reader commits blamed in the issue measure identically on both sides of the window (same wall time, same IOPS, byte-identical written layouts). Warm up once and record five rounds so the published statistics track the code path deterministically. Fixes lance-format#8289
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
An unmeasured warmup followed by five recorded rounds turns the previously uncontrolled one-shot into an explicit warm slice-latency signal while retaining dataset-open cost. This is the right contract for stable steady-state tracking; cold-read performance should remain a separate, explicitly cache-cleared benchmark if needed.
Xuanwo
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
test_scan_slice[tpch]measures a single unwarmed round (rounds=1, iterations=1), so its recorded value tracks whatever OS page-cache state the preceding benchmarks left behind — not the scan itself. The same slice reads ~2 ms warm and >100 ms cold on unchanged code.The apparent ~20x regression in #8289 began exactly when the merge_insert (#8052) and data-overlay (#7544) suites were added ahead of
test_scanin the session (both merged 2026-07-31, hours after the last "good" measured commit). The reader commits the issue names were tested directly and are not at fault:elapsed_compute, identical IO (27 IOPS / 14,378 bytes for the full open+scan)Even within
test_scan.pyalone, the slice shot recorded right aftertest_full_scan's multi-GB materialization reads ~3x slower than steady state, on every version equally. #8290's cached-search step at the same date is likely the same mechanism.Fix
Warm up once and record five rounds so the published statistics track the code path deterministically. Adds ~5 extra 2-4 ms reads per run, next to the 10 s
test_full_scanbeside it.Validation
Run against TPC-H SF-10 locally: before — one polluted 6.8 ms shot; after — five stable warm rounds (min 3.59 ms, median 4.16 ms).
ruff checkandruff formatpass.Fixes #8289