fix(index): keep a commit alive when an index cannot be opened - #8441
fix(index): keep a commit alive when an index cannot be opened#8441wombatu-kun wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
❌ Gate recommendation: request changes.
Making index repair best-effort is the right boundary, but the deferred state is not yet durable: one path can stamp unverified legacy coverage as current and suppress future repair, while another can discard an unknown same-name segment on a later commit.
A viable revision should persist deferred coverage as unknown and preserve unknown segments through manifest filtering, so commits succeed without turning recoverable index metadata into stale or lost state.
| // The bitmap is optional metadata and recalculating it means | ||
| // opening the index. Failing here fails every commit the | ||
| // dataset takes, since migration runs on all of them, so keep | ||
| // the coverage as it stands and leave the repair to a build |
There was a problem hiding this comment.
A failed legacy repair is not actually left for a later compatible build. must_recalculate_fragment_bitmap uses the old manifest writer version, but a successful commit writes WriterVersion::default(). For the checked-in v0.8.14 fixture, this branch carries the present-but-corrupt bitmap into a current-writer manifest; after index storage recovers, the next commit skips repair and fragment 0 remains absent, which can omit rows from prefiltered index results. Persist None (the existing safe unknown-coverage state) or another durable retry signal when this repair fails.
Reproducer
I added a regression that copies v0.8.14/corrupt_index, renames _indices away, commits delete(false), restores _indices, reopens, commits again, and asserts the compatible writer restored fragment 0.
CARGO_TARGET_DIR=/home/agent/tmp/target-pr8441-implementation-2177bdd cargo test -p lance test_unopenable_old_index_still_retries_bitmap_migration -- --nocapture
Observed: exit 101 with the compatible writer must repair the old corrupt fragment bitmap.
There was a problem hiding this comment.
Fixed on 0b7802db3. The failure path now distinguishes the one-shot pre-0.8.15 repair from retryable missing/overlap triggers and persists fragment_bitmap: None when that legacy repair cannot open the index. I verified test_v0_8_14_invalid_index_fragment_bitmap_repair_is_not_lost passes, including the later compatible repair restoring fragment 0. Resolving this finding.
|
|
||
| // And an unrelated commit after it, since the missing bitmap is now what | ||
| // the manifest holds and migration retries on every commit. | ||
| dataset.delete("false").await.unwrap(); |
There was a problem hiding this comment.
This later-commit check misses same-name multi-segment indices. After the first commit such a group can contain the unavailable segment with fragment_bitmap: None and a readable segment with known non-empty coverage. delete(false) calls retain_relevant_indices before migration; that function currently classifies None as empty and retains only the non-empty same-name segment, permanently deleting the unavailable UUID instead of carrying it through. Preserve unknown segments until they can be repaired.
Reproducer
I added a unit regression with same-name segments [fragment_bitmap: None, fragment_bitmap: Some({2})], called retain_relevant_indices, and asserted that both remain.
CARGO_TARGET_DIR=/home/agent/tmp/target-pr8441-implementation-2177bdd cargo test -p lance test_retain_unknown_and_nonempty_segments_keeps_unknown_segment -- --nocapture
Observed: exit 101 with unknown coverage is not empty coverage; the retained length was 1 instead of 2.
There was a problem hiding this comment.
Fixed on 0b7802db3. retain_relevant_indices now partitions unknown coverage before same-name pruning and preserves every segment whose fragment_bitmap is None, instead of treating it as known-empty coverage. I verified test_retain_unknown_coverage_alongside_nonempty_sibling passes with both segments retained. Resolving this finding.
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The two durability gaps from the previous review are fixed: failed legacy repairs now persist unknown coverage for a compatible retry, and same-name retention preserves unknown segments. The write path remains best-effort only for optional index reconstruction while schema invariants stay fatal, with historical and multi-segment regressions covering recovery and preservation.
wjones127
left a comment
There was a problem hiding this comment.
This looks reasonable. Happy to merge once merge conflicts are addressed.
0b7802d to
bc07c4d
Compare
migrate_indicesrecalculates a missingfragment_bitmapby opening the index, and it runs on every commit. The open is propagated with?, so an index this build cannot open fails that commit and every commit after it: the dataset becomes unwritable rather than unreadable. An index can be unopenable for reasons unrelated to the write - files removed or never finished being written, a newer writer, a shallow clone whose base is out of reach - and the two neighbouring steps of the same function are already best-effort, so the open and the coverage calculation are now caught together and logged atwarn. The field lookup above them stays fatal: a manifest naming a field the schema does not have is a broken invariant, not a condition of the environment.Carrying the entry through is only safe where the recalculation will be asked for again. Of the three triggers, a missing bitmap and overlapping segment bitmaps are re-derived from the index metadata and ask again for free; the pre-0.8.15 trigger reads the previous manifest's writer version, which a successful commit replaces, so a bitmap left in place looks migrated from then on - which is how the corrupt bitmap in
v0.8.14/corrupt_index, missing fragment 0, would become permanent. Only that case drops the coverage to unknown. The other two must not:calculate_included_fragsis unimplemented for the modern index types, which cannot recover from a bitmap-less state.retain_relevant_indicesthen has to stop counting a missing bitmap as empty coverage, since it runs before migration in the same commit and was deleting the segment before the retry could reach it. Independent of #8427, which guards the same function against an unsupported index version; that guard does not fire here. Rebased onto #8481, which mademigrate_indicesreport the indices whose coverage it replaced so the caller can withdraw their MemWAL catch-up: dropping a bitmap to unknown is reported the same way, since coverage is derived before migration runs and could have credited a position off the bitmap this path discards.