Skip to content

fix(index): make vector append segment-set-native - #8047

Merged
Xuanwo merged 2 commits into
mainfrom
xuanwo/segment-native-vector-append
Jul 28, 2026
Merged

fix(index): make vector append segment-set-native#8047
Xuanwo merged 2 commits into
mainfrom
xuanwo/segment-native-vector-append

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Closes #6398.

A logical vector index can contain immutable physical segments trained with different IVF and quantizer models. The append path currently validates the entire segment set before selecting work and then tries to encode unindexed fragments as though every segment shared one model. This makes append fail on valid heterogeneous segment sets.

This makes append segment-set-native. Append builds one new physical segment over only unindexed fragments using the complete model of the deterministic manifest suffix segment, while preserving all existing segment metadata and fragment coverage. Explicit merge selects its requested suffix before compatibility validation, and steady-state rebalance continues to rewrite only one segment. Explicit retrain remains the operation that source-rebuilds and unifies models.

The compatibility guard remains fail-closed and now applies to legacy and V3 storage, including IVF, quantizer/codebook/rotation, metric, dimension, and index-type metadata. This intentionally leaves distributed merge planning in #7730 out of scope.

@github-actions github-actions Bot added the bug Something isn't working label Jul 28, 2026
@Xuanwo
Xuanwo marked this pull request as ready for review July 28, 2026 09:37
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.82301% with 114 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/index/vector.rs 33.33% 48 Missing ⚠️
rust/lance/src/index/append.rs 86.62% 26 Missing and 18 partials ⚠️
rust/lance/src/index/vector/ivf.rs 69.01% 20 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@BubbleCal BubbleCal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left three inline comments on compatibility, logical-index validation, and the ANN regression test.

Comment thread rust/lance/src/index/append.rs Outdated
Comment thread rust/lance/src/index/append.rs
Comment thread rust/lance/src/index/append.rs Outdated
@Xuanwo
Xuanwo merged commit 9509a44 into main Jul 28, 2026
34 checks passed
@Xuanwo
Xuanwo deleted the xuanwo/segment-native-vector-append branch July 28, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make append and optimize segment-set-native

2 participants