feat: support create vector index distributedly - #5117
Merged
Conversation
chenghao-guo
force-pushed
the
ivf_distribute_builder
branch
8 times, most recently
from
November 10, 2025 06:46
fd9c15e to
d571f55
Compare
chenghao-guo
force-pushed
the
ivf_distribute_builder
branch
9 times, most recently
from
November 13, 2025 07:00
e544b47 to
dc8d4bf
Compare
chenghao-guo
force-pushed
the
ivf_distribute_builder
branch
from
November 17, 2025 09:29
8ee88df to
e0dbea4
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
chenghao-guo
force-pushed
the
ivf_distribute_builder
branch
3 times, most recently
from
November 20, 2025 11:09
b591569 to
d627974
Compare
yanghua
force-pushed
the
ivf_distribute_builder
branch
3 times, most recently
from
November 25, 2025 02:27
6f08001 to
c37bbb8
Compare
chenghao-guo
force-pushed
the
ivf_distribute_builder
branch
2 times, most recently
from
November 28, 2025 10:25
e3d15f3 to
204d3f0
Compare
chenghao-guo
force-pushed
the
ivf_distribute_builder
branch
from
January 4, 2026 02:57
b12b3e4 to
af8249b
Compare
This was referenced Jan 4, 2026
Collaborator
|
@BubbleCal Thanks for reviewing! I have filed some tickets to track further work: #5621 and #5622. |
chenghao-guo
added a commit
to lance-format/lance-ray
that referenced
this pull request
Jan 16, 2026
close #66 Depends on this PR: lance-format/lance#5117 - New Public API: lance_ray.create_index, is introduced as the primary entry point for building distributed vector indices, currently support distributed IVF_FLAT, IVF_SQ, and IVF_PQ indices. The new create_index function orchestrates a multi-phase workflow: 1. Global Training: It uses existing lance.IndicesBuilder to train IVF centroids and, if applicable, PQ codebooks on a sample of the dataset. 2. Distributed Task Execution: Per-fragment index building tasks are distributed across a pool of Ray workers. Each worker receives the pre-trained models and processes a subset of the data fragments. 3. Metadata Finalization: After all fragment-level indices are built, the main process merges the metadata and commits the new index to the dataset manifest.
jackye1995
pushed a commit
to jackye1995/lance
that referenced
this pull request
Jan 21, 2026
close lance-format#4723 **Key Changes:** The distributed index creation leverages the existing IVF framework while adding coordination mechanisms for multi-node execution. The index merger component now handles distributed fragment consolidation and metadata synchronization. This work enables scalable vector index creation for large-scale datasets, significantly reducing index build time. - Implemented distributed IVF index building infrastructure for parallel index construction across multiple nodes - Enhanced the index merger component for distributed operations - For IVF_HNSW part, the HNSW graph is built locally within the shard as a sub-index of the partition; there is no cross-shard graph merging and no cross-shard edges. These are supported but distribution only happens in IVF. - CPU only, torch accelerator will not be supported and fall back to single node IVF index creation. **Current Status in this PR:** • FLAT/SQ: Should work now, it is under active testing phase, validating distributed performance and accuracy • PQ (Product Quantization): Currently depends on global training codebook, requiring centralized training before distributed deployment. • RQ (Residual Quantization): I didn't consider this when I design this PR. Not yet supported in distributed mode maybe, planned for future implementation Once I finish all the testing phase in my side on performance and recall accuracy, I will mark it ready to review. --------- Co-authored-by: yanghua <yanghua1127@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
close #4723
Key Changes:
The distributed index creation leverages the existing IVF framework while adding coordination mechanisms for multi-node execution. The index merger component now handles distributed fragment consolidation and metadata synchronization. This work enables scalable vector index creation for large-scale datasets, significantly reducing index build time.
Current Status in this PR:
• FLAT/SQ: Should work now, it is under active testing phase, validating distributed performance and accuracy
• PQ (Product Quantization): Currently depends on global training codebook, requiring centralized training before distributed deployment.
• RQ (Residual Quantization): I didn't consider this when I design this PR. Not yet supported in distributed mode maybe, planned for future implementation
Once I finish all the testing phase in my side on performance and recall accuracy, I will mark it ready to review.