Does lance index supports distributed building?

[frankliee]

Lance format has the following index types, do them support distributed building?

scalar

1. btree
2. bitmap
3. lable_list

vector

1. IVF_PQ
2. IVF_HNSW_SQ

full text search

1. FTS

[eddyxu]

No yet.

There are community efforts in projects, such as https://github.com/lancedb/lance-ray to make it happen.

[frankliee]

No yet.

There are community efforts in projects, such as https://github.com/lancedb/lance-ray to make it happen.

Thanks, ray will provide distributed processing, but does lance provide the spilt strategy on how to divide the index building and index search into multiple independent tasks currently?

[yanghua]

No yet.
There are community efforts in projects, such as https://github.com/lancedb/lance-ray to make it happen.

Thanks, ray will provide distributed processing, but does lance provide the spilt strategy on how to divide the index building and index search into multiple independent tasks currently?

@frankliee You are right! There is definitely some related work that needs to be done in lance, e.g. https://github.com/lancedb/lance/pull/3452/files#diff-fd8f9387990c3d704a30b1d70fbae497cc2bb982923996fc32947c11053afb43R196 FYI.

[chenghao-guo]

@frankliee We're currently working with @yanghua on implementing distributed full-text search (FTS) index capability. This work is being tracked in our main feature issue:
🔗 lance-ray #12 - Distributed FTS Index

Implementation Timeline:

• 🚧 Current Status: Active development of distributed FTS infrastructure
• 🎯 Target Completion: Mid-August (~August 15)
• 🔄 Integration Plan: Will submit PR to lancedb/lance-ray once implementation is validated

Future Indexing Roadmap:
After FTS completion, we'll expand support to additional index types including vector index

We'll maintain updates on this thread as we hit key development milestones. Appreciate your patience while we build this foundational capability for the Lance ecosystem.

[frankliee]

@frankliee We're currently working with @yanghua on implementing distributed full-text search (FTS) index capability. This work is being tracked in our main feature issue: 🔗 lance-ray #12 - Distributed FTS Index

Implementation Timeline:

• 🚧 Current Status: Active development of distributed FTS infrastructure
• 🎯 Target Completion: Mid-August (~August 15)
• 🔄 Integration Plan: Will submit PR to lancedb/lance-ray once implementation is validated

Future Indexing Roadmap: After FTS completion, we'll expand support to additional index types including vector index

We'll maintain updates on this thread as we hit key development milestones. Appreciate your patience while we build this foundational capability for the Lance ecosystem.

We are planning to implement the JNI index building interface, which could be used in Spark.

[Xuanwo]

Closing this as stale due to substantial API and implementation drift. Lance now has fragment-level distributed index construction and commit APIs together with segmented index implementations, so the original general question no longer maps reliably to current behavior. This does not claim every distributed workflow has been verified. @frankliee, if a current workflow is still unsupported, please reopen this issue with the index type, execution model, and a minimal reproduction.