feat(index): share IVF partition scans across batch vector queries by sezruby · Pull Request #2 · sezruby/lance

sezruby · 2026-06-15T23:08:08Z

Implements #6822: extend batch vector queries to indexed/ANN search. Rebased on latest main.

Summary

Batch vector search (#6821, PR lance-format#6828) made indexed multi-query search work by looping the full single-query plan once per query vector (re-opening the index and rebuilding the prefilter each time) and unioning the results. This PR makes the indexed/ANN path share index-level state across the batch: it reads each probed IVF partition's storage once and scores every query that probes it, with the prefilter built once and shared.

Approach

VectorIndex trait (lance-index): defaulted supports_batch_partition_search() + search_partitions_batch(...) (default returns not_supported), so non-IVF indices are explicitly unsupported.
IVFIndex (ivf/v2.rs): batch search for flat-style sub-indices (IVF_FLAT/PQ/SQ/RQ). Invert per-query partition lists, load each distinct partition once, accumulate one top-k heap per query, reusing accumulate_prepared_partition_search / global_heap_to_batch.
ANNIvfBatchExec (io/exec/knn.rs): ranks each query against the centroids, runs the shared-scan batch search per delta, merges per-query top-k across deltas, emits {query_index, _distance, _rowid}. Prefilter wiring shared with the single-query node via build_dataset_prefilter.
Each query vector is normalized independently for cosine (normalize_batch_query_for_index).

Design notes (pre-empting review questions)

Why a new exec node instead of extending KNNVectorDistanceExec / the two ANN nodes? The two-node single-query pipeline streams one partition-list per delta through a per-query top-k. Sharing the scan requires inverting queries onto partitions and keeping one heap per query in a single pass — a different dataflow. The new node still reuses the underlying primitives (partition load, build_dataset_prefilter, and the index's per-partition accumulate), and the single-query nodes are untouched. Happy to fold it in differently if you'd prefer.
Why gate on the index-type string, not supports_batch_partition_search()? The gate is a planning-time decision and the single-query path likewise doesn't open the index there; derive_vector_index_type reads metadata with no I/O. The opened index re-checks the trait as a defensive invariant.
nprobes gate (correctness). The shared path searches exactly minimum_nprobes partitions/query. The single-query path is adaptive (early_pruning floor + late-search expansion), so it only matches when nprobes is fixed. The fast path is therefore gated to minimum_nprobes == maximum_nprobes; adaptive nprobes falls back to the per-query loop (verified: an unpinned batch diverged on every query before the gate; 0 divergence after). Open question for you: fixed-nprobes-first with batched early/late as a follow-up, or the full adaptive path in one PR?
Memory. Peak = the union of probed partitions held during scoring — the same buffering the existing single-query global-heap path uses (search_partitions), widened to the batch's partition union. Per-delta output is k-bounded, so cross-delta accumulation is O(deltas × k), not O(nprobes × rows).

Fallback matrix (no regression)

Case	Behavior
IVF_FLAT/PQ/SQ/RQ, fixed nprobes, fully indexed	shared-scan fast path
adaptive nprobes / `refine_factor` / IVF_HNSW_* / mixed indexed+unindexed	per-query indexed loop (exact)

Test plan

cargo test -p lance --lib test_batch_knn — 15 tests: plan shape, exact batch-vs-repeated-single equivalence (nprobes pinned), cosine regression, shared prefilter, multi-delta cross-delta merge, and explicit fallbacks for refine, adaptive nprobes, and IVF_HNSW (acceptance: "unsupported index types have explicit behavior and tests").
cargo test -p lance --lib dataset::scanner::test::test_knn (29) — no single-query regression (exercises the shared build_dataset_prefilter).
cargo fmt --all && cargo clippy -p lance -p lance-index --tests --benches -- -D warnings.
Python: pytest -k batch (L2 + cosine × three/single queries); ruff clean; pyright clean on changed lines.
Benchmark (benchmarks/test_search.py): batch vs repeated-single ANN; standalone timing (50k rows, dim 128, IVF_PQ 64 partitions, m=32, k=10, nprobes=10) → 2.48× speedup.

Closes lance-format#6822

## Summary - preserve exact nullable Float32 `_distance` and `_score` placeholders during initial expression discovery, including beside legal mixed-case stored columns - carry case-insensitive stored scoring-column matches into physical projection while deferring final scoring-expression resolution to the actual input schema - preserve stored scoring names and types while generated search fields remain Float32, with coverage for absent, colliding, mixed-case, and schema-dependent function expressions ## Root cause Projection expressions are initially parsed before search operators add their generated scoring columns. Schema-dependent functions resolve arguments during this first parse, so scoring identifiers need exact provisional bindings before references can be detected. Omitting a placeholder when a mixed-case stored field exists makes function parsing fail case-sensitively, while keeping the placeholder without carrying the stored field forward breaks ordinary case-insensitive projections. The discovery schema now retains exact generated-name placeholders without duplicating exact stored names. Matching mixed-case stored fields are separately included in the physical projection, and scoring expressions are reparsed against the actual physical input schema. Ordinary projections therefore preserve stored names and types, while search projections bind generated lowercase Float32 fields. ## Validation - `cargo fmt --all` - `cargo test -p lance-datafusion --locked` - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#4712  --------- Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

…at#8165)  ## Root cause The JSON index training pipeline projected and rebuilt every transformed batch with a hard-coded `_rowid` column. Inner trainers such as ZoneMap and FM-index request `_rowaddr`, so JSON extraction discarded the requested addresses before delegating to those trainers. ## Fix Preserve every scanner-provided row-location column through JSON extraction and type conversion. The existing JSON trainer test helper now follows the target training criteria, with regression cases verifying that ZoneMap and FM-index retain addresses from multiple fragments. ## Validation - `cargo fmt --all -- --check` - `cargo test -p lance-index scalar::json::tests` (10 passed) - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#7859 --------- Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

## Summary - bound attached and detached commit-conflict backoff by a shared 30-second retry budget - add CommitBuilder::with_retry_timeout for callers that need a different budget - reuse the write-retry timeout helper and clean up transaction files on timeout - add regression coverage with the outer commit timeout disabled ## Root cause commit_transaction derived SlotBackoff delays from the first attempt latency but awaited each delay directly. A slow first attempt could therefore produce a backoff sleep longer than the intended retry wall-clock budget. ## Validation - cargo fmt --all -- --check - cargo test -p lance commit_retry_timeout --lib - cargo test -p lance test_commit_timeout --lib - cargo test -p lance --doc with_retry_timeout - cargo clippy --all --tests --benches -- -D warnings Fixes lance-format#7882  Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

## Summary - preserve the canonical schema field path while initializing an index - use the full path for target validation and scalar or vector index creation - cover simple nested fields and nested fields that require path quoting ## Root cause initialize_index resolved each source field by ID but then used only the leaf field name. This discarded the parent path and canonical quoting before the target lookup and index initializer ran. ## Validation - cargo test -p lance --lib test_initialize_ - cargo fmt --all -- --check - cargo clippy --all --tests --benches -- -D warnings Fixes lance-format#8149  Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

…-format#8203) ## Summary - keep rewritten update fragments outside ZoneMap and BloomFilter coverage under stable row IDs - let partially indexed scans read those rewritten fragments directly - add regression coverage for ZoneMap range filters and BloomFilter point lookups ## Root cause The pure RewriteRows update path extended unchanged-column index coverage to each new fragment because stable row IDs preserve row-ID-domain index entries. ZoneMap and BloomFilter entries are instead physical row addresses, so their existing entries cannot describe rows moved into the new fragment. Claiming coverage suppressed the fallback scan and silently dropped matching rows. ## Validation - `cargo test -p lance dataset::write::update::tests` - `cargo fmt --all` - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#8202  Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

…ce-format#7994) The optimized local paths — `LocalWriter`, `LocalObjectReader`, the io_uring readers, and the local `copy` / recursive-delete shortcuts — go straight to the filesystem, so they never reach `MeteredObjectStore` and published no metrics at all. Writing a dataset to a local path produced zero object store metrics even though the IO tracker recorded it. `IOTracker` now carries the metrics `base` label of the store it belongs to (its store prefix, the same value `MeteredObjectStore` is given) and hands out an `IoMetricsGuard` for IO that bypasses the `object_store` layer. Each bypassing path now records requests, bytes, latency, errors and the in-flight gauge under the same labels, so local IO aggregates with cloud IO: | path | operation | | --- | --- | | `LocalWriter` (one `put` per file, from open to durable at its final path) | `put` | | `LocalObjectReader::get_range` / `get_all` / streamed chunks | `get` | | `LocalObjectReader::size` (the local equivalent of a HEAD) | `head` | | `UringReader` / `UringCurrentThreadReader` `get_range` / `get_all` | `get` | | local `ObjectStore::copy` | `copy` | | local `ObjectStore::remove_dir_all` (one request, like `delete_stream`) | `delete` | Two things worth a second opinion: - Metering a store's `inner` and labelling its `IOTracker` now happen together in `meter_store`, called from all three constructors — the registry, `from_uri_and_params`, and `ObjectStore::new`. Previously only the first two metered `inner`, so a caller-supplied store routed through `ObjectStore::new` (which is what `DatasetBuilder::build_object_store` does) would have published local reads and writes but nothing for `list` / `delete` / `rename`. A store now publishes for all of its IO or none of it; stores built by calling a provider's `new_store` directly (`ObjectStore::local` / `memory`) are the "none" case. This would still double-count if a caller passed in a store Lance had already metered. - `IoStats` is left alone. The local `copy`, recursive delete and size lookup were never counted there and still aren't, so existing IO assertions are unaffected; adding them would shift IO counts across the test suite. The io_uring readers have no coverage here — they need Linux plus a working ring, so the new tests exercise the non-uring local paths only. Fixes lance-format#7993 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…lance-format#8208)

…overflow (lance-format#7753) Closes: lance-format#7973 When create bloom filter index on a column with more than 1.07B rows, we might ran into arrow byte array offset overflow error. This pr fix it by writing index in batch.  ## Summary by CodeRabbit * **Bug Fixes** * Improved Bloom Filter index load/write to enforce a maximum serialized payload size, preventing offset overflow and oversized reads. * Added validation to reject bloom-filter records whose payload spans exceed the allowed limit. * Large bloom-filter indexes are now processed via bounded chunking for safer, more reliable reloads. * **Compatibility** * Preserves prior behavior when optional null-row information is not provided. * Chunked writes continue to persist and reload null-row bitmap state correctly when present.

## Summary - replace the fallible Lance-schema expect with a contextual InvalidInput error - add a regression test for an index column present only in the Arrow schema ## Root cause validate_index_configs checked the independent Arrow schema first and assumed that a matching Lance-schema field must exist. A caller-supplied divergent schema pair therefore reached expect and panicked at a public fallible API boundary. ## Validation - cargo test -p lance test_validate_index_configs --lib - cargo fmt --all -- --check - cargo clippy --all --tests --benches -- -D warnings Fixes lance-format#8194  Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

…or Update/Delete (lance-format#8210) Closes: lance-format#8209

…-format#8191) ## Why `Run Regression Benchmarks` fails during **Generate datasets** when building BTREE indexes on `merge_insert_narrow` string keys: ``` OSError: LanceError(IO): Resources exhausted: Additional allocation failed for ExternalSorterMerge[0] ... fair(pool_size: 150.0 MB) ``` Example: https://github.com/lance-format/lance/actions/runs/30819924543/job/91706969640 ## Approach Library-wide tuning of `sort_spill_reservation_bytes` is not a reliable fix for this class of failure (there is no safe general value for that knob). Index build here is one-shot datagen setup, not a measured benchmark path. Temporarily set `LANCE_MEM_POOL_SIZE=1GiB` only around the `create_scalar_index` loop in merge_insert datagen, then restore the previous env so measured paths keep the default pool. ## Notes - Reverts the earlier default-reservation change from this PR. - Scoped to merge_insert datagen only; other datagen paths were not failing on this error.

…at#8166) ## Summary - make an index type change replace every same-name segment only when the incoming segments cover all current fragments - reject partial cross-type commits before publishing metadata, while preserving same-type partial segment updates - treat legacy same-name segments without index details as an unconditional structural mismatch that requires full replacement - add regression coverage for full, partial, legacy-metadata, and unknown-type-URL collision cases ## Root cause commit_existing_index_segments originally excluded different-type metadata from removal. Removing that filter fixed full replacements, but the normal coverage logic still retained disjoint old-type segments during a partial type switch. The later type-change guard first skipped legacy segments whose index details were absent, then represented that absence with a string sentinel that could collide with an accepted incoming type URL. Both paths could publish mixed logical index segments. ## Validation - cargo fmt --all -- --check - cargo clippy --all --tests --benches -- -D warnings - cargo test -p lance --lib commit_existing_index_segments (14 passed) - cargo test -p lance --lib test_partial_type_change_with_legacy_missing_details_is_rejected -- --nocapture (1 passed, including the unknown-type-URL collision) Fixes lance-format#7842  --------- Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

## Summary - consume every trailing string literal when extracting paths from variadic `get_field` expressions - preserve chained `get_field` support and canonical nested field formatting - cover the production planner path for a two-level nested scalar index filter ## Root cause DataFusion optimization combines deeply nested field access into one variadic `get_field` call. The scalar-index path extractor required exactly two arguments, returned no path for depth two or greater, and left the filter as a full-scan refinement. ## Validation - `cargo fmt --all -- --check` - `cargo test -p lance-index` (925 passed, 2 ignored; 7 doctests passed) - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#8170 Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

## Summary - add a documented LanceException unchecked exception for operation failures - wrap synchronous and asynchronous scanner failures with the typed exception - cover the native async failure callback with a regression test ## Root cause LanceScanner and AsyncScanner converted scanner I/O and native-operation failures into bare RuntimeException instances. Callers therefore could not distinguish Lance operation failures from unrelated programming errors. ## Validation - ./mvnw -Dtest=AsyncScannerTest#testNativeScanFailureUsesLanceException test (passes; JNI Rust unit tests also pass) - ./mvnw spotless:check (passes) - cargo fmt --manifest-path ./lance-jni/Cargo.toml --all (passes) - ./mvnw -Djava.io.tmpdir=/home/repo/java/target/test-tmp test (397 passed, 26 skipped; one existing forked-JVM classloader test fails because its child JVM extracts JNI libraries to the non-executable /tmp mount) Fixes lance-format#7611  Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

## Summary - stable-deduplicate MemWAL point-lookup IN-list literals before planning lookups - preserve first-seen key order so duplicate literals cannot consume a LIMIT slot - cover exact-type fast lookup, coercible per-key fallback, NULL literals, and limit interaction ## Root cause The point-lookup route forwarded every IN-list literal to lookup_many, which resolves and emits once per supplied key. Duplicate literals therefore produced duplicate rows instead of the set-like predicate semantics of the general scan path. ## Validation - cargo fmt --all - cargo test -p lance point_lookup_filter_routes_to_fast_path -- --nocapture - cargo test -p lance dataset::mem_wal::scanner::builder::tests - cargo clippy --all --tests --benches -- -D warnings Fixes lance-format#8195  --------- Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com>

## Summary - replace `rust-stemmers` entirely with exact-pinned `frostem` for every supported stemming language - enable only the 18 Snowball algorithms exposed by the existing Lance `Language` API - add a regression covering the multibyte Greek suffix that previously panicked - leave inverted-index protobuf details and capability versions unchanged ## Root cause The unmaintained `rust-stemmers` 1.2.0 Greek implementation can retain stale UTF-8 byte offsets after shortening a word, then panic while slicing the shortened string. `frostem` is generated from current upstream Snowball and handles the affected Greek input without invalid byte slicing. ## Validation - cargo test -p lance-tokenizer - cargo fmt --all - cargo clippy --all --tests --benches -- -D warnings - cargo check --locked --manifest-path python/Cargo.toml - cargo check --locked --manifest-path java/lance-jni/Cargo.toml - cargo metadata --format-version 1 --locked confirms `rust-stemmers` is absent - git diff --check Fixes lance-format#5235  --------- Co-authored-by: Lance Gatekeeper <lance-gatekeeper[bot]@users.noreply.github.com> Co-authored-by: Xuanwo <github@xuanwo.io>

) Display writes `"RQ"` for `QuantizationType::Rabit` but `FromStr` only accepted `"RABIT"`, so the two never round-tripped. `IvfIndexState::serialize` stores the quantization type with `to_string()`; `deserialize` reads it back with `parse::<QuantizationType>()`. For an IVF_RQ index that parse always failed, and since `CacheCodec::deserialize` turns a body error into `CacheDecode::Miss`, the failure was silent — the index just never reused its serialized state. `Display` has to keep emitting `"RQ"`: `index_type_string` builds `IVF_{quantization_type}` and `IndexType::try_from` expects `"IVF_RQ"`. So the fix belongs on the `FromStr` side. `"RABIT"` stays accepted for headers already on disk. The existing round-trip coverage missed this because `test_ivf_index_state_roundtrip` hardcodes `QuantizationType::Flat` and the `test_prewarm_and_query_with_serializing_backend` cases are PQ and HNSW_SQ only. The new `#[rstest]` covers every variant. ## Testing - `cargo test -p lance-index --lib vector::quantizer` — 6 pass; `case_5_rabit` fails without the one-line fix - `cargo test -p lance-index --lib vector::` — 331 pass - `cargo fmt --all -- --check`, `cargo clippy -p lance-index --tests --benches -- -D warnings` Co-authored-by: Claude <noreply@anthropic.com>

## Summary - Ignore a typed `NotFound` only when a manifest listed by cleanup is older than the cleanup snapshot and has been removed concurrently. - Keep failures for the current snapshot manifest, newer manifests, and all other I/O errors fatal. - Add regression coverage for a manifest deleted between listing and reading. ## Root cause V2 manifest filenames are intentionally encoded as `u64::MAX - version`, so `18446744073709548523.manifest` represents logical version 3092 rather than an underflowed version. Two concurrent cleanup calls can race after one lists that old manifest: the other removes it before the first reads it, and the first propagated the resulting `NotFound`. ## Validation - `cargo test -p lance cleanup_ignores_old_manifest_removed_after_listing` - `cargo test -p lance dataset::cleanup::tests` - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#8212 Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>

## Summary Full-text search currently treats each dataset row as a document. This change adds an explicit `DocumentGranularity` while preserving row-level behavior as the default. - `Row`: each dataset row is one document. - `ListElement`: each element of the deepest `List` on the schema path is one document. - Field paths remain ordinary public paths such as `groups.docs.content`. - The source field, deepest list boundary, traversal, and coordinate rank are derived from the final stable field ID and the current schema. - Row and list-element indexes can coexist on the same field and are routed by `(final_field_id, document_granularity)`. - List-element results include `_doc_index`, containing root-to-leaf physical list ordinals. The same document semantics are used by indexed search, flat search, mixed-fragment search, MemWAL, append/merge/optimize, and the Rust, Python, and Java APIs. Existing metadata without `document_granularity` naturally remains row-granular because `ROW = 0`. ## Python example ```python from lance.query import DocumentGranularity, MatchQuery dataset.create_scalar_index( "groups.docs.content", "INVERTED", document_granularity=DocumentGranularity.LIST_ELEMENT, ) query = MatchQuery( "lance", "groups.docs.content", document_granularity=DocumentGranularity.LIST_ELEMENT, ) results = dataset.to_table(full_text_query=query) # Root-to-leaf list ordinals for every matched element document. print(results["_doc_index"].to_pylist()) ```  ## Summary by CodeRabbit - **New Features** - Full-text search (FTS) now supports **row-level** and **list-element** document granularity. - Added a `DocumentGranularity` option that can be set for **inverted/FTS index configuration** and for **match/phrase queries**. - For list-element granularity, FTS results can include per-hit element coordinates via **`_doc_index`**, preserving document boundaries. - Phrase queries support list-element documents (including nested list fields) with appropriate granularity behavior. - **Bug Fixes** - Improved granularity-aware index selection and validation, with defaults preserving prior row-level behavior.

) The safe From<&[T]> implementations for fixed-width SIMD vectors unconditionally loaded 4, 8, or 16 lanes from the slice pointer. A caller could pass a shorter slice through safe Rust and trigger an out-of-bounds read. This validates the minimum lane count before every raw load while preserving the existing conversion API and support for longer slices. Regressions cover every affected scalar type in debug and optimized builds.

Several safe distance APIs only used debug assertions before runtime-dispatched SIMD or C kernels that iterate using the first slice length. In optimized builds, a shorter second slice could therefore be read out of bounds. Scalar fallbacks also silently truncated through zip, producing backend-dependent behavior. This establishes one checked contract across generic dot/L2, direct trait calls, f32 dispatch, fp16/bf16/f64/u8 implementations, scalar helpers, and batch layouts. Invalid lengths now fail before any backend selection or raw load, including optimized builds.

`h2` 0.4.15 is flagged by cargo-deny as [RUSTSEC-2026-0258](https://rustsec.org/advisories/RUSTSEC-2026-0258): it accepts and queues empty DATA frames without limit. If streams are not actively drained, that can lead to unbounded memory use or a panic if the length overflows. This bump locks the transitive `h2` crate to the patched 0.4.16 release across `Cargo.lock`, `python/Cargo.lock`, and `java/lance-jni/Cargo.lock`.

The process-wide background executor was exposed as a mutable static reference even though its API only requires shared access. Repeated callers could therefore obtain aliased `&mut` references to the same executor, violating Rust reference rules. Return a shared reference while preserving the singleton and fork-reinitialization behavior.

Public data blocks can be constructed directly, so a safe caller could previously pass `validate = false` with a malformed layout and reach Arrow `build_unchecked`. Keep the existing argument for source compatibility but make Arrow layout validation mandatory across fixed-width, nullable, list, struct, and dictionary conversions.

BQ distance calculation grew scratch vectors with `set_len` before their elements were initialized, then exposed those elements through safe mutable slices and iterators. Initialize each output range with `resize` before SIMD and scalar writers reuse it, preserving scratch allocation reuse without creating references to uninitialized values.

Bitpacking pack and unpack paths extended vectors with `set_len` before their elements were initialized, then passed safe mutable slices over those elements to the kernels. Initialize each output range before packing or unpacking, while retaining the existing buffer sizing and tail layout decisions.

`RowAddrTreeMap` iterators were declared unsafe only because they panic when a `Full` fragment has no known size. Panicking is not an unsafe operation and callers had no memory-safety invariant to uphold. Expose ordinary safe iterators, document the panic contract, and remove the unnecessary unsafe blocks from consumers.

`BytepackedIntegerEncoder::append` was marked unsafe even though overflow only caused silent integer truncation, not memory unsafety. Make the API safe and fallible, convert to the selected storage width with checked conversions, and propagate failures from repetition-index serialization while preserving the disabled zero-width encoder behavior.

…8591) A MemWAL HNSW writer publishes the committed batch count before the committed vector length. Snapshots previously loaded those atomics in the opposite order, so a snapshot racing a second append could retain the first batch's contiguous pointer while exposing the new two-batch length. Accessing a vector from the second batch would then construct a slice beyond the first Arrow allocation. This makes snapshot acquisition load the visible length first and the batch count second. Observing a new length now synchronizes with the preceding batch publication; observing an old length remains a valid prefix. It also enforces the safe `VectorSource` bounds contract in release builds and adds a barrier-controlled concurrent-commit regression.

BatchStore is Send + Sync and exposes safe append methods taking &self, but its slot initialization relied on an architectural single-writer convention. Concurrent safe callers could select the same uninitialized slot and create a data race through UnsafeCell. This adds an internal RAII writer guard so accidental concurrent appends are serialized while readers remain lock-free. The normal WriteBatchHandler path stays uncontended, and the release publication protocol for readers is unchanged.

…format#8521) Adds a two-commit migration path to enable stable row IDs on an existing table without requiring a full rewrite. - Commit 1 (Merge): assigns a RowIdSequence to every fragment that lacks one, using a manual retry loop so each conflict re-reads the latest fragment list and recomputes IDs from scratch. - Commit 2 (UpdateConfig): validates that no concurrent write snuck in a fragment without row IDs between the two commits, then activates FLAG_STABLE_ROW_IDS and writes the correct next_row_id watermark. Supporting changes: - `apply_feature_flags`: removes auto-detection of FLAG_STABLE_ROW_IDS from fragment content; the flag is now carried across the word-reset (like FLAG_MEM_WAL_INDEX_CATCHUP) and set only when explicitly requested or previously present. - `ManifestWriteConfig`: adds `migration_next_row_id` to thread the watermark through to `build_manifest_with_read_version`. - `CommitBuilder`: adds `with_stable_row_id_migration_activation` to force `use_stable_row_ids = true` and bypass the "cannot enable on existing dataset" guard for the migration activation commit. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

## Summary - use the fragment-reuse metadata already loaded before entering the cache loader - prevent the loader from recursively probing its own single-flight cache key - add a zero-capacity Moka cache regression test for the eviction fallback ## Root cause `open_frag_reuse_index` loaded the matching metadata, then its cache-miss loader called `load_index` for the same UUID. `load_index` delegates to `load_indices`, which probes the same fragment-reuse cache key. Under eviction pressure, the single-flight leader therefore waited on its own in-flight load indefinitely. ## Validation - `cargo test -p lance test_open_frag_reuse_index_with_zero_capacity_cache -- --nocapture` - `cargo test -p lance test_load_indices -- --nocapture` - `cargo fmt --all` - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#8619  Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>

`FileFragment::write_column` accepts a schema and record-batch stream that can contain multiple columns, so the singular name misrepresents the API contract. Rename it to `write_columns` and update its Rust callers and tests. The API was introduced by lance-format#8313 and has not appeared in a release tag, so this intentionally does not retain a deprecated alias.

…tion (lance-format#8618) `RepDefUnraveler::decimate` steps through the definition levels with `read_idx += dimension` and copies with `get_unchecked_mut`. A dimension of 0 never advances the read index, so the loop runs forever and writes past the end of the buffer (SIGSEGV in release; the `get_unchecked_mut` precondition check aborts in debug). A zero dimension can only come from a malformed schema. lance-format#7247 added guards on the write path (`Schema::validate`) and in the field-scheduler factories, but nothing protected the unsafe loop itself, and the decoder tree (`StructuralStructDecoder::field_to_decoder`) can be built without going through those factories. Reject dimension 0 in `decimate` before the unsafe loop, document the invariant the loop relies on, and run the existing dimension guard when building a fixed-size-list decoder. Co-authored-by: Claude <noreply@anthropic.com>

…e-format#8053) `lance/src/dataset/transaction.rs` is 6767 lines and is the next thing we want to move down into `lance-table`. Its production code turns out to have no dependency on `Dataset`, `Session`, DataFusion, or async I/O — only six couplings to code above `lance-table`. This PR clears five of them so the move itself can be a plain file rename. Each commit moves one self-contained piece down to the layer that already owns the types it touches, and leaves a re-export behind so no caller changes: - **`is_system_index`** compared an `IndexMetadata` name against two constants that already live in `lance-table`'s `system_index` module. It now sits next to them; `lance-index` re-exports it. - **The key existence filter** (`KeyExistenceFilter` and friends, previously `merge_insert/inserted_rows.rs`) depends only on arrow, `lance-core`'s bloom filter, and the transaction protobuf. It is serialized into that protobuf, so it moves to `lance-table`. - **Overlay staleness checks** decide which rows an overlay makes stale with respect to an index, reading only coverage bitmaps, the overlay `committed_version`, and indexed field ids. `lance::dataset::overlay` keeps the read-resolution half. - **MemWAL index metadata helpers** read and write the MemWAL index's `IndexMetadata` entry. Every type they touch was already in `lance-table`, so they join the data structures they serialize. - **`ManifestBuildConfig`** is new. `build_manifest` took `ManifestWriteConfig`, whose `timestamp` field is the `lance` crate's mockable `SystemTime` — and that mock is `cfg(test)` of the `lance` crate, so resolving the timestamp inside a lower crate would silently un-mock it. The new config carries the timestamp already resolved to nanoseconds, and `ManifestWriteConfig` converts into it at the call sites, keeping the clock mockable. The sixth coupling, `ManifestWriteConfig` itself, deliberately stays in `lance` for that reason. ## Not included The move of `transaction.rs` into `lance-table`, and its split into a module tree, come as two follow-up PRs stacked on this one. Splitting them keeps the cross-crate move reviewable as a detected rename rather than a 6767-line add/delete pair. `io/commit/conflict_resolver.rs` is the other half of the transaction story and a natural later target, but it depends on `Dataset` and `DatasetIndexExt`, so it stays put. ## Testing The one behavioral question here is whether the mock clock still works, since that is what `ManifestBuildConfig` exists to protect. Verified locally: the `MockClock`-based suites (`dataset::cleanup`, `dataset::delta`, `dataset::tests::dataset_versioning`) pass, 67 tests. The `to_build_config()` conversion is called inside the two commit retry loops rather than hoisted above them, so each retry still resolves its own timestamp as before. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ce-format#8629) `main` does not compile, which fails the Rust, Python, and Java workflows on main and on every open PR. Two PRs that were each green on their own collided semantically. lance-format#8521 added a `migration_next_row_id` option to `ManifestWriteConfig` and read it while building the manifest. lance-format#8053 then moved that config down into `lance-table` as the new `ManifestBuildConfig`; because it branched before lance-format#8521, the new struct had no such field. Git merged both cleanly, so the break only appeared once the second one landed: ``` error[E0609]: no field `migration_next_row_id` on type `&lance_table::format::ManifestBuildConfig` --> rust/lance/src/dataset/transaction.rs:2230:23 ``` This PR adds the missing field to `ManifestBuildConfig` and passes it through `ManifestWriteConfig::to_build_config`. The code that reads it is already correct and is unchanged. No new test: the five `migrate_to_stable_row_ids` tests added by lance-format#8521 already cover this behavior, and they pass again now that the crate compiles. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…rd partitions_searched Three follow-ups on the shared-scan batch IVF path: - Gate eligibility on whether the *selected* index_segments cover every requested fragment, not whether the whole logical index does. A subset selected via with_index_segments could otherwise let the batch node search only the selected segments and silently drop a fragment covered solely by an unselected segment. Extracted the coverage check into fragments_missing_from_index_segments, shared by the gate and knn_combined so eligibility and fallback stay in lockstep. - Run the per-query centroid ranking on the dedicated CPU runtime (find_partitions_batch_on_cpu) instead of the async worker: the ranking is pure CPU and batch width multiplies it, so a wide batch over a large centroid set could monopolize a Tokio worker. - Record partitions_searched on ANNIvfBatchExec. It built the metric but never incremented it, so EXPLAIN ANALYZE reported 0 for every batch query. Report the distinct partitions read -- the shared I/O this node exists to save -- mirroring the single-query ANNIvfSubIndexExec. Tests: partial-segment fallback, CPU-runtime ranking, and a partitions_searched=2 assertion (distinct union, not the per-query sum of 4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

) Stacked on lance-format#8053. Review that one first; this PR targets its branch. Building a manifest from a transaction reads and writes only table metadata. Now that lance-format#8053 has moved the helpers it depended on, `transaction.rs` has no dependency on `Dataset`, `Session`, DataFusion, or async I/O — `restore_old_manifest` is the only thing that touches storage at all, and it uses only `ObjectStore` and `CommitHandler`. So the file moves to `lance-table`. The file itself is unchanged apart from import paths. `lance::dataset::transaction` becomes a re-export, so nothing changes at any call site — not in `lance`, not in the Python bindings, not in the Java bindings. Git reports the move as a **97% rename**, so the review surface is the import block rather than 6500 added and 6500 deleted lines. Keeping it that way is why the file arrives here whole and gets split into a module tree in a separate follow-up, and why the shim is an inline `pub mod transaction` in `dataset.rs` rather than a file at the old path — a file there would have left the addition with nothing to pair against, and the rename would not have been detected. Beyond import paths, the moved file changes in three ways, all visible in the rename diff: - `build_manifest`, `restore_old_manifest`, `modifies_same_metadata` and `upsert_key_conflict` widen from `pub(crate)` to `pub`, because their callers in `io/commit.rs` and `io/commit/conflict_resolver.rs` are now in another crate. `lance`'s own public API is unchanged. - The test module gains a `default_build_config()` helper, since `ManifestWriteConfig` stays in `lance` and its `Default` is what the 22 `build_manifest` test call sites used. - One comment that named `ManifestWriteConfig::default()` now describes the default config without naming a type from a higher crate. The one test that needed `Dataset` moves to `dataset_transactions.rs` in the first commit, so the second commit is the rename alone. Test counts confirm nothing was dropped: 58 tests before, 57 in `lance-table` after, plus that one. ## Not included The split into a module tree — `operation.rs`, `conflicts.rs`, `manifest_build.rs`, `proto.rs` and friends — is the next PR in the stack. This PR leaves a single 6488-line file in `lance-table`. Collapsing the new `lance_table::transaction::Transaction` with the existing `lance_table::format::Transaction` (a thin `pb::Transaction` wrapper whose doc comment says it exists so that "lance-table does not depend on higher layers" — the exact inversion this removes) would change the public `CommitHandler` trait signature, so it is left for later. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…e-format#8625) ## What `WriteStats` tracks flush counts and cumulative time, but a running total cannot be resampled into a distribution — the individual observations are gone by the time anything polls it. An embedder can compute an average and nothing else, which is the wrong shape for latency: a flush pipeline is judged on its tail, not its mean. This observes each flush individually through the `metrics` facade that `lance-io` already uses for object store operations. Observations route to whatever `Recorder` the embedding process installed, so this crate takes no position on the exporter, and the emit sites compile away with the feature off. ## Shape `lance_mem_wal_flush_duration_seconds{kind="wal"|"memtable"}` — one family with a label rather than two, because the WAL buffer flush and the memtable flush are stages of the same write pipeline and get read together. They differ by orders of magnitude, hence bucket bounds spanning a single object-store round trip through a multi-second dataset write. Counts and byte totals stay on `WriteStats`: cumulative values lose nothing to sampling, so there is no reason to route them through a recorder. ## Notes - New `dataset/mem_wal/metrics.rs`, mirroring `lance_io::object_store::metrics` (name constants, bucket bounds, a `describe_metrics` an exporter calls after installing its recorder). - Two emit sites, in `WriteStats::record_wal_flush` and `record_memtable_flush`, where the individual duration is already in hand. - `metrics` becomes an optional dependency of the `lance` crate; the existing `metrics` feature now enables it alongside `lance-io/metrics`. - Compiles with the feature on and off; `mem_wal` tests pass both ways. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

…t#8056) Stacked on lance-format#8054, which is stacked on lance-format#8053. Review those first; this PR targets lance-format#8054's branch. `lance_table::transaction` arrived as a single 6488-line file. This splits it into nine modules along the lines the code was already divided by, leaving `transaction.rs` as declarations, re-exports, and a map of where each concern lives: | module | lines | what it answers | | --- | --- | --- | | `builder` | 90 | what a transaction is: an operation plus the version it was based on | | `operation` | 304 | the vocabulary of changes an operation can describe | | `update_map` | 128 | incremental edits to the manifest's string maps | | `validate` | 357 | pre-commit checks against the manifest being replaced | | `manifest_build` | 1676 | applying an operation to produce the next manifest | | `index_maintenance` | 1027 | how that narrows or drops index metadata | | `row_version` | 1139 | how it assigns row ids and per-row version metadata | | `conflicts` | 1027 | whether two operations collide, for the commit retry path | | `proto` | 816 | the persisted protobuf encoding of all of the above | Each of the nine commits extracts one module, so the "was any logic altered?" question is answerable a module at a time rather than across a 4000-line redistribution. Test counts hold at 57 throughout, and each commit compiles and passes on its own. The 57 tests move with the code they cover. Six fixtures used by more than one module's tests live in a `test_support` module rather than being duplicated. Nothing outside `lance-table` sees a change: the re-export list in `transaction.rs` is the same set of names the module exported before. Items used across submodules are `pub(super)` rather than `pub(crate)`, since the submodules are private — clippy's `pub(crate)`-inside-a-private-module lint is what settles that. One deliberate non-change: `PartialEq for Operation` and `PartialEq for RewriteGroup` each define their own local `compare_vec`. That duplication was there before and is left alone to keep every commit a pure move. ## Not included `manifest_build` stays the outlier at 1676 lines, of which `build_manifest` is about 890 and its tests about 700. The original plan was to break its 15-arm match into per-operation appliers in a `manifest_build/` subdirectory, and I stopped short of it deliberately: unlike everything else here, that is not a move. Each arm mutates four or five pieces of shared state (`final_fragments`, `final_indices`, `next_row_id`, `fragment_id`), so extracting them means threading that state through `&mut` parameters, and a free function taking five `&mut` arguments is not obviously easier to read than the match arm it replaced. Worth doing as its own PR if we want it, where the signatures can be discussed on their merits rather than riding along with a mechanical split. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

## Summary - enable SQL bitwise shift parsing in the Lance dialect - translate left and right shift operators into DataFusion expressions - cover UInt64 parsing and evaluation with regression tests ## Root cause The custom Lance SQL dialect wrapped GenericDialect but inherited the Dialect trait default that disables bitwise shift operators. Once parsing is enabled, the Lance planner also needs explicit mappings for the resulting sqlparser shift operators. ## Validation - `cargo test -p lance-datafusion` - `cargo fmt --all -- --check` - `cargo clippy --all --tests --benches -- -D warnings` Fixes lance-format#3524  Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>

## Summary - validate buffered Arrow arrays before either primitive encoding pipeline starts background work - reject malformed variable-width offsets with a field-specific `InvalidInput` error instead of allowing a buffer-slice panic - convert element offsets to byte offsets when slicing 32-bit and 64-bit Arrow offset buffers - cover malformed string offsets and valid nonzero `ArrayData` offsets with regression tests ## Root cause The writer trusted Arrow variable-width offset buffers until encoding ran in a spawned task. Negative, non-monotonic, or out-of-bounds offsets could therefore reach unchecked offset stitching and buffer slicing, producing the reported Arrow panic. The conversion also passed the element-based `ArrayData::offset()` directly to a byte-based buffer slice. ## Validation - `cargo test -p lance-encoding data::tests::` (34 passed) - `cargo test -p lance-encoding --lib -- --skip test_sparse_large_string_list` (554 passed, 5 ignored, 2 filtered) - `cargo clippy --all --tests --benches -- -D warnings` - `cargo fmt --all` - `git diff --check` The unfiltered crate run was stopped after the existing `test_sparse_large_string_list` miniblock stress case ran for several minutes; the library suite was then rerun with its two parameterized cases filtered as shown above. Fixes lance-format#5303  --------- Co-authored-by: Gatefixer <312823363+lance-gatefixer[bot]@users.noreply.github.com>

## Summary - include Boolean must-not clauses in FTS column introspection and implicit column filling - include must-not index fragment coverage when building the shared FTS prefilter - add regression coverage for cross-column Boolean queries with different index coverage and partially specified columns ## Tests - `cargo fmt --all -- --check` - `cargo test -p lance-index scalar::inverted::query::tests --lib` - `cargo test -p lance --features slow_tests --test integration_tests test_boolean_must_not_uses_all_index_fragment_coverage` - `cargo clippy --all --tests --benches -- -D warnings` *(blocked by pre-existing `main` test compilation errors in `rust/lance/src/dataset/transaction.rs`: test constructors still use `merged_generations`, while `Operation::Update` and the protobuf now expose `compacted_sstables`)*

…format#8588) ## Why `lance-format#7589` made the FSST output-buffer contract 8×, but still trusted on-disk symbol lengths and value offsets. A crafted Lance file can inflate `lens[]` so `decompress_bulk` writes past that buffer. Readers that open untrusted datasets (dataset viewers, upload scanners) crash, and the overflow is a heap write with attacker-controlled values and stride. This change makes `fsst::decompress` the security boundary. Declared symbol lengths must be `1..=8`. Offsets must convert with `to_usize`, be non-decreasing, and stay inside the compressed buffer. Corrupt input returns `InvalidData`, mapped to `corrupt_file` by the encoding adapters. Valid files and the 8× `write_unaligned` fast path are unchanged. ## Benchmark FSST string decode of a 1 MiB Hamlet corpus, 2000 `decompress` calls, release, same host. The measured head is this PR; the baseline is `origin/main` (`8a8fb20c32`). | Workload | Baseline | This PR | |---|---|---| | Decode 1 MiB FSST strings | 11.9 GB/s | 11.7 GB/s | The difference is within run-to-run noise. Decode cost is unchanged.

## Bug Fix WAND can visit clauses in a different order from the query's canonical scoring order. Comparing that dynamic `f32` sum to the competitive floor before canonical rescoring can drop exact ties. Grouped BM25 and document-weight bounds also need outward rounding, and a zero competitive floor must not erase zero-score membership. This PR: - preserves query-order `f32` scoring for final results across classic WAND, MAXSCORE, and bulk AND - uses conservative widened sums only for pruning bounds - separates inclusive compound-scoring floors from standalone exclusive top-k floors - widens BM25 and grouped-posting upper bounds - preserves zero-score membership in composable WAND cursors - adds adversarial ULP, tie, zero-score, MAXSCORE, bulk-AND, and grouped-bound regressions This is the first PR in the OSS-1603 stack and only establishes scorer exactness. It does **not** enable the cross-column planner/executor yet. ## Scope boundary This stack does not include the previously deferred same-column delayed `MUST_NOT` probing work from OSS-1705. This PR only contains general WAND scoring and bound correctness needed by the cross-column execution path. ## Validation - `cargo test -p lance-index scalar::inverted::wand::tests` — 93 passed - `cargo check -p lance-index --tests` - `cargo clippy -p lance-index --tests -- -D warnings` - `cargo fmt --all -- --check` The full workspace CI matrix is left to GitHub CI. ## Stack 1. **This PR:** WAND scoring and bound exactness 2. Posting loading and cache policy 3. Row-address scorer foundations 4. Cross-column compound scorer core 5. Dataset planner / execution integration Part of [OSS-1603](https://linear.app/lancedb/issue/OSS-1603/add-candidate-driven-execution-for-cross-column-boolean-fts-queries).

github-actions Bot added enhancement New feature or request A-index A-python labels Jun 15, 2026

sezruby force-pushed the knn-batch-6822 branch 2 times, most recently from dd01c55 to 35e21ad Compare June 22, 2026 15:18

sezruby force-pushed the knn-batch-6822 branch from 35e21ad to 05923f1 Compare July 6, 2026 20:42

sezruby mentioned this pull request Jul 6, 2026

Extend batch vector queries to ANN and indexed search lance-format/lance#6822

Open

sezruby force-pushed the knn-batch-6822 branch from 05923f1 to a322a34 Compare July 6, 2026 20:55

sezruby closed this Jul 6, 2026

sezruby reopened this Jul 6, 2026

sezruby force-pushed the knn-batch-6822 branch from a322a34 to e0f8167 Compare July 24, 2026 22:58

lance-gatekeeper Bot and others added 19 commits August 4, 2026 00:48

fix: retain empty vector index definitions in retain_relevant_indices (…

b56b80d

…lance-format#8208)

perf(transaction): avoid O(n*m) fragment compares in build_manifest f…

14c5f3c

…or Update/Delete (lance-format#8210) Closes: lance-format#8209

Xuanwo and others added 30 commits August 18, 2026 19:47

chore: release beta version 11.0.0-beta.15

8064b3a

fix(java): preserve sessions on dataset checkout (lance-format#8650)

b945b3e

Merge branch 'main' into knn-batch-6822

7282281

Merge branch 'main' into knn-batch-6822

a1060cd

Navigation Menu

feat(index): share IVF partition scans across batch vector queries - #2

feat(index): share IVF partition scans across batch vector queries#2
sezruby wants to merge 535 commits into
mainfrom
knn-batch-6822

sezruby commented Jun 15, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

sezruby commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Approach

Design notes (pre-empting review questions)

Fallback matrix (no regression)

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

sezruby commented Jun 15, 2026 •

edited

Loading