fix(fts): preserve exact wand score bounds - #8666
Merged
Merged
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
Both prior rounding findings are fixed. The revision now preserves canonical f32 scoring while outward-conservative partial, grouped-list, and grouped-block bounds keep inclusive ties and strictly better grouped hits eligible.
Xuanwo
approved these changes
Aug 20, 2026
BubbleCal
added a commit
that referenced
this pull request
Aug 21, 2026
## Other Changes
Read-ahead posting groups reduce storage requests and improve cache
reuse, but a future candidate-driven consumer also needs a way to avoid
reading neighboring token rows on a cold miss.
This PR adds the internal loading infrastructure for that choice:
- introduces `PostingReadPolicy::{ReadAhead, CacheAwareExact}`
- lets exact reads reuse an already-resident read-ahead group
- uses a singleton `[token_id, token_id + 1)` cache entry on a cold
exact miss
- retains cache singleflight for concurrent reads of the same singleton
- coalesces multiple demanded tokens from one group back to one
read-ahead load
- treats repeated logical occurrences of one physical token as one load
demand
- simplifies position-group matching without changing its semantics
- adds deterministic I/O and cache-metric regression coverage
## Behavior and compatibility
There is no public API or file-format change.
The existing production entry point still selects `ReadAhead`. The
`CacheAwareExact` constructor is test-only in this layer and will be
enabled by the later cross-column consumer PR, so merging this PR alone
does not change production query behavior.
A cache-only probe miss is intentionally omitted from the query-level
`MetricsCollector`; only the path that serves the posting records a
query cache hit or miss. The lower-level `LanceCache::stats()` still
observes the probe.
## Scope boundary
This is PR 2 of the OSS-1603 stack and depends only on the exactness
foundation merged in #8666.
It does not include the previously deferred same-column delayed
`MUST_NOT` probing work from OSS-1705, and it does not add or change
Boolean execution.
## Validation
- `cargo fmt --all -- --check`
- `git diff --check origin/main...HEAD`
- regression coverage for cold singleton reads, concurrent singleflight,
warm hits, prewarmed-group reuse, singleton-then-prewarm precedence,
unchanged default read-ahead, same-group multi-term coalescing, and
repeated-token occurrences
Per project workflow, GitHub CI is the authoritative Cargo test and
Clippy run for this rebased head.
## Performance validation
No end-to-end performance benefit is claimed for this layer because
there is no production `CacheAwareExact` caller yet. A main-vs-PR2 query
benchmark would execute the same `ReadAhead` path on both builds and
measure noise rather than this code.
The first consumer PR will benchmark the policy independently with cold
singleton-demand, cold same-group multi-demand, and warm/prewarmed ABBA
cases, including bytes, requests, posting loads, cache metrics, and
exact result digests.
## Stack
1. #8666 — WAND scoring and bound exactness (merged)
2. **This PR:** posting loading and cache policy
3. Row-address scorer foundations
4. Cross-column compound scorer core
5. Dataset planner / execution integration
Part of
[OSS-1603](https://linear.app/lancedb/issue/OSS-1603/add-candidate-driven-execution-for-cross-column-boolean-fts-queries).
BubbleCal
added a commit
that referenced
this pull request
Aug 21, 2026
## Feature
Each FTS index source scores in its own partition-local DocId domain.
Cross-column compound queries need to compose those scorers in one
globally ordered row-address domain without assuming independently built
indices share local document layouts.
This PR adds the cross-column scorer core:
- validates and caches strictly ordered row-address projections
- maps local scorer iteration, shallow bounds, and advances into
row-address order
- materializes an exact fallback for reordered projections
- rejects duplicate row-address mappings that could combine content from
different local documents
- lazily merges multiple physical sources for one semantic leaf
- preserves conservative bounds, competitive floors, and equal-score
row-address ties
- analyzes compound plans for positive generator coverage, cost,
feasibility, and conservative score bounds
- supports staged positive generation followed by candidate-scoped
required, optional, and prohibited probing
- performs one exact global top-k collection after composing Match,
Phrase, Boolean, MultiMatch, and Boost scorers
- uses candidate-scoped row-address and scoring-length reads when
staging is selective
The dataset Scanner does not call this core yet; that integration
remains in the final consumer PR.
## API and compatibility
There is no file-format change.
This PR exports the low-level async Rust entry point used by the later
`lance` planner integration:
```rust
pub async fn cross_column_compound_search(
columns: &[(String, Vec<Arc<InvertedIndex>>)],
query: &FtsQuery,
params: &FtsSearchParams,
prefilter: Arc<dyn PreFilter>,
metrics: Arc<dyn MetricsCollector>,
) -> Result<(Vec<u64>, Vec<f32>)>
```
The API requires a bounded limit, preserves exact `(score DESC,
row_address ASC)` ordering, and returns an error for unsupported or
internally inconsistent scorer state.
Existing `PartitionDocuments::resolve_addresses` and
`estimated_address_read_bytes` behavior remains unchanged from `main`,
so current production cache and I/O behavior is not altered before
Scanner integration.
## CI failure addressed
The previous head split scorer foundations from their production
consumer, leaving 43 groups of private items unused under `-D
dead-code`. This revision folds the cross-column core into the same PR
so those components have real production call paths.
Unused leaf-role metadata was deleted. No `allow(dead_code)`, test-only
gating, or visibility workaround was added.
## Scope boundary
This is PR 3 of the OSS-1603 stack and builds on #8666 and #8667, both
merged.
It does not include the previously deferred same-column delayed
`MUST_NOT` probing work from OSS-1705. Cross-column prohibited clauses
are supported as ordinary query semantics, but this PR does not change
the same-column `BooleanScorer` or its probing strategy.
## Validation
- `cargo fmt --all -- --check`
- `git diff --check origin/main...HEAD`
- deterministic projection tests for ordered, deleted, remapped,
duplicate, and out-of-order layouts
- materialized fallback and collision regressions
- merge-scorer ordering, lazy initialization, advance, shallow-bound,
floor, tie, and duplicate tests
- seeded randomized exhaustive oracle coverage for SUM, MAX, MUST,
required-optional, signed Boost, and nested prohibited shapes
- staged generator, candidate resolution, phrase, visibility, and
quantized-length tests
- static reachability audit covering every item reported by the failed
CI jobs
Per project workflow, GitHub CI is the authoritative Cargo test and
Clippy run for this revised head.
## Performance validation
No end-to-end performance benefit is claimed yet because the dataset
Scanner does not call this entry point in this PR. A main-vs-this-PR
dataset benchmark would execute the existing fallback path and measure
noise.
The final consumer PR will enable this core and report independent
warm/cold ABBA results with latency, throughput, CPU, bytes, requests,
cache metrics, and exact result digests.
## Stack
1. #8666 — WAND scoring and bound exactness (merged)
2. #8667 — posting loading and cache policy (merged)
3. **This PR:** row-address and cross-column compound scorer core
4. Dataset planner / execution integration and end-to-end benchmark
Part of
[OSS-1603](https://linear.app/lancedb/issue/OSS-1603/add-candidate-driven-execution-for-cross-column-boolean-fts-queries).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug Fix
WAND can visit clauses in a different order from the query's canonical scoring order. Comparing that dynamic
f32sum to the competitive floor before canonical rescoring can drop exact ties. Grouped BM25 and document-weight bounds also need outward rounding, and a zero competitive floor must not erase zero-score membership.This PR:
f32scoring for final results across classic WAND, MAXSCORE, and bulk ANDThis is the first PR in the OSS-1603 stack and only establishes scorer exactness. It does not enable the cross-column planner/executor yet.
Scope boundary
This stack does not include the previously deferred same-column delayed
MUST_NOTprobing work from OSS-1705. This PR only contains general WAND scoring and bound correctness needed by the cross-column execution path.Validation
cargo test -p lance-index scalar::inverted::wand::tests— 93 passedcargo check -p lance-index --testscargo clippy -p lance-index --tests -- -D warningscargo fmt --all -- --checkThe full workspace CI matrix is left to GitHub CI.
Stack
Part of OSS-1603.