Skip to content

[core] Search primary-key vector indexes#8579

Merged
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/pk-vector-06-snapshot-scan
Jul 12, 2026
Merged

[core] Search primary-key vector indexes#8579
JingsongLi merged 4 commits into
apache:masterfrom
JingsongLi:codex/pk-vector-06-snapshot-scan

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the complete Core path for single-query vector search on primary-key tables. A query captures one snapshot, searches bucket-local ANN payloads with exact fallback for uncovered compact files, merges a deterministic global Top-K, and materializes the selected physical rows through the existing IndexedSplit read path.

Changes

  • Plan one snapshot consistently across data files, deletion vectors, and active vector payloads, grouped by (partition, bucket).
  • Execute PrimaryKeyVectorBucketSearch for every planned bucket, applying snapshot deletion vectors to both ANN and exact fallback results.
  • Merge bucket candidates into a deterministic global Top-K by distance and physical location.
  • Convert (partition, bucket, file, position, distance) candidates into one-file IndexedSplits with compressed physical ranges and scores aligned to expanded range order.
  • Let DataTableBatchScan consume the physical result directly at the captured snapshot instead of scanning a newer snapshot.
  • Introduce GlobalIndexSplitResult so DataTableBatchScan depends only on the generic snapshotId() and splits() capability, not on the primary-key vector implementation.
  • Route VectorSearchBuilder automatically to the primary-key vector path when the selected field has a configured primary-key vector index.
  • Preserve compatibility with delegated table wrappers when constructing the key-value vector reader.
  • Rename the existing data-evolution implementations to DataEvolutionVectorScan and DataEvolutionVectorRead, including Spark inheritance and batch scan routing.

API and behavior

The public API remains unchanged:

GlobalIndexResult result =
        table.newVectorSearchBuilder()
                .withVectorColumn("embedding")
                .withVector(query)
                .withLimit(k)
                .executeLocal();

TableScan.Plan readPlan =
        table.newReadBuilder().newScan().withGlobalIndexResult(result).plan();

For a configured primary-key vector field, executeLocal() returns a snapshot-scoped physical result. Passing that result to the table scan produces existing IndexedSplits; no new table split or row-id namespace is introduced.

Partition filters are supported. Non-partition scalar filters are rejected for this path because they cannot currently be applied before bucket Top-K without changing search correctness. Batch vector search remains outside this PR.

Testing

  • Merge fragmented data splits into complete bucket inputs while preserving deletion-vector alignment.
  • Keep data files and vector payloads on the same captured snapshot.
  • Filter payloads by vector field id and index type.
  • Preserve bucket search inputs through Java serialization.
  • Merge global Top-K deterministically.
  • Compress physical positions and align scores and deletion files in IndexedSplit.
  • Run an end-to-end public builder query through synchronous index creation, ANN search, DataTableBatchScan, and physical row reading.
  • Run 14 focused Core tests covering scan, bucket search, result conversion, indexed split reads, position reads, raw-file reads, and end-to-end search.
  • Run all 33 VectorSearchBuilderTest cases plus the primary-key end-to-end test after the rename.
  • Run Checkstyle, Spotless, Maven enforcers, compilation, and focused tests without -Pfast-build.

Notes

@JingsongLi JingsongLi changed the title [core] Plan primary-key vector search by snapshot [core] Search primary-key vector indexes Jul 12, 2026
@JingsongLi JingsongLi marked this pull request as ready for review July 12, 2026 13:08

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the latest Core head (80e1f79). The snapshot-scoped planning, deletion-vector alignment, bucket-local ANN/exact fallback, deterministic global Top-K, and physical-position materialization are consistent. The new GlobalIndexSplitResult abstraction also cleanly removes the batch-scan dependency on the concrete primary-key vector result. I ran the focused 11-test Core suite on JDK 8 with Checkstyle, Spotless, Maven enforcers, compilation, and packaging; all passed. Looks good to me.

@JingsongLi JingsongLi merged commit a50a36f into apache:master Jul 12, 2026
12 checks passed
@JingsongLi JingsongLi deleted the codex/pk-vector-06-snapshot-scan branch July 13, 2026 00:18
JingsongLi added a commit that referenced this pull request Jul 13, 2026
Add end-to-end Spark SQL and Flink procedure support for primary-key
vector search backed by the bucket-local indexes introduced in #8579.
The PR supports every primary-key merge engine, distributes sufficiently
large Spark searches by bucket group, and documents table creation,
index maintenance, compaction freshness, query APIs, and current
limitations.
JunRuiLee added a commit to JunRuiLee/paimon-rust that referenced this pull request Jul 13, 2026
Rust equivalent of Java `PrimaryKeyVectorRead` + the read subset of
`PrimaryKeyVectorResult.splits()` (apache/paimon#8579). Per-bucket search
via `bucket_search`, cross-bucket global Top-K merge, grouping survivors
by data file into `PkVectorIndexedSplit`s, and lazy materialization via
`PkVectorIndexedSplitRead`.

Global Top-K is a full sort + truncate over the collected candidates
(N <= buckets * limit) on a five-level BEST_FIRST key
(distance, partition bytes, bucket, file name, row position; `total_cmp`
for NaN-safety). Grouping fails loud on cross-split ambiguity, duplicate
(file, position), or a hit referencing a file absent from its bucket
split.

Adds `VectorSearchMetric::distance_to_score` (mirrors Java
`PrimaryKeyVectorResult.score(distance)`). Inputs are synthetic
per-bucket splits; snapshot/manifest planning, table routing, and real
dependency construction are not yet implemented, so the module carries a
temporary `#[allow(dead_code)]`.
JunRuiLee added a commit to JunRuiLee/paimon-rust that referenced this pull request Jul 14, 2026
Rust equivalent of Java `PrimaryKeyVectorRead` + the read subset of
`PrimaryKeyVectorResult.splits()` (apache/paimon#8579). Per-bucket search
via `bucket_search`, cross-bucket global Top-K merge, grouping survivors
by data file into `PkVectorIndexedSplit`s, and lazy materialization via
`PkVectorIndexedSplitRead`.

Global Top-K is a full sort + truncate over the collected candidates
(N <= buckets * limit) on a five-level BEST_FIRST key
(distance, partition bytes, bucket, file name, row position; `total_cmp`
for NaN-safety). Grouping fails loud on cross-split ambiguity, duplicate
(file, position), or a hit referencing a file absent from its bucket
split.

Adds `VectorSearchMetric::distance_to_score` (mirrors Java
`PrimaryKeyVectorResult.score(distance)`). Inputs are synthetic
per-bucket splits; snapshot/manifest planning, table routing, and real
dependency construction are not yet implemented, so the module carries a
temporary `#[allow(dead_code)]`.
JunRuiLee added a commit to JunRuiLee/paimon-rust that referenced this pull request Jul 14, 2026
Rust equivalent of Java `PrimaryKeyVectorRead` + the read subset of
`PrimaryKeyVectorResult.splits()` (apache/paimon#8579). Per-bucket search
via `bucket_search`, cross-bucket global Top-K merge, grouping survivors
by data file into `PkVectorIndexedSplit`s, and lazy materialization via
`PkVectorIndexedSplitRead`.

Global Top-K is a full sort + truncate over the collected candidates
(N <= buckets * limit) on a five-level BEST_FIRST key
(distance, partition bytes, bucket, file name, row position; `total_cmp`
for NaN-safety). Grouping fails loud on cross-split ambiguity, duplicate
(file, position), or a hit referencing a file absent from its bucket
split.

Adds `VectorSearchMetric::distance_to_score` (mirrors Java
`PrimaryKeyVectorResult.score(distance)`). Inputs are synthetic
per-bucket splits; snapshot/manifest planning, table routing, and real
dependency construction are not yet implemented, so the module carries a
temporary `#[allow(dead_code)]`.
JunRuiLee added a commit to JunRuiLee/paimon-rust that referenced this pull request Jul 14, 2026
Rust equivalent of Java `PrimaryKeyVectorRead` + the read subset of
`PrimaryKeyVectorResult.splits()` (apache/paimon#8579). Per-bucket search
via `bucket_search`, cross-bucket global Top-K merge, grouping survivors
by data file into `PkVectorIndexedSplit`s, and lazy materialization via
`PkVectorIndexedSplitRead`.

Global Top-K is a full sort + truncate over the collected candidates
(N <= buckets * limit) on a five-level BEST_FIRST key
(distance, partition bytes, bucket, file name, row position; `total_cmp`
for NaN-safety). Grouping fails loud on cross-split ambiguity, duplicate
(file, position), or a hit referencing a file absent from its bucket
split.

Adds `VectorSearchMetric::distance_to_score` (mirrors Java
`PrimaryKeyVectorResult.score(distance)`). Inputs are synthetic
per-bucket splits; snapshot/manifest planning, table routing, and real
dependency construction are not yet implemented, so the module carries a
temporary `#[allow(dead_code)]`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants