[core] Search primary-key vector indexes#8579
Merged
JingsongLi merged 4 commits intoJul 12, 2026
Merged
Conversation
leaves12138
approved these changes
Jul 12, 2026
leaves12138
left a comment
Contributor
There was a problem hiding this comment.
Reviewed the latest Core head (80e1f79). The snapshot-scoped planning, deletion-vector alignment, bucket-local ANN/exact fallback, deterministic global Top-K, and physical-position materialization are consistent. The new GlobalIndexSplitResult abstraction also cleanly removes the batch-scan dependency on the concrete primary-key vector result. I ran the focused 11-test Core suite on JDK 8 with Checkstyle, Spotless, Maven enforcers, compilation, and packaging; all passed. Looks good to me.
9 tasks
Open
6 tasks
JingsongLi
added a commit
that referenced
this pull request
Jul 13, 2026
Add end-to-end Spark SQL and Flink procedure support for primary-key vector search backed by the bucket-local indexes introduced in #8579. The PR supports every primary-key merge engine, distributes sufficiently large Spark searches by bucket group, and documents table creation, index maintenance, compaction freshness, query APIs, and current limitations.
JunRuiLee
added a commit
to JunRuiLee/paimon-rust
that referenced
this pull request
Jul 13, 2026
Rust equivalent of Java `PrimaryKeyVectorRead` + the read subset of `PrimaryKeyVectorResult.splits()` (apache/paimon#8579). Per-bucket search via `bucket_search`, cross-bucket global Top-K merge, grouping survivors by data file into `PkVectorIndexedSplit`s, and lazy materialization via `PkVectorIndexedSplitRead`. Global Top-K is a full sort + truncate over the collected candidates (N <= buckets * limit) on a five-level BEST_FIRST key (distance, partition bytes, bucket, file name, row position; `total_cmp` for NaN-safety). Grouping fails loud on cross-split ambiguity, duplicate (file, position), or a hit referencing a file absent from its bucket split. Adds `VectorSearchMetric::distance_to_score` (mirrors Java `PrimaryKeyVectorResult.score(distance)`). Inputs are synthetic per-bucket splits; snapshot/manifest planning, table routing, and real dependency construction are not yet implemented, so the module carries a temporary `#[allow(dead_code)]`.
JunRuiLee
added a commit
to JunRuiLee/paimon-rust
that referenced
this pull request
Jul 14, 2026
Rust equivalent of Java `PrimaryKeyVectorRead` + the read subset of `PrimaryKeyVectorResult.splits()` (apache/paimon#8579). Per-bucket search via `bucket_search`, cross-bucket global Top-K merge, grouping survivors by data file into `PkVectorIndexedSplit`s, and lazy materialization via `PkVectorIndexedSplitRead`. Global Top-K is a full sort + truncate over the collected candidates (N <= buckets * limit) on a five-level BEST_FIRST key (distance, partition bytes, bucket, file name, row position; `total_cmp` for NaN-safety). Grouping fails loud on cross-split ambiguity, duplicate (file, position), or a hit referencing a file absent from its bucket split. Adds `VectorSearchMetric::distance_to_score` (mirrors Java `PrimaryKeyVectorResult.score(distance)`). Inputs are synthetic per-bucket splits; snapshot/manifest planning, table routing, and real dependency construction are not yet implemented, so the module carries a temporary `#[allow(dead_code)]`.
JunRuiLee
added a commit
to JunRuiLee/paimon-rust
that referenced
this pull request
Jul 14, 2026
Rust equivalent of Java `PrimaryKeyVectorRead` + the read subset of `PrimaryKeyVectorResult.splits()` (apache/paimon#8579). Per-bucket search via `bucket_search`, cross-bucket global Top-K merge, grouping survivors by data file into `PkVectorIndexedSplit`s, and lazy materialization via `PkVectorIndexedSplitRead`. Global Top-K is a full sort + truncate over the collected candidates (N <= buckets * limit) on a five-level BEST_FIRST key (distance, partition bytes, bucket, file name, row position; `total_cmp` for NaN-safety). Grouping fails loud on cross-split ambiguity, duplicate (file, position), or a hit referencing a file absent from its bucket split. Adds `VectorSearchMetric::distance_to_score` (mirrors Java `PrimaryKeyVectorResult.score(distance)`). Inputs are synthetic per-bucket splits; snapshot/manifest planning, table routing, and real dependency construction are not yet implemented, so the module carries a temporary `#[allow(dead_code)]`.
JunRuiLee
added a commit
to JunRuiLee/paimon-rust
that referenced
this pull request
Jul 14, 2026
Rust equivalent of Java `PrimaryKeyVectorRead` + the read subset of `PrimaryKeyVectorResult.splits()` (apache/paimon#8579). Per-bucket search via `bucket_search`, cross-bucket global Top-K merge, grouping survivors by data file into `PkVectorIndexedSplit`s, and lazy materialization via `PkVectorIndexedSplitRead`. Global Top-K is a full sort + truncate over the collected candidates (N <= buckets * limit) on a five-level BEST_FIRST key (distance, partition bytes, bucket, file name, row position; `total_cmp` for NaN-safety). Grouping fails loud on cross-split ambiguity, duplicate (file, position), or a hit referencing a file absent from its bucket split. Adds `VectorSearchMetric::distance_to_score` (mirrors Java `PrimaryKeyVectorResult.score(distance)`). Inputs are synthetic per-bucket splits; snapshot/manifest planning, table routing, and real dependency construction are not yet implemented, so the module carries a temporary `#[allow(dead_code)]`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the complete Core path for single-query vector search on primary-key tables. A query captures one snapshot, searches bucket-local ANN payloads with exact fallback for uncovered compact files, merges a deterministic global Top-K, and materializes the selected physical rows through the existing
IndexedSplitread path.Changes
(partition, bucket).PrimaryKeyVectorBucketSearchfor every planned bucket, applying snapshot deletion vectors to both ANN and exact fallback results.(partition, bucket, file, position, distance)candidates into one-fileIndexedSplits with compressed physical ranges and scores aligned to expanded range order.DataTableBatchScanconsume the physical result directly at the captured snapshot instead of scanning a newer snapshot.GlobalIndexSplitResultsoDataTableBatchScandepends only on the genericsnapshotId()andsplits()capability, not on the primary-key vector implementation.VectorSearchBuilderautomatically to the primary-key vector path when the selected field has a configured primary-key vector index.DataEvolutionVectorScanandDataEvolutionVectorRead, including Spark inheritance and batch scan routing.API and behavior
The public API remains unchanged:
For a configured primary-key vector field,
executeLocal()returns a snapshot-scoped physical result. Passing that result to the table scan produces existingIndexedSplits; no new table split or row-id namespace is introduced.Partition filters are supported. Non-partition scalar filters are rejected for this path because they cannot currently be applied before bucket Top-K without changing search correctness. Batch vector search remains outside this PR.
Testing
IndexedSplit.DataTableBatchScan, and physical row reading.VectorSearchBuilderTestcases plus the primary-key end-to-end test after the rename.-Pfast-build.Notes
IndexedSplit.