Skip to content

[core] Add bucket-level primary-key vector index maintenance#8562

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/pk-vector-02-maintainer
Jul 11, 2026
Merged

[core] Add bucket-level primary-key vector index maintenance#8562
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/pk-vector-02-maintainer

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Builds the bucket-local maintenance layer for primary-key vector indexes on top of #8549. The maintainer restores active ANN state, reads complete vectors from compact data files, and produces index-file increments for compaction transitions without activating the write or query paths yet.

Changes

  • Restore immutable bucket index state from active ANN payloads and validate vector field, index type, payload uniqueness, and one-to-one source-file coverage.
  • Read vector-only projections from physical compact data-file rows without deletion-vector filtering, preserving ordinals for later search-time filtering.
  • Maintain ANN payloads across compaction by removing segments that reference replaced files and rebuilding one multi-source segment for uncovered active sources.
  • Restrict eligible sources to COMPACT files above level 0, where rows are complete for deduplicate and partial-update tables.
  • Allow source-backed ANN segments to coexist in index manifests even though their ordinal ranges are segment-local and overlap.
  • Clean up newly created ANN payloads when a maintenance attempt fails.

Testing

  • Build a single-source ANN segment from a compact data file.
  • Rebuild a multi-source ANN segment after partial compaction.
  • Reject duplicate source coverage and mismatched vector index types during restore.
  • Read vectors from a real Avro compact data file with vector-only projection while preserving NULL physical positions.
  • Combine source-backed manifest entries with overlapping segment-local ordinal ranges.
  • Preserve ordinary global-index overlap validation in mixed source-backed/range-backed manifests.
  • Run the focused Core suite: 15 tests passed.
  • Run the non-fast Core reactor build with Checkstyle, Spotless, and Maven enforcers.

Notes

  • Follows [core] Add primary-key vector index foundation #8549, which introduced the primary-key vector index metadata and configuration foundation.
  • This PR intentionally does not wire the maintainer into the write lifecycle and does not add query routing. Those are follow-up changes.

@JingsongLi JingsongLi marked this pull request as ready for review July 11, 2026 14:44

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one manifest correctness regression. The PR's focused 14-test suite passes, but a mixed source-backed/range-backed regression test reproduces the issue.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the latest head. The source-backed exemption is now scoped to source-backed pairs, the mixed-entry regression is covered, and the focused 15-test suite passes locally. LGTM.

@JingsongLi JingsongLi merged commit 50b0f70 into apache:master Jul 11, 2026
12 checks passed
JingsongLi added a commit that referenced this pull request Jul 11, 2026
Activates primary-key vector index maintenance in the write path on top
of #8549 and #8562. Complete Level 1 compact files become ANN sources,
and bucket-local ANN additions and removals are published with the same
compact increment as their source data transition.
@JingsongLi JingsongLi deleted the codex/pk-vector-02-maintainer branch July 13, 2026 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants