Skip to content

[core] Maintain primary-key full-text index archives#8651

Open
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/pk-fulltext-maintenance
Open

[core] Maintain primary-key full-text index archives#8651
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/pk-fulltext-maintenance

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Build on #8649 by adding LSM-style incremental maintenance for primary-key full-text index archives.

Changes

  • add primary-key full-text index configuration and schema validation
  • integrate bucket-local full-text maintenance with BucketedPrimaryKeyIndexMaintainer
  • reuse PrimaryKeyIndexLevels and the Vector/Sorted compaction options to consolidate similarly sized archives and rebuild stale coverage
  • publish archive replacements through the existing append/compaction increments, including asynchronous completion, abort, rollback, and cleanup handling

Scope

  • no index.type option; full text remains the implicit/default index type
  • no definition fingerprint is persisted
  • query/search integration and Flink/Spark exposure are deferred to follow-up PRs

Testing

  • mvn -pl paimon-core -am -DskipTests package
  • mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest='PkFullTextIndexFileTest,PkFullTextBucketIndexStateTest,BucketedFullTextIndexMaintainerTest,BucketedPrimaryKeyIndexMaintainerTest,PrimaryKeyIndexDefinitionsTest,PrimaryKeyIndexWriteTest,PrimaryKeyFullTextIndexValidationTest,SchemaManagerTest' test
  • git diff --check origin/master..HEAD

The focused test set runs 84 tests.

@JingsongLi JingsongLi force-pushed the codex/pk-fulltext-maintenance branch from 0f8f29c to 5fe12ce Compare July 15, 2026 06:44
@JingsongLi JingsongLi force-pushed the codex/pk-fulltext-maintenance branch from 5fe12ce to a82e386 Compare July 15, 2026 06:49
@JingsongLi JingsongLi marked this pull request as ready for review July 15, 2026 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant