Skip to content

[core] Support full-text search for primary-key tables#8648

Draft
JingsongLi wants to merge 4 commits into
apache:masterfrom
JingsongLi:codex/pk-fulltext
Draft

[core] Support full-text search for primary-key tables#8648
JingsongLi wants to merge 4 commits into
apache:masterfrom
JingsongLi:codex/pk-fulltext

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

Add compaction-visible full-text indexing and search for primary-key tables. The index follows the primary-key table lifecycle, filters physical rows with deletion vectors, and incrementally consolidates immutable archives with the shared primary-key index LSM policy.

Changes

  • Add primary-key full-text configuration, schema validation, deterministic analyzer fingerprints, and source metadata for ordered multi-file archives.
  • Build bucket-local native full-text archives from level-1-or-higher compact output and maintain them asynchronously with shared fanout and stale-ratio compaction options.
  • Support multi-source archive row-ID mapping, deletion-vector offsets, atomic publish/rollback, stale-source reclamation, and compatibility with existing single-source archives.
  • Add snapshot-scoped full-text scan/read/search over physical data-file positions, deterministic RRF ranking, temporary-index fallback for uncovered eligible files, and hybrid vector/full-text fusion.
  • Expose primary-key full-text search through Spark SQL and the Flink full_text_search procedure, with documentation for configuration, visibility, maintenance, and limitations.

Testing

  • JDK 8: 21 focused paimon-core test classes (144 tests)
  • Spark 3 / JDK 8: FullTextSearchTest and HybridSearchTest (13 tests)
  • Spark 4 / JDK 17: FullTextSearchTest and HybridSearchTest (13 tests)
  • Flink 1 / JDK 8: FullTextSearchProcedureITCase
  • Flink 2 / JDK 11: FullTextSearchProcedureITCase
  • JDK 8: clean paimon-core and paimon-full-text compile with Checkstyle, Spotless, and Maven Enforcer
  • Native primary-key full-text contract test invoked; it runs when PAIMON_FTINDEX_JNI_LIB_PATH is available

Notes

  • Primary-key full-text uses the fixed full-text implementation; no index.type option is introduced.
  • Level-0 foreground files remain outside full-text coverage until compaction.
  • The first release supports one character full-text column per primary-key table.
  • Python API changes are intentionally out of scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant