Skip to content

[WIP] Support full-history Paimon table clone#8616

Draft
LsomeYeah wants to merge 2 commits into
apache:masterfrom
LsomeYeah:full-history-clone
Draft

[WIP] Support full-history Paimon table clone#8616
LsomeYeah wants to merge 2 commits into
apache:masterfrom
LsomeYeah:full-history-clone

Conversation

@LsomeYeah

Copy link
Copy Markdown
Contributor

Summary

Add a physical full-history clone mode for Paimon tables across explicitly mapped storage locations. The clone preserves retained table history and rewrites metadata references at the target while leaving the existing logical clone behavior unchanged.

Changes

  • add clone_mode=full-history and repeatable path_mapping options to the Flink clone action and procedure
  • discover payload files reachable from retained main and branch snapshots, tags, and long-lived changelogs
  • copy normal and external data files, extra files, and index files with independent source and target FileIO instances
  • restore historical schemas and rewrite manifests, manifest lists, snapshots, tags, branches, changelogs, statistics, indexes, and hints
  • add source fingerprints, ownership/completion markers, two-phase payload publication, and idempotent retry checks
  • keep file-copy parallelism independent of table bucket count
  • document the full-history interface, operational contract, and limitations

Testing

  • mvn -s ~/.m2/settings.xml.paimon-ali -pl paimon-core -Dtest='org.apache.paimon.clone.*Test,org.apache.paimon.tag.TagTimeExpireTest' test
  • mvn -s ~/.m2/settings.xml.paimon-ali -pl paimon-flink/paimon-flink-common -Pflink1 -DwildcardSuites=none -DfailIfNoTests=false -Dtest='org.apache.paimon.flink.action.CloneActionTest,org.apache.paimon.flink.clone.history.*Test,org.apache.paimon.flink.procedure.CloneProcedureTest,org.apache.paimon.flink.procedure.ProcedureTest' clean test
  • mvn -s ~/.m2/settings.xml.paimon-ali -pl paimon-flink/paimon-flink-common -Pflink1 -DwildcardSuites=none -DfailIfNoTests=false -Dtest=org.apache.paimon.flink.action.CloneFullHistoryActionITCase test

Current Limitations

  • target catalog registration is not included
  • blob descriptors, blob views, and Iceberg compatibility metadata are not migrated
  • filtered, format-converting, metadata-only, and Spark execution modes are not included
  • mapped external data and index locations must be dedicated to the clone
  • existing-file validation compares sizes rather than checksums

Notes

This PR is marked WIP to collect feedback on the API, metadata rewrite protocol, and first-version scope before it is made ready for merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant