Skip to content

feat(c): add write/commit C FFI bindings#522

Open
suxiaogang223 wants to merge 3 commits into
apache:mainfrom
suxiaogang223:feat/c-ffi-write-support
Open

feat(c): add write/commit C FFI bindings#522
suxiaogang223 wants to merge 3 commits into
apache:mainfrom
suxiaogang223:feat/c-ffi-write-support

Conversation

@suxiaogang223

Copy link
Copy Markdown
Member

Summary

Add write-side C FFI bindings (bindings/c/src/write.rs) following the same opaque handle / extern "C" patterns as the existing read-side bindings. Closes #520.

API surface (14 functions)

// WriteBuilder
paimon_table_new_write_builder(table)        → paimon_result_write_builder
paimon_write_builder_free(wb)                → void
paimon_write_builder_with_overwrite(wb)      → *mut paimon_error
paimon_write_builder_new_write(wb)           → paimon_result_table_write
paimon_write_builder_new_commit(wb)          → paimon_result_table_commit

// TableWrite
paimon_table_write_free(tw)                  → void
paimon_table_write_write_arrow_batch(tw, array, schema) → *mut paimon_error
paimon_table_write_prepare_commit(tw)        → paimon_result_prepare_commit

// TableCommit
paimon_table_commit_free(tc)                 → void
paimon_table_commit_commit(tc, msgs)         → *mut paimon_error
paimon_table_commit_overwrite(tc, msgs)      → *mut paimon_error
paimon_table_commit_truncate_table(tc)       → *mut paimon_error
paimon_table_commit_abort(tc, msgs)          → *mut paimon_error

// CommitMessages
paimon_commit_messages_free(msgs)            → void

Design decisions

  • In-process transfer: CommitMessages are passed as opaque handles (no serialization), matching Python bindings
  • Shared commit_user: WriteBuilder generates a UUID shared by writer and committer
  • Arrow C Data Interface: Batch import via FFI_ArrowArray/FFI_ArrowSchema raw pointers

Tests (15 cases, all passing)

  • Read path (3), Predicate (3), Write path (7), Error handling (1), Integration (1)

Run: cargo +1.91.0 test -p paimon-c

Add write-side C FFI bindings following the existing read-side patterns:

API surface (14 functions):
- WriteBuilder: new_write_builder, free, with_overwrite, new_write, new_commit
- TableWrite: free, write_arrow_batch (via Arrow C Data Interface), prepare_commit
- TableCommit: free, commit, overwrite, truncate_table, abort
- CommitMessages: free

Tests (15 cases):
- Read path: empty table read, data read, column projection
- Predicate: equal, IS NOT NULL, IS IN, less_than filter, AND combinator
- Write path: builder lifecycle, full roundtrip, multiple batches, empty commit,
  overwrite mode, truncate, abort
- Error handling: null pointer checks
- Integration: two commits with same builder

Design follows Python bindings: in-process commit message transfer,
shared commit_user between writer and committer, Arrow C Data Interface
for zero-copy batch import.

Closes apache#520
Comment thread bindings/c/src/write.rs Outdated
Comment thread bindings/c/src/write.rs Outdated
Comment thread bindings/c/src/write.rs Outdated
Comment thread bindings/c/src/write.rs Outdated
Comment thread bindings/c/src/write.rs Outdated
Comment thread bindings/c/src/write.rs Outdated
@suxiaogang223

Copy link
Copy Markdown
Member Author

Thanks for the detailed review, @QuakeWang! I added regression tests to reproduce each of the issues you pointed out and confirmed the failures before making the fixes. The new tests now cover Arrow C Data ownership, invalid and schema-mismatched batches, commit message provenance and lifetime, index cleanup on abort, and caller-provided commit identity with idempotent retries. Everything is passing now, including the full paimon-c test suite and the targeted core tests. The fixes are in 27a9d90. When you get a chance, could you please take another look? Thanks!

@JingsongLi

Copy link
Copy Markdown
Contributor

No issue from me.

Comment thread crates/paimon/src/table/table_commit.rs Outdated
Comment thread bindings/c/src/write.rs Outdated
Comment thread bindings/c/src/write.rs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add write/commit C FFI bindings for bindings/c/

3 participants