[Feature] Add write/commit C FFI bindings for bindings/c/

[suxiaogang223]

Background

Apache Doris (an MPP OLAP database) is adding Paimon write support. We currently have a working JNI-based write path that bridges to the Paimon Java SDK, but we'd like to add a Rust FFI backend for better performance (zero-copy Arrow, async I/O, no JVM overhead).

The existing bindings/c/ already provides a complete read-side C FFI (catalog, table, scan, plan, predicate, record batch streaming via Arrow C Data Interface). We'd like to contribute write-side C FFI bindings following the same patterns.

Current state

• bindings/c/ — read-side C FFI ✅ (catalog, table, read builder, scan, plan, predicate, record batch reader)
• bindings/python/ — Python bindings with full write support ✅ (PyWriteBuilder, PyTableWrite, PyTableCommit)
• bindings/c/ — write-side C FFI ❌ (nothing yet)

Proposed API surface

Following the existing naming conventions and opaque handle patterns from bindings/c/:

// === WriteBuilder ===
paimon_write_builder *paimon_table_new_write_builder(paimon_table *table);
void paimon_write_builder_free(paimon_write_builder *wb);
void paimon_write_builder_with_overwrite(paimon_write_builder *wb);

// === TableWrite ===
paimon_table_write *paimon_write_builder_new_write(paimon_write_builder *wb, paimon_error **err);
void paimon_table_write_free(paimon_table_write *tw);

// write a single Arrow record batch via C Data Interface (zero-copy)
paimon_error *paimon_table_write_write_batch(
    paimon_table_write *tw,
    FFI_ArrowArray *array,
    FFI_ArrowSchema *schema);

// close file writers and produce commit messages
// returns paimon_commit_messages* on success, NULL + err on failure
paimon_commit_messages *paimon_table_write_prepare_commit(
    paimon_table_write *tw, paimon_error **err);

// === TableCommit ===
paimon_table_commit *paimon_write_builder_new_commit(paimon_write_builder *wb);
void paimon_table_commit_free(paimon_table_commit *tc);

paimon_error *paimon_table_commit_commit(
    paimon_table_commit *tc, paimon_commit_messages *msgs);
paimon_error *paimon_table_commit_overwrite(
    paimon_table_commit *tc, paimon_commit_messages *msgs,
    ... static_partitions ...);
paimon_error *paimon_table_commit_truncate_table(paimon_table_commit *tc);
paimon_error *paimon_table_commit_abort(
    paimon_table_commit *tc, paimon_commit_messages *msgs);

// === CommitMessage serialization ===
// For cross-process transfer, commit messages need serialization.
// Proposal: use protobuf for schema-safe, language-agnostic serialization,
// matching Paimon's existing DataFileMeta / CommitMessage struct fields.
paimon_bytes paimon_commit_messages_serialize(paimon_commit_messages *msgs);
paimon_commit_messages *paimon_commit_messages_deserialize(
    paimon_bytes data, paimon_error **err);
void paimon_commit_messages_free(paimon_commit_messages *msgs);

CommitMessage serialization — discussion needed

This is the key open question. Paimon's CommitMessage is a Rust struct with nested Vec<DataFileMeta>, Vec<IndexFileMeta>, etc. For C FFI consumers (like Doris) that need to transfer messages across a process boundary and eventually to a remote coordinator, serialization is required.

Options:

Format	Pros	Cons
Protobuf	Schema-safe, cross-language, Paimon already uses protobuf internally	Adds prost-build dependency to bindings/c
JSON	Simple, no schema needed	Bulky for DataFileMeta, slower
FlatBuffers	Zero-copy deserialization	More complex, less community familiarity
Paimon binary format	Reuses existing CommitMessageSerializer logic	Java-specific, needs porting to Rust

We lean toward Protobuf — it's the standard approach, Paimon already uses it, and it generates bindings for all languages. But we'd like community input before committing.

Implementation plan

1. Add bindings/c/src/write.rs with the API surface above
2. Add CommitMessage protobuf schema and serialization
3. Add tests under bindings/c/tests/ (following Python test patterns)
4. Update bindings/c/src/lib.rs to register the new module

Questions for the community

1. Does this API surface look reasonable? Any naming or pattern preferences?
2. Protobuf vs JSON vs other formats for CommitMessage serialization?
3. Should write_arrow_batch also accept a batch of batches (&[RecordBatch]) or keep it single-batch per call?
4. Any concerns about adding a protobuf build dependency to bindings/c/?

---

We're happy to implement this and contribute back. Looking forward to your feedback!

[QuakeWang]

@suxiaogang223 Thanks for proposing this. Adding write-side C FFI support is useful for Doris and other native engines, and the overall API looks reasonable.

For the commit message serialization format, @JingsongLi could you help decide whether we should keep compatibility with the existing Java CommitMessageSerializer or introduce a new format for Rust FFI?

For Arrow input, one RecordBatch per call should be enough. Please also consider commit_user and commit_identifier for distributed writes and retries.

Looking forward to the contribution.

[suxiaogang223]

I looked into how the Python bindings handle this, and turns out there is no serialization at all — everything is in-process. PyCommitMessage just wraps the Rust CommitMessage directly via pyo3, and the code even says "PR1 supports same-process transfer only (no pickle/serialization)". On the storage side, manifests use Apache Avro and snapshots use JSON, no protobuf anywhere.

So I think we can just follow the same pattern for the C FFI: keep it in-process for now, no serialization needed. If cross-process transfer becomes a real requirement later, we can tackle it as a separate enhancement. This should not block the current write-side C FFI work.