Skip to content

feat(datafusion): fill omitted INSERT columns with Iceberg write-default values#2804

Open
moomindani wants to merge 5 commits into
apache:mainfrom
moomindani:datafusion-write-default
Open

feat(datafusion): fill omitted INSERT columns with Iceberg write-default values#2804
moomindani wants to merge 5 commits into
apache:mainfrom
moomindani:datafusion-write-default

Conversation

@moomindani

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

Per the spec, writers must use a column's write-default for columns they do not supply. DataFusion's insert planner consults TableProvider::get_column_default for columns omitted from an INSERT and falls back to NULL; IcebergTableProvider did not implement it, so tables with write-default values silently got NULLs.

  • IcebergTableProvider caches the schema's top-level write-default values as DataFusion expressions at construction and serves them via get_column_default.
  • Adds a literal_to_scalar_value conversion covering the primitive types (boolean, int, long, float, double, string, date, time, timestamp/timestamptz in µs and ns, decimal, binary, fixed, uuid); defaults with no scalar representation are skipped. The planner casts the expression to the target arrow type, so representation differences are reconciled downstream.
  • This mirrors iceberg-java's approach of delegating write-default application to the engine: its Spark integration exposes write-default as the engine column default (TypeToSparkType), and Spark materializes it at INSERT planning.
  • The static provider is read-only and keeps the trait default (no column defaults).

Are these changes tested?

Yes — an end-to-end test (INSERT omitting defaulted columns, then scanning to assert the defaults land and explicitly provided values win) plus unit tests for the literal conversion. The end-to-end test fails without the get_column_default implementation. cargo test -p iceberg-datafusion --lib (90 tests) and cargo clippy -p iceberg-datafusion --lib --tests pass locally.

This pull request and its description were written by Claude Fable 5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataFusion INSERT fills omitted columns with NULL instead of the column's write-default

1 participant