fix(connectors): iceberg sink writes real partitions on iceberg 0.10 - #3969
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3969 +/- ##
============================================
- Coverage 85.00% 84.99% -0.02%
Complexity 1402 1402
============================================
Files 1225 1226 +1
Lines 180283 180701 +418
Branches 146587 147005 +418
============================================
+ Hits 153248 153583 +335
- Misses 22993 23042 +49
- Partials 4042 4076 +34
🚀 New features to boost your workflow:
|
The Iceberg sink built its partition key from an empty PartitionSpec::builder() with hardcoded zero literals, so every record landed in one dummy partition (e.g. year=1970) whatever the table's spec said, and a batch spanning several partitions was never split. Read the default partition spec from table metadata instead and push every batch through one TableWriter: partitioned tables split it with RecordBatchPartitionSplitter and fan out to one data file writer per partition value, unpartitioned tables map it to the single key of their spec (with one null per void field, which the commit-time arity check requires). Files finalized by a failed batch are deleted before the error is returned so nothing is left orphaned on the object store; files are kept on commit failure because the catalog may already have applied the commit. Ports #3860 with its review feedback applied. Fixes #3853
The workspace pinned arrow/parquet 57 while deltalake pulled 58, so the lock carried two copies of the whole arrow family (16 crates) plus parquet. iceberg 0.9 was the last consumer holding us on 57. Move to arrow/parquet 58.4 and iceberg 0.10.1 so every consumer shares one arrow. iceberg 0.10 drops configured_scheme from the S3 storage factory, borrows table metadata in DefaultLocationGenerator, requires an explicit Runtime when building a Table, and switches the S3 default to virtual-host-style addressing, so the sink now pins s3.path-style-access=true to keep MinIO-style endpoints working. thrift 0.17 (GHSA-2f9f-gq7v-9h6m) stays until parquet 59 lands in iceberg-rust and deltalake; parquet only uses its output protocol.
All other sink crates set publish = false; clickhouse was the only one missing it.
hubcio
force-pushed
the
fix/iceberg-partition-routing
branch
from
September 1, 2026 11:58
4aaa0d3 to
3593e95
Compare
iceberg-storage-opendal 0.10 switched the S3 default to virtual-host addressing, and the sink pinned path-style back on with no way to turn it off, which breaks stores that only accept virtual-hosted URLs. Add store_path_style_access (default true so MinIO setups keep working) and pass it through to the file IO. A batch without JSON payloads (stream schema other than json) reached the catalog as an empty append, which iceberg rejects with an opaque PreconditionFailed that stops the consumer. Fail before any storage I/O with InvalidPayloadType so the log names the misconfiguration. Also drop the per-batch PartitionKey and Struct clones in the partition writer lookup and the unused commit binding.
mmodzelewski
approved these changes
Sep 1, 2026
numinnex
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sink keyed every record on an empty PartitionSpec::builder()
with zero literals, so all rows landed in one dummy partition (e.g.
year=1970) regardless of the table's spec, and multi-partition
batches were never split.
Read the default partition spec from table metadata and route every
batch through one TableWriter: partitioned tables get one data file
writer per partition value, unpartitioned tables use their spec's
single key (one null per void field for the commit-time arity
check). Files from a failed batch are deleted before the error is
returned; commit failures keep them since the catalog may already
have applied the commit.
Also bumps to arrow/parquet 58.4 and iceberg 0.10.1 (one arrow
family in the lock), pins path-style S3 access (0.10 defaults to
virtual-host) and marks the clickhouse sink publish = false like
its siblings.
Ports #3860 with review feedback applied. Fixes #3853
Co-authored-by: Ashutosh Prajapati asprajapati@deqode.com