fix(deps): update datafusion ecosystem

[renovate]

This PR contains the following updates:

Package	Type	Update	Change
datafusion (source)	dependencies	major	50.3 → 54.0
deltalake	dependencies	minor	0.29.4 → 0.32.0
object_store	dependencies	minor	0.12 → 0.14

---

Release Notes

apache/datafusion (datafusion)

v54.1.0

Compare Source

v54.0.0

Compare Source

v53.1.0

Compare Source

v53.0.0

Compare Source

v52.5.0

Compare Source

v52.4.0

Compare Source

v52.3.0

Compare Source

v52.2.0

Compare Source

v52.1.0

Compare Source

v52.0.0

Compare Source

v51.0.0

Compare Source

delta-io/delta.rs (deltalake)

v0.32.3

Full Changelog

Merged pull requests:

• docs: update the changelog for 0.32.2 #​4459 (rtyler)
• feat: add support for variant type #​4325 (abhiaagarwal)

v0.32.2

Full Changelog

Closed issues:

• [Bug]: DynamoDB setup instruction in docs #​4452
• [Feature]: Support passing datatypes other than string in custom_metadata in CommitProperties from python #​4443
• [Bug]: can't change partition column by overwriting table #​4434
• [Bug]: Python 1.5.1 MERGE regression: small source batch uses substantially more RSS than 1.5.0 #​4422
• [Bug]: python import deltalake crashes on linux kerner with 64KB page size #​4413
• [Bug]: Clarification (or regression) dataSkippingNumIndexedCols #​4398

Merged pull requests:

• refactor: deprecate legacy DeltaScan codec surface #​4458 (ethan-tyler)
• docs: update dynamodb locking setup #​4456 (fizyxbt)
• fix: normalize commit metadata handling #​4455 (ethan-tyler)
• fix: restore merge safe cast behavior #​4454 (ethan-tyler)
• chore: adopt the released buoyant_kernel version #​4453 (rtyler)
• test(core): consolidate integration tests into two binaries #​4451 (rtyler)
• fix(core): add checked RecordBatchWriter constructor #​4450 (fksegundo)
• fix(python): gate jemallocator off on FreeBSD (fall back to system allocator) #​4449 (sahuagin)
• fix: support Windows UNC paths #​4447 (rtyler)
• chore: clippy and dependency API fixes #​4446 (rtyler)
• chore: upgrade to the buoyant_kernel 0.22 #​4445 (rtyler)
• fix: allow partition column changes on schema overwrite #​4444 (ethan-tyler)
• refactor: remove DeltaTableProvider #​4435 (ethan-tyler)
• test: port DeltaTableProvider tests to next provider #​4432 (ethan-tyler)
• refactor: clean up row index scan contract #​4431 (ethan-tyler)
• chore: clarify cached stats arrow fields #​4429 (ethan-tyler)
• fix: reduce duplicate validation memory #​4428 (ethan-tyler)
• fix(core): reject unsupported column mapping writes #​4424 (fksegundo)
• refactor: selective stats materialization policy #​4421 (ethan-tyler)
• fix: local file table root identity #​4419 (ethan-tyler)
• ci: build wheels with 64k pages #​4418 (rtyler)
• chore: upgrade to the 200 series of buoyant_kernel #​4416 (rtyler)
• chore: update the changelog and prepare for the next patch release #​4415 (rtyler)
• fix: exclude partition columns from add stats schema #​4408 (ethan-tyler)
• feat: nanosecond timestamps support, gated by a Cargo feature #​4370 (itamarst)
• fix: propagate schema_force_view_types through merge write projection #​4302 (sheganinans)

v0.32.1

Full Changelog

Closed issues:

• [Bug]: Regression in 1.5.1: schema_mode='overwrite' with predicate fails for mixed-case column names (utcDate) #​4404
• [Bug]: update_incremental() and transaction_version() hold the GIL for the entire S3 round-trip #​4402
• [Feature]: Build Musl arm64 wheels #​4392
• [Bug]: deltalake_catalog_unity failes to list catalogs #​4360
• [Bug]: UnityCatalogFactory::parse_url_opts only allows unity-specific options #​4225

Merged pull requests:

• chore: cap the upper range of the buoyant_kernel for this release #​4412 (rtyler)
• fix(catalog-unity): skip non-Unity keys in try_with_options #​4410 (1fanwang)
• fix: listing catalogs from Unity #​4409 (hjohnss6)
• fix: preserve mixed case columns in replaceWhere writes #​4407 (ethan-tyler)
• fix: release GIL for DeltaTable update calls #​4406 (ethan-tyler)
• chore: add tests for cleanup_expired_logs_for #​4405 (mbkroese)
• refactor: introduce selective stats projection primitives #​4403 (ethan-tyler)
• chore: regenerate updated changelog #​4400 (rtyler)
• feat: add aarch64 musl wheel builds #​4399 (Slyrc)
• chore: next iteration of development #​4396 (rtyler)
• fix: allow boolean to exist in our stats schema #​4395 (rtyler)
• chore: bump python for next release #​4393 (rtyler)

v0.32.0

Full Changelog

Implemented enhancements:

• feature: Add post_commithook_properties to DeltaTable.restore #​4251
• Create deltalake-engine crate based off arrow engine #​4133

Fixed bugs:

• [Bug]: DeltaScan ignores DeltaScanConfig::schema unlike old DeltaTableProvider #​4306
• bug: Streamed merge loses file pruning for string partition IN (...) predicates #​4301

Closed issues:

• [Bug]: append fails when generated column uses TRUNC #​4386
• [Bug]: num_deleted_rows is 0 for partition-only DELETEs #​4379
• [Bug]: DeltaTable.to_pyarrow_dataset() hangs indefinitely with moto S3 mock (regression in 1.3.2) #​4362
• [Bug]: Number of deleted rows incorrect when table had "enableChageDataFeed" set to true #​4353
• [Bug]: MERGE on partitioned table produces duplicate rows when partition value contains a space #​4352
• Datafusion/Arrow/Delta Kernel Upgrade #​4344
• [Bug]: intermittent S3 integration test failures in CI #​4332
• [Bug]: az:// URL scheme causes path inconsistency between log store and kernel, leading to 'version already exists' errors on Azure #​4320
• Documentation for Remove::data_change does not match spec #​4317
• [Bug]: Wrong log store registry for file:// schema #​4316
• [Bug]: Insertion Order not maintained when compacting #​4310
• [Bug]: DeltaScan scan includes file column although it's not projected #​4307
• [Bug]: Unable to write on GCP bucket in S3 compatibility mode #​4303
• [Bug]: merge_schema=True fails on first merge into fresh table partitioned by string column #​4298
• Deduplicate _last_checkpoint parsing in deltalake-core #​4294
• Inconsistent i64/u64 usage for defining versions #​4293
• [Bug]: 1.5.0 regression a partitioned Delta table with existing data and a merge into it with merge_schema: True #​4292
• [Bug]: Vacuuming FULL after compaction potentially removes data files preventing time travel #​4290
• [Bug]: Vacuuming with keep_versions requires versions to be in sorted order #​4286
• bug: same version snapshot update doesn't adopt checkpoint written later #​4280
• Re-use stored state in EagerSnapshot #​4269
• [Bug]: Datafusion not reading all partitions on Linux machine #​4268
• [Feature]: CommitData::new should respect "clientVersion" from app_metadata #​4263
• delta_kernel 0.20 upgrade #​4256
• [Feature]: add some changelog content to the docs #​4253
• feature: make commit control args keyword only across mutating APIs #​4252
• Passing cert file in python storage_options #​4246
• [Bug]: After lazy writes, writing large files fails #​4229
• [Bug]: deltalake get_add_actions not exposing bool min/max #​4224
• [Bug]: slack link in issue #​4214

v0.31.1

Full Changelog

Implemented enhancements:

• Use batched delete in DeltaStorageHandler.delete_dir #​4243
• [Feature]: refactor DeltaDataSink to go through delta writer builder #​4189

Fixed bugs:

• [Bug]: Read delta table from s3 directory bucket #​4136

Closed issues:

• [Bug]: update_incremental is not incremental #​4276
• [Bug]: pub fn create_add signature includes types that are not pub use #​4273
• [Feature]: Support for Writing to Delta Tables with Identity Columns in delta-rs #​4237
• [Bug]: Python delection_vectors() truncates bool list. #​4235
• [Feature]: Vacuum Lite to avoid unnecessary file listing #​4228
• [Feature]: Write incrementally to table when merging #​4217
• [Feature]: change existing table's enableChangeDataFeed configuration to be updated to True #​4216
• [Bug]: Contributing guide could be improved #​4213
• [Docs]: usage/appending-overwriting-delta-lake-table has broken language tabs #​4200
• [Bug]: s3-native-tls feature still brings aws-lc-rs dependency #​4170
• bug: generated column expr fails when SchemaMode::Merge would add referenced column #​4169
• [Bug]: create_write_transaction no longer overwrites partitions #​4126

v0.31.0

Full Changelog

Implemented enhancements:

• update_datafusion_session has the wrong order of arguments #​4139

Closed issues:

• [Bug]: write_deltalake with schema_mode="merge" drops generated column metadata #​4186
• [Bug]: is_deltatable creates paths for non existing delta table location #​4175
• [Bug]: Rust CdfLoadBuilder.build function is crate private and prohibits using feature #​4160
• Expose deletion vectors information to Python #​4159
• [Bug]: polars.DataFrame.write_delta() failing for unknown argument 'schema' #​4157
• [Bug]: Delete regression in 1.4.0: partition‑only predicate removes too few files #​4149
• [Bug]: Unity Catalog Azure credentials error #​4129

Merged pull requests:

• chore: enable snappy compression on checkpoints #​4209 (rtyler)
• chore: set compression for partition optimization as well #​4208 (rtyler)
• chore: change the versions for the next "majorish" release of 🦀 #​4207 (rtyler)
• fix: enforce file-id filter semantics in scan planning #​4206 (ethan-tyler)
• refactor(python): get add action return arrow table #​4204 (vsmanish1772)
• fix: propagate session config through Delta factory path #​4202 (ethan-tyler)
• chore: drop unused ReceiverStreamBuilder spawn #​4201 (ethan-tyler)
• chore: harden scan adapter caching and DV mask edge cases #​4199 (ethan-tyler)
• fix(datafusion): avoid overflow when scanning add actions #​4197 (vsmanish1772)
• fix: clarify vacuum command documentation for DeltaTable #​4196 (khalidmammadov)
• refactor: use BatchAdapterFactory for scan adaptation #​4195 (ethan-tyler)
• perf: parallel partition writers via per-stream JoinSet #​4193 (fvaleye)
• fix: make session the first argument of update_datafusion_session #​4192 (pauldouane)
• fix: preserve generated column metadata during schema merge #​4191 (ethan-tyler)
• refactor: use FileSelection for matched file scans #​4188 (ethan-tyler)
• feat: add DeltaScan insert_into with runtime log_store #​4187 (ethan-tyler)
• chore: bump version from 1.4.1 to 1.4.2 #​4182 (ion-elgreco)
• fix: is_deltatable must not to create paths for not existing tables paths #​4176 (khalidmammadov)
• refactor: remove table level stats on TableProvider #​4174 (roeap)
• docs: add storage backend configuration reference tables #​4173 (immohamedadhil)
• refactor: add FileSelection to next provider for DeltaTableProvider removal #​4172 (ethan-tyler)
• feat: expose DV metadata and payloads as Arrow streams #​4168 (ethan-tyler)
• fix: align file_id with DataFusion UInt16 dictionary #​4167 (ethan-tyler)
• chore: add the unity catalog dependency back #​4165 (hntd187)
• fix: preserve kernel column segments #​4164 (ethan-tyler)
• chore: use default-https-client for aws sdk to avoid deps on hyper 0.14 #​4163 (BugenZhao)
• fix: unblock schema merge appends with generated columns #​4162 (ethan-tyler)
• fix(cdf): make cdf builders build function accessible #​4161 (khalidmammadov)
• fix(docs): open_table examples to use URLs instead of str #​4154 (khalidmammadov)
• perf: cache schema per stream instead of per batch #​4152 (fvaleye)
• refactor: avoid mutable updates of inner snapshot #​4151 (roeap)
• fix(delete): use Add metadata for partition only DELETE #​4150 (ethan-tyler)
• fix: nested runtimes in stream adapter #​4148 (ion-elgreco)
• feat: session-first DataFusion integration + session resolution policies #​4145 (ethan-tyler)
• fix: align stats to view typed schema + harden parquet predicate pushdown #​4144 (ethan-tyler)
• fix(python): guard DataFusion FFI export on datafusion major version #​4142 (ethan-tyler)
• chore: upgrade python version for a patch release #​4141 (rtyler)
• feat: update asserted nullability in DataValidation output schema #​4132 (roeap)
• docs: incorporate some AI guidance for contributors #​4131 (rtyler)
• fix(core): align file stats with parquet read schema #​4130 (roeap)
• fix(datafusion): resolve DML predicates against execution scan schema #​4127 (ethan-tyler)
• chore: upgrade python version for the next release #​4124 (rtyler)
• chore: allow integration tests to be run in parallel with nextest #​4122 (rtyler)
• chore: upgrade azurite and purge the need for a local az CLI to run tests #​4121 (rtyler)
• feat: migrate delete #​4117 (roeap)

v0.30.2

Full Changelog

Fixed bugs:

• [Bug]: DeltaTable.delete fails during equality comparison with a string #​4125

Closed issues:

• [Bug]: write_deltalake errors on missing nullable columns without generation expressions #​4156
• Bug Report: Nested Tokio Runtime Panic in write_deltalake #​4147
• [Bug]: Segmentation fault when registering table to datafusion session #​4135
• [Bug]: slack link no longer active #​4118

v0.30.1

Full Changelog

v0.30.0

⚠️ There are a number of API changes between 0.30.x and 0.29.4, but we expect to see better performance as a result!

Full Changelog

Merged pull requests:

• refactor: remove log_data call sites in find_files #​4026 (roeap)
• chore: remove wildcard dependency for publishing #​4025 (rtyler)
• refactor: use logical type ref when getting stats #​4019 (roeap)
• fix: handle stats config in data sink #​4016 (roeap)
• fix: null handling when extracting scalars #​4014 (roeap)
• fix: between range handling in expression translations #​4013 (roeap)
• chore: fix windows uri test #​4011 (hntd187)
• refactor: towards lazier snapshots #​4010 (roeap)
• fix: pin pyspark and clear disk space in runners #​4007 (ion-elgreco)
• test: add utilities for asserting DAT scan results #​4005 (roeap)
• chore: update delta-kernel to 0.19 #​4004 (roeap)
• refactor: simplify kernel extensions #​4003 (roeap)
• chore: clippy #​4002 (roeap)
• refactor: handle target version when resolving snapshot #​4001 (roeap)
• refactor: use rstest for running DAT tests #​4000 (roeap)
• feat: kernel expression conversion #​3998 (roeap)
• chore: add easier local coverage reporting #​3995 (rtyler)
• feat: expose operations on DeltaTable #​3987 (roeap)
• chore: remove some warnigs #​3986 (roeap)
• chore: normalize Url going into logstore and update everything to take references #​3985 (rtyler)
• fix: add missing field to snapshot serde #​3984 (roeap)
• feat: allow for concurrent deletes in conflict checker if data_change is false #​3982 (abhiaagarwal)
• fix: remove 3.9 from ci matrix #​3978 (ion-elgreco)
• fix: decode path before lookup #​3976 (ion-elgreco)
• chore: remove deprecated pyo3 methods #​3975 (ion-elgreco)
• chore: removing APIs and deprecation warnings: 0.30.x here we come #​3962 (rtyler)
• feat: update to DataFusion 51, arrow 57, delta-kernel 0.18.0, pyo3 26, pyo3-arrow 0.14 #​3949 (hntd187)
• fix: schema evolution for merge operation #​3945 (JustinRush80)
• chore: remove Python 3.9 from our build infrastructure #​3937 (rtyler)
• docs: fix small typo issue #​3935 (bmoreau8)
• chore: removing references to using partition_filters for partition overwrite #​3912 (zyd14)
• feat(datafusion): add max_temp_directory_size parameter for z-order and compact operations for DataFusion #​3847 (fvaleye)

Fixed bugs:

• Asked to increase max_temp_directory_size in the disk manager configuration when optimizing large table #​3833

Closed issues:

• [Bug]: Count / get_add_actions exception for an empty table #​4023
• [Bug]: MERGE with schema evolution does not add new columns #​4009
• [Bug]: vacuum does not respect retention_hours when full=True #​3989
• [Bug]: write table by FFI call from go may memory leak? #​3973
• [Bug]: Table merging fails with merge_schema=True #​3943
• [Bug]: _internal.DeltaError: Generic DeltaTable error: Unable to map __delta_rs_path to action during overwrite with predicate #​3939
• [Feature]: update to DataFusion 51.0.0 #​3920
• [Bug]: get_add_actions() panics with "index out of bounds" when table has no data files #​3918
• [Bug]: Docs describe partition_filters parameter to write_deltalake that doesn't exist #​3904
• [Feature]: split delta-rs into multiple crates #​3899
• [Feature]: Drop python 3.9 support once EOL #​3886
• [Bug]: PyPi storage limit hit for deltalake [python releases blocked for time-being] #​3876

apache/arrow-rs-object-store (object_store)

v0.14.1

Compare Source

Full Changelog

Implemented enhancements:

• Cache Azure user delegation key for SAS signing #​800
• Add deregister to ObjectStoreRegistry #​795
• Support for S3 dual-stack endpoints #​779
• Coalesce contiguous ranges in LocalFileSystem::get_ranges #​778
• Support for HF Storage Buckets #​706

Fixed bugs:

• Clippy failing on main #​798
• Upgrade quick-xml to resolve RUSTSEC-2026-0194 and RUSTSEC-2026-0195 #​786
• InMemory and LocalFileSystem return unquoted ETags, violating RFC 9110 #​769

Closed issues:

• Bump quick-xml to 0.41.0 for RustSec advisories #​787

Merged pull requests:

• Pin Rust toolchain to 1.97.0 and fix clippy #​799 (alamb)
• chore: reduce lock hold time in DefaultObjectStoreRegistry::deregister #​792 (kszucs)
• Do not explicitly add Host header when making requests with S3Builder #​790 (ntjohnson1)
• impl HttpError::new_boxed to provide user with ability to ctor from already boxed error #​789 (DoumanAsh)
• build(deps): update quick-xml requirement from 0.40.1 to 0.41.0 #​785 (dependabot[bot])
• feat: add deregister to ObjectStoreRegistry #​784 (kszucs)
• Cache Azure user delegation key for SAS signing #​781 (emilk)
• fix(aws): recognize S3 dual-stack endpoints in URL parsing #​780 (a10y)
• Coalesce contiguous ranges in LocalFileSystem::get_ranges #​777 (Dandandan)
• Update copyright year in NOTICE.txt #​772 (kevinjqliu)
• fix: quote ETags per RFC 9110 in InMemory and LocalFileSystem #​770 (bidord)
• build(deps): bump actions/checkout from 6 to 7 #​768 (dependabot[bot])

* This Changelog was automatically generated by github_changelog_generator

v0.14.0

Compare Source

v0.13.2

Compare Source

v0.13.1

Compare Source

v0.13.0

Compare Source

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Enabled.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.