Add FixedSizeList support for recursive struct schema adaptation by kosiew · Pull Request #22980 · apache/datafusion

kosiew · 2026-06-16T11:48:44Z

Which issue does this PR close?

Part of Support additive schema evolution for List<Struct> / nested container types in Parquet scans #20835

Rationale for this change

FixedSizeList containing Struct values was not handled by the existing recursive nested adaptation logic used for schema evolution. As a result, planner-time compatibility checks, nested cast detection, and runtime casting did not support additive struct evolution within FixedSizeList containers.

This change adds FixedSizeList support and verifies planner/runtime parity so that planning allows exactly the cases runtime can adapt while continuing to reject incompatible schema changes.

What changes are included in this PR?

Extend cast_column to support recursive casting of FixedSizeList values when source and target list sizes match.
Add FixedSizeList handling to:
- requires_nested_struct_cast
- validate_data_type_compatibility
Implement recursive casting of nested Struct values contained in FixedSizeList.
Preserve planner/runtime parity by validating child type compatibility before runtime fallback logic is applied.
Add handling for null-parent FixedSizeList entries by masking hidden child values before retrying casts, avoiding failures caused by semantically inaccessible child data.
Refactor list and list-view casting helpers to use Arrow AsArray accessors.

Are these changes tested?

Yes.

The following tests were added:

test_cast_fixed_size_list_struct
test_validate_fixed_size_list_struct_compatibility
test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected
test_validate_fixed_size_list_struct_size_mismatch_rejected
test_cast_fixed_size_list_struct_all_null
test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type
test_cast_fixed_size_list_struct_missing_non_nullable_field_runtime_rejected
test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent

Existing coverage in test_requires_nested_struct_cast was also extended to include FixedSizeList cases.

These tests cover:

Additive nullable nested-field evolution
All-null and partially null list cases
Incompatible nested type changes
Non-nullable field addition rejection
Planner/runtime parity validation

Are there any user-facing changes?

No user-facing changes. This is an internal enhancement to nested schema adaptation and casting behavior for FixedSizeList<Struct> types.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.

…untime parity - Updated runtime casting to recognize FixedSizeList when source and target list sizes match, allowing recursive adaptation of the child struct via a new helper. - Extended planner/runtime parity by updating `validate_data_type_compatibility` and `requires_nested_struct_cast` to handle equal-size FixedSizeList children. test: add regression coverage for nested field changes - Added tests for: - Additive nullable nested-field evolution - Planner acceptance for compatible cases - Non-nullable added-field rejection - All-null column handling - Planner/runtime parity on incompatible nested type changes

- Introduced `downcast_array!` macro to simplify downcasting of array types and improve error handling with descriptive messages. - Refactored existing functions `cast_list_column`, `cast_list_view_column`, and `cast_fixed_size_list_column` to utilize the new macro for improved readability and maintainability. - Added helper function `create_fsl_test_fields` to streamline test field creation for fixed-size list struct tests. - Updated related tests to use the new helper function for improved clarity and reduced redundancy.

- Prevent validation from accepting unsupported length changes - Ensure planner behavior aligns with runtime casting - Add regression test for rejected mismatch case

…t_fields for clarity

- Added regression test for null FixedSizeList parent hiding invalid child values. - Implemented runtime retry mechanism for child slots masked as null when cast fails due to hidden null-parent values. - Added runtime parity test to ensure missing non-nullable nested fields are still rejected.

- Removed redundant FixedSizeList size-mismatch match arm - Bound repeated source_values and target_type - Renamed masked fallback local for better readability - Inlined single-use mask wrapper - Added test helpers to reduce duplication in tests - Reused fields in parity test for efficiency - Reused s1 and s2 in requires_nested_struct_cast test to minimize duplication

Jefffrey · 2026-06-17T04:49:44Z

+    let source_values = source_list.values();
+    let target_type = target_inner_field.data_type();
+
+    validate_data_type_compatibility(


The other cast_*_column methods don't do this validate call, is this significant?

This fixed-size-list path differs from the list/list-view paths because it has a retry that masks child values hidden behind null parent list slots. The schema compatibility check ensures that retry is only used for value-level failures in hidden child positions, not to accidentally relax planner/runtime type compatibility. The parity tests for incompatible nested type and missing non-nullable fields cover this. I’ll add a comment or restructure the code so this guard is explicit.

I still don't think we need to compatibility check again here on this path, as we're just meant to be casting at this point 🤔

Amended:

Removed fixed-size-list compatibility check from the normal cast path.

Moved compatibility guard into the null-parent retry path only.

Adjusted cast_struct_column all-null fast path to follow struct compatibility validation, ensuring all-null structs reject missing non-nullable fields.

Jefffrey · 2026-06-17T04:51:31Z

+    let cast_values = match cast_column(source_values, target_type, cast_options) {
+        Ok(cast_values) => cast_values,
+        Err(error) => match source_list.nulls() {
+            Some(parent_nulls) if parent_nulls.null_count() > 0 => {
+                let hidden_child_nulls = parent_nulls.expand(target_list_size as usize);
+                let masked_values =


I dont understand whats happening here; why do we have this fallback if we fail to cast?

The fallback is intentional. FixedSizeListArray stores len * list_size child values even for null parent list entries, so recursive casting can fail on child values that are semantically hidden by a null parent. The fallback expands the parent null bitmap to child positions, masks those hidden child values to null, and retries; visible child values are still cast normally. test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent covers this case. I’ll add a targeted comment and keep the fallback limited to this null-parent masking case.

This is interesting; it technically applies for the other list methods too but those would be rare cases where the null slots are non-empty (and have values that would error on cast), not to mention this would only occur when cast options has safe=false

Also I think this can also occur in the arrow-rs upstream cast code, so might need to raise an issue there too 🤔

Created apache/arrow-rs#10233

…_array! - Removed local `downcast_array!` implementation - Used Arrow methods: - `source_col.as_list::<O>()` - `source_col.as_list_view::<O>()` - `source_col.as_fixed_size_list()` - Added `AsArray` import

- Added comment before `validate_data_type_compatibility` to explain planner/runtime guard. - Restructured fallback into an explicit helper: - Only attempts masked retry when parent nulls are present. - Returns original cast error in all other cases. - Clarified the split between guard and fallback logic.

… behavior - Explain hidden child slots handling for null parents - Clarify that retries occur only for parent-null masking - State that behavior remains unchanged for scenarios without parent nulls, retaining original cast error

…ing logic - Removed fixed-size-list compatibility check from the normal cast path. - Moved compatibility guard into the null-parent retry path only. - Adjusted cast_struct_column all-null fast path to follow struct compatibility validation, ensuring all-null structs reject missing non-nullable fields.

TheBuilderJR · 2026-07-21T14:28:46Z

Hey @alamb @Jefffrey, could we get this merged? This is still affecting my users in a pretty pathological way. Right now I have to rewrite the data just to make the query work.

alamb · 2026-07-22T22:25:19Z

Hi @TheBuilderJR -- I am confused about what this PR is fixing / adding -- it clearly has a bunch of unit tests, but I am missing the bigger picture -- what type of plan / SQL / DataFrame are you running on what type of input that is not working?

I think it would help to have a fuller end to end test of what you are doing (e.g. is it reading some nested structure from parquet files and they are getting confused)

Otherwise I worry that we won't be able ot maintain this code as it is refactored over time as if we change how it is structured we won't know if we broke the original usecase

codecov-commenter · 2026-07-22T22:49:55Z

Codecov Report

❌ Patch coverage is 79.35979% with 619 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.72%. Comparing base (c118002) to head (39ce782).
⚠️ Report is 31 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto/src/physical_plan/mod.rs	9.54%	218 Missing ⚠️
...ysical-plan/src/aggregates/ordered_final_stream.rs	75.93%	82 Missing and 8 partials ⚠️
...ion/physical-plan/src/joins/symmetric_hash_join.rs	82.84%	18 Missing and 23 partials ⚠️
datafusion/physical-plan/src/joins/cross_join.rs	0.00%	34 Missing ⚠️
datafusion/physical-plan/src/repartition/mod.rs	77.03%	14 Missing and 17 partials ⚠️
datafusion/session/src/catalog.rs	56.52%	20 Missing ⚠️
datafusion/physical-plan/src/sorts/sort.rs	77.92%	6 Missing and 11 partials ⚠️
datafusion/session/src/table.rs	85.47%	17 Missing ⚠️
datafusion/physical-plan/src/joins/proto.rs	85.32%	7 Missing and 9 partials ⚠️
...ical-plan/src/aggregates/ordered_partial_stream.rs	73.68%	10 Missing and 5 partials ⚠️
... and 27 more

Additional details and impacted files

@@            Coverage Diff            @@
##             main   #22980     +/-   ##
=========================================
  Coverage   80.72%   80.72%             
=========================================
  Files        1089     1090      +1     
  Lines      368911   370385   +1474     
  Branches   368911   370385   +1474     
=========================================
+ Hits       297785   298982   +1197     
- Misses      53374    53598    +224     
- Partials    17752    17805     +53

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

TheBuilderJR · 2026-07-22T22:54:34Z

@alamb my high level goal is to get #20835 fixed. @kosiew can you advise here?

Jefffrey · 2026-07-23T05:31:47Z

+    // Before masking and retrying, guard schema compatibility so the retry only
+    // handles value-level failures in hidden child slots and cannot make runtime
+    // accept schemas that planning rejects.
+    if let Err(error) = validate_data_type_compatibility(


i still dont agree with having this validate call on the cast path

we dont do anything similar for other list types even though they could technically contain invalid data (that cant be casted) in a non-empty null slot

Removed the validation.

…bility validation and field_name plumbing - Removed retry-path compatibility validation. - Eliminated field_name plumbing. - Retained recursive cast enforcement.

- Implemented old/new schema evolution via ListingTable - Added support for SELECT * - Enabled nested-field projection - Implemented rejection for missing non-nullable fields - Added rejection for incompatible fields - Corrected existing negative fixture to truly omit chain

- Added constructor-error regression test. - Extended hidden-child regression to utilize sliced FixedSizeList input.

test: enhance size-mismatch test to verify compatibility validation rejection, runtime cast_column rejection, and width-specific Arrow diagnostic

…ion into schema-evolution-02-20835

kosiew · 2026-07-24T09:04:53Z

@alamb

I think it would help to have a fuller end to end test of what you are doing (e.g. is it reading some nested structure from parquet files and they are getting confused)

Added e2e tests.

kosiew · 2026-07-25T10:38:57Z

thanks @Jefffrey, @alamb for your reviews and feedback

…che#22980) ## Which issue does this PR close? * Part of apache#20835 ## Rationale for this change `FixedSizeList` containing `Struct` values was not handled by the existing recursive nested adaptation logic used for schema evolution. As a result, planner-time compatibility checks, nested cast detection, and runtime casting did not support additive struct evolution within `FixedSizeList` containers. This change adds `FixedSizeList` support and verifies planner/runtime parity so that planning allows exactly the cases runtime can adapt while continuing to reject incompatible schema changes. ## What changes are included in this PR? * Extend `cast_column` to support recursive casting of `FixedSizeList` values when source and target list sizes match. * Add `FixedSizeList` handling to: * `requires_nested_struct_cast` * `validate_data_type_compatibility` * Implement recursive casting of nested `Struct` values contained in `FixedSizeList`. * Preserve planner/runtime parity by validating child type compatibility before runtime fallback logic is applied. * Add handling for null-parent `FixedSizeList` entries by masking hidden child values before retrying casts, avoiding failures caused by semantically inaccessible child data. * Refactor list and list-view casting helpers to use Arrow `AsArray` accessors. ## Are these changes tested? Yes. The following tests were added: * `test_cast_fixed_size_list_struct` * `test_validate_fixed_size_list_struct_compatibility` * `test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected` * `test_validate_fixed_size_list_struct_size_mismatch_rejected` * `test_cast_fixed_size_list_struct_all_null` * `test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type` * `test_cast_fixed_size_list_struct_missing_non_nullable_field_runtime_rejected` * `test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent` Existing coverage in `test_requires_nested_struct_cast` was also extended to include `FixedSizeList` cases. These tests cover: * Additive nullable nested-field evolution * All-null and partially null list cases * Incompatible nested type changes * Non-nullable field addition rejection * Planner/runtime parity validation ## Are there any user-facing changes? No user-facing changes. This is an internal enhancement to nested schema adaptation and casting behavior for `FixedSizeList<Struct>` types. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

kosiew added 3 commits June 16, 2026 19:41

fix: close FixedSizeList size-mismatch loophole in nested_struct.rs

62c5adf

- Prevent validation from accepting unsupported length changes - Ensure planner behavior aligns with runtime casting - Add regression test for rejected mismatch case

github-actions Bot added the common Related to common crate label Jun 16, 2026

kosiew added 3 commits June 16, 2026 19:57

refactor: rename create_fsl_test_fields to create_fixed_size_list_tes…

b561ef9

…t_fields for clarity

kosiew marked this pull request as ready for review June 16, 2026 15:27

Jefffrey reviewed Jun 17, 2026

View reviewed changes

kosiew added 3 commits June 18, 2026 14:08

feat: refactor nested_struct to use Arrow methods and remove downcast…

d8c1b17

…_array! - Removed local `downcast_array!` implementation - Used Arrow methods: - `source_col.as_list::<O>()` - `source_col.as_list_view::<O>()` - `source_col.as_fixed_size_list()` - Added `AsArray` import

kosiew mentioned this pull request Jun 29, 2026

Parent-null-aware casting for nested arrays with hidden child values apache/arrow-rs#10233

Open

5 tasks

Merge branch 'main' into schema-evolution-02-20835

dd00ef9

Jefffrey reviewed Jul 23, 2026

View reviewed changes

kosiew marked this pull request as draft July 24, 2026 08:20

kosiew added 6 commits July 24, 2026 16:21

Merge branch 'main' into schema-evolution-02-20835

d943826

feat(datafusion): simplify nested struct handling by removing compati…

96ab349

…bility validation and field_name plumbing - Removed retry-path compatibility validation. - Eliminated field_name plumbing. - Retained recursive cast enforcement.

feat: replace panicking FixedSizeListArray::new with fallible try_new

6600725

- Added constructor-error regression test. - Extended hidden-child regression to utilize sliced FixedSizeList input.

docs: add mention of equal-width FixedSizeList

8b0c478

test: enhance size-mismatch test to verify compatibility validation rejection, runtime cast_column rejection, and width-specific Arrow diagnostic

Merge branch 'schema-evolution-02-20835' of github.com:kosiew/datafus…

39ce782

…ion into schema-evolution-02-20835

github-actions Bot added the core Core DataFusion crate label Jul 24, 2026

kosiew marked this pull request as ready for review July 24, 2026 09:11

Jefffrey approved these changes Jul 24, 2026

View reviewed changes

kosiew added this pull request to the merge queue Jul 25, 2026

Merged via the queue into apache:main with commit f1ab86d Jul 25, 2026
37 checks passed

kosiew deleted the schema-evolution-02-20835 branch July 25, 2026 10:52

Uh oh!

Conversation

kosiew commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

LLM-generated code disclosure

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheBuilderJR commented Jul 21, 2026

Uh oh!

alamb commented Jul 22, 2026

Uh oh!

codecov-commenter commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

TheBuilderJR commented Jul 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew commented Jul 24, 2026

Uh oh!

kosiew commented Jul 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kosiew commented Jun 16, 2026 •

edited

Loading

kosiew Jun 29, 2026 •

edited

Loading

codecov-commenter commented Jul 22, 2026 •

edited

Loading