fix: unwrap identity Date cast in comparison unwrapping by adriangb · Pull Request #23727 · apache/datafusion

adriangb · 2026-07-20T18:33:48Z

Which issue does this PR close?

No existing issue; this PR both reports and fixes the bug. Happy to file a tracking issue if preferred.

Rationale for this change

unwrap_cast_in_comparison fails to fold an identity CAST(col AS DATE) on a Date32 column. Instead of rewriting the predicate to compare against the bare column, it leaves a residual Cast(col AS Date32) in place, which defeats downstream optimizations (pruning / filter pushdown) that expect a bare-column comparison.

Reproduction (logical plan for SELECT * FROM t WHERE cast(d AS date) = DATE '2024-01-01', where d is Date32): the cast(d AS Date32) survives simplification rather than collapsing to d. Note arrow_cast(d, 'Date32') already folds, because the arrow_cast UDF's simplify() short-circuits an identity cast; the SQL CAST ... AS date planner plants a real Expr::Cast with no such elision, so it reaches try_cast_literal_to_type and is wrongly rejected.

The root cause is in is_lossy_temporal_cast (datafusion/expr-common/src/casts.rs):

(is_date_type(from_type) && to_type.is_temporal())
    || (is_date_type(to_type) && from_type.is_temporal())

For an identity Date32 -> Date32 cast this evaluates to true && true, because DataType::is_temporal() is true for both Date32 and Date64. The identity cast is therefore misclassified as a lossy temporal cast, try_cast_literal_to_type returns None, and unwrap_cast_in_comparison leaves the cast in the plan.

What changes are included in this PR?

Short-circuit an identity cast as non-lossy at the top of is_lossy_temporal_cast:

if from_type == to_type {
    return false; // an identity cast never changes comparison semantics
}

This is deliberately limited to identical types, not "any date-to-date". Date32 counts days while Date64 counts milliseconds, but try_cast_numeric_literal uses mul = 1 for both, so allowing a Date32 <-> Date64 unwrap would convert units incorrectly. The from_type == to_type identity guard is the exact correct scope, and Date32 <-> Date64 remains blocked.

Are these changes tested?

Yes. The PR is structured as a test-driven, stacked sequence so the effect of the fix is visible in the diff:

test: characterize identity Date cast in comparison unwrapping — adds an SLT test to simplify_expr.slt (explain select d from dates where cast(d as date) = DATE '2024-01-01') whose assertion records the current, buggy output: the logical plan keeps the residual Filter: CAST(dates.d AS Date32) = Date32(...). Passes on main.
fix: unwrap identity Date cast in comparison unwrapping — the production change plus two expr-common unit tests:
- test_try_cast_identity_date_allowed — identity Date32 -> Date32 / Date64 -> Date64 now fold (try_cast_literal_to_type returns Some), and is_lossy_temporal_cast reports them non-lossy.
- test_try_cast_date32_date64_still_blocked — Date32 <-> Date64 stays lossy/blocked (guards the units caveat above, which SLT can't easily express).
At this commit the SLT test is intentionally red, proving it exercises the bug.
test: update identity Date cast assertion to folded plan — updates the SLT assertion to the corrected Filter: dates.d = Date32(...). The one-line diff is the observable effect of the fix.

cargo fmt --check and cargo clippy -D warnings are clean on the changed crate; datafusion-expr-common and the simplify_expr sqllogictest pass.

Are there any user-facing changes?

No API changes. Queries with CAST(date_col AS DATE) predicates on Date32 columns simplify to bare-column comparisons, which can enable additional pruning/pushdown. No behavioral change to query results.

codecov-commenter · 2026-07-20T19:03:49Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.75%. Comparing base (5b65e70) to head (1fe86c8).
⚠️ Report is 39 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23727      +/-   ##
==========================================
+ Coverage   80.70%   80.75%   +0.04%     
==========================================
  Files        1089     1089              
  Lines      368137   368857     +720     
  Branches   368137   368857     +720     
==========================================
+ Hits       297121   297860     +739     
+ Misses      53311    53246      -65     
- Partials    17705    17751      +46

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Add an SLT test to `simplify_expr.slt` for the predicate `cast(d AS date) = DATE '...'` where `d` is a `Date32` column. This is an identity cast that should fold away, leaving a bare-column comparison. This commit records the *current* (buggy) behavior: the logical plan retains a residual `CAST(dates.d AS Date32)` instead of simplifying to `dates.d = Date32(...)`. The residual cast defeats downstream pruning / filter pushdown that expects a bare-column comparison. The next commit fixes this and this test's assertion is updated to the corrected plan, making the fix's effect visible in the diff. Signed-off-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

`unwrap_cast_in_comparison` refused to fold an identity `CAST(col AS DATE)` on a `Date32` column, leaving a residual `Cast(col AS Date32)` in the predicate instead of comparing against the bare column. SQL `cast(col AS date)` plants a real `Expr::Cast` with no identity elision (unlike the `arrow_cast` UDF, whose `simplify()` short-circuits identity), so this cast reached `try_cast_literal_to_type` and was rejected. The residual cast defeats downstream pruning/pushdown that expects a bare-column comparison. The root cause is in `is_lossy_temporal_cast`: (is_date_type(from) && to.is_temporal()) || (is_date_type(to) && from.is_temporal()) For an identity `Date32 -> Date32` cast this is `true && true`, because `DataType::is_temporal()` is true for both `Date32` and `Date64`. The cast is misclassified as a lossy temporal cast, `try_cast_literal_to_type` returns `None`, and the residual cast is left in place. Fix: short-circuit an identity cast (`from_type == to_type`) as non-lossy at the top of `is_lossy_temporal_cast`. An identity cast can never change comparison semantics. This is deliberately scoped to *identical* types only, not "any date-to-date". `Date32` counts days while `Date64` counts milliseconds, but `try_cast_numeric_literal` uses `mul = 1` for both, so a `Date32 <-> Date64` cast would convert units wrongly and must stay blocked. A regression test asserts that. The SLT characterization test added in the previous commit now fails (its assertion still shows the residual cast); the next commit updates it to the corrected plan, so the diff shows exactly what the fix changed. Signed-off-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

With the identity short-circuit in `is_lossy_temporal_cast`, the predicate `cast(d AS date) = DATE '...'` on a `Date32` column now folds the identity cast away. Update the SLT assertion from the residual Filter: CAST(dates.d AS Date32) = Date32("2024-01-01") to the corrected bare-column comparison Filter: dates.d = Date32("2024-01-01") The diff of this commit is the observable effect of the fix. Signed-off-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>

adriangb · 2026-07-21T00:37:22Z

@kosiew another hopefully small one to look at

kosiew

@adriangb
Thanks for the ping! I took a look and didn't find any blocking issues. I have one small suggestion that could improve the regression coverage.

kosiew · 2026-07-23T08:03:39Z

+create table dates(d date) as values (DATE '2024-01-01'), (DATE '2024-01-02');
+
+query TT
+explain select d from dates where cast(d as date) = DATE '2024-01-01';


Nice addition! One optional follow-up would be to add an IN predicate regression, for example cast(d AS date) IN (DATE '2024-01-01'). IN goes through a separate validation and rewrite path that also relies on the same literal-cast helper, so this would help extend the date-specific integration coverage beyond binary comparisons.

added in 1fe86c8

`IN` predicates go through a separate validation and rewrite path but rely on the same literal-cast helper as binary comparisons. Add an SLT regression that `cast(d AS date) IN (DATE '2024-01-01')` on a Date32 column folds to a bare-column comparison, extending the date-specific integration coverage beyond binary comparisons. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

adriangb · 2026-07-23T17:35:06Z

Thanks for the review!

…he#23729) ## Which issue does this PR close? - N/A. Small, self-contained enhancement to `unwrap_cast_in_comparison`; happy to file a tracking issue if preferred. ## Rationale for this change `unwrap_cast_in_comparison` could not unwrap a cast between the two date types, so a predicate like `CAST(date32_col AS Date64) <op> <date64 literal>` never folded onto the bare column. Folding it lets the literal be compared against the unmodified column, keeping predicate pushdown / pruning effective on date columns. There were two coupled causes in `try_cast_literal_to_type` (`datafusion/expr-common/src/casts.rs`): 1. `try_cast_numeric_literal` scaled both `Date32` and `Date64` literals by the same target multiplier (`mul = 1`). But `Date32` counts **days** since the epoch while `Date64` counts **milliseconds**, so a cross conversion needs a factor of `MILLISECONDS_IN_DAY` (86_400_000). 2. `is_lossy_temporal_cast` classified every `Date <-> temporal` pair as lossy, which swept in `Date32 <-> Date64` and blocked the unwrap outright. The reverse direction is subtle and unsound if handled naively: narrowing a `Date64` **column** down to `Date32` truncates milliseconds to the day (many-to-one), so `CAST(date64 AS Date32) = <day>` matches any millisecond within that day. arrow-rs does not require `Date64` values to be whole-day (apache/arrow-rs#5288), so the column may carry sub-day values the planner cannot see, and unwrapping would drop those rows. That direction is therefore explicitly blocked. ## What changes are included in this PR? - Relax `is_lossy_temporal_cast` so a date-to-date (and identity) cast is not pre-classified as lossy; per-value exactness is enforced downstream. - Add `scale_date_literal` with exact-only semantics: `Date32 -> Date64` multiplies by `MILLISECONDS_IN_DAY` (overflow-guarded with checked arithmetic); `Date64 -> Date32` divides only on a whole-day boundary and otherwise returns `None`. This mirrors the existing Decimal scaling path in the same function. - Add `is_date_narrowing_cast` and block the narrowing `Date64 -> Date32` column cast in the two logical-optimizer gates (comparison and in-list) **and** in the physical-expr simplifier, mirroring `is_timestamp_precision_narrowing_cast`. The physical-expr guard is required for soundness on the pruning / row-group-filter path (verified by a unit test that fails without it); the widening `Date32 -> Date64` column cast is injective and stays supported. Scope is intentionally limited to `Date32 <-> Date64` scaling, the narrowing gate, and tests. ## Are these changes tested? Yes, at two levels. **End-to-end (`datafusion/sqllogictest/test_files/simplify_expr.slt`)** — the PR is structured as three commits so the behavior change is legible in the diff: 1. `test:` characterizes current behavior (passes on unmodified `main`): neither direction is unwrapped, results are correct. The fixture stores sub-day and pre-epoch `Date64` values on purpose. 2. `feat:` applies only the code change; the recorded widening `EXPLAIN` plans now fail intentionally. 3. `test:` regenerates the expectations. The commit-3 diff is exactly the widening plans flipping from `CAST(d32 AS Date64) <op> Date64(..)` to `d32 <op> Date32(..)`; **every result row and every narrowing plan is byte-identical**, which is the soundness proof. Coverage includes `=`/`<`/`<=`/`>`/`>=`/`IN`, whole-day vs sub-day literals (the latter yields zero rows and is left as-is), the narrowing soundness case (the noon row is still returned), pre-epoch dates (arrow's toward-zero truncation is pinned), and NULL three-valued logic. **Unit** — `scale_date_literal` exactness and `i32::MIN`/`i32::MAX` overflow, `is_date_narrowing_cast`, the relaxed `is_lossy_temporal_cast` date-pair behavior, and the physical-expr narrowing guard. `cargo fmt --check` is clean, `cargo clippy -p datafusion-expr-common -- -D warnings` passes, and the full sqllogictest suite passes. ## Are there any user-facing changes? No public API changes. The optimizer now additionally rewrites widening `Date32 -> Date64` cast comparisons where it previously left them untouched; results are unchanged, plans are simplified. Narrowing `Date64 -> Date32` cast comparisons are deliberately left as-is. ## Note for reviewers The open PR apache#23727 adds the `if from_type == to_type { return false }` identity guard to this same function. This PR is based independently on `main` and includes that identity line as part of the clean gate shape here, so depending on merge order the two may need a trivial rebase where those lines overlap. --------- Signed-off-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

## Which issue does this PR close? No existing issue; this PR both reports and fixes the bug. Happy to file a tracking issue if preferred.  ## Rationale for this change `unwrap_cast_in_comparison` fails to fold an identity `CAST(col AS DATE)` on a `Date32` column. Instead of rewriting the predicate to compare against the bare column, it leaves a residual `Cast(col AS Date32)` in place, which defeats downstream optimizations (pruning / filter pushdown) that expect a bare-column comparison. Reproduction (logical plan for `SELECT * FROM t WHERE cast(d AS date) = DATE '2024-01-01'`, where `d` is `Date32`): the `cast(d AS Date32)` survives simplification rather than collapsing to `d`. Note `arrow_cast(d, 'Date32')` already folds, because the `arrow_cast` UDF's `simplify()` short-circuits an identity cast; the SQL `CAST ... AS date` planner plants a real `Expr::Cast` with no such elision, so it reaches `try_cast_literal_to_type` and is wrongly rejected. The root cause is in `is_lossy_temporal_cast` (`datafusion/expr-common/src/casts.rs`): ```rust (is_date_type(from_type) && to_type.is_temporal()) || (is_date_type(to_type) && from_type.is_temporal()) ``` For an identity `Date32 -> Date32` cast this evaluates to `true && true`, because `DataType::is_temporal()` is true for both `Date32` and `Date64`. The identity cast is therefore misclassified as a lossy temporal cast, `try_cast_literal_to_type` returns `None`, and `unwrap_cast_in_comparison` leaves the cast in the plan. ## What changes are included in this PR? Short-circuit an identity cast as non-lossy at the top of `is_lossy_temporal_cast`: ```rust if from_type == to_type { return false; // an identity cast never changes comparison semantics } ``` This is deliberately limited to *identical* types, not "any date-to-date". `Date32` counts days while `Date64` counts milliseconds, but `try_cast_numeric_literal` uses `mul = 1` for both, so allowing a `Date32 <-> Date64` unwrap would convert units incorrectly. The `from_type == to_type` identity guard is the exact correct scope, and `Date32 <-> Date64` remains blocked. ## Are these changes tested? Yes. The PR is structured as a test-driven, stacked sequence so the effect of the fix is visible in the diff: 1. **`test: characterize identity Date cast in comparison unwrapping`** — adds an SLT test to `simplify_expr.slt` (`explain select d from dates where cast(d as date) = DATE '2024-01-01'`) whose assertion records the *current, buggy* output: the logical plan keeps the residual `Filter: CAST(dates.d AS Date32) = Date32(...)`. Passes on `main`. 2. **`fix: unwrap identity Date cast in comparison unwrapping`** — the production change plus two `expr-common` unit tests: - `test_try_cast_identity_date_allowed` — identity `Date32 -> Date32` / `Date64 -> Date64` now fold (`try_cast_literal_to_type` returns `Some`), and `is_lossy_temporal_cast` reports them non-lossy. - `test_try_cast_date32_date64_still_blocked` — `Date32 <-> Date64` stays lossy/blocked (guards the units caveat above, which SLT can't easily express). At this commit the SLT test is intentionally red, proving it exercises the bug. 3. **`test: update identity Date cast assertion to folded plan`** — updates the SLT assertion to the corrected `Filter: dates.d = Date32(...)`. The one-line diff is the observable effect of the fix. `cargo fmt --check` and `cargo clippy -D warnings` are clean on the changed crate; `datafusion-expr-common` and the `simplify_expr` sqllogictest pass. ## Are there any user-facing changes? No API changes. Queries with `CAST(date_col AS DATE)` predicates on `Date32` columns simplify to bare-column comparisons, which can enable additional pruning/pushdown. No behavioral change to query results. --------- Signed-off-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…he#23729) ## Which issue does this PR close? - N/A. Small, self-contained enhancement to `unwrap_cast_in_comparison`; happy to file a tracking issue if preferred. ## Rationale for this change `unwrap_cast_in_comparison` could not unwrap a cast between the two date types, so a predicate like `CAST(date32_col AS Date64) <op> <date64 literal>` never folded onto the bare column. Folding it lets the literal be compared against the unmodified column, keeping predicate pushdown / pruning effective on date columns. There were two coupled causes in `try_cast_literal_to_type` (`datafusion/expr-common/src/casts.rs`): 1. `try_cast_numeric_literal` scaled both `Date32` and `Date64` literals by the same target multiplier (`mul = 1`). But `Date32` counts **days** since the epoch while `Date64` counts **milliseconds**, so a cross conversion needs a factor of `MILLISECONDS_IN_DAY` (86_400_000). 2. `is_lossy_temporal_cast` classified every `Date <-> temporal` pair as lossy, which swept in `Date32 <-> Date64` and blocked the unwrap outright. The reverse direction is subtle and unsound if handled naively: narrowing a `Date64` **column** down to `Date32` truncates milliseconds to the day (many-to-one), so `CAST(date64 AS Date32) = <day>` matches any millisecond within that day. arrow-rs does not require `Date64` values to be whole-day (apache/arrow-rs#5288), so the column may carry sub-day values the planner cannot see, and unwrapping would drop those rows. That direction is therefore explicitly blocked. ## What changes are included in this PR? - Relax `is_lossy_temporal_cast` so a date-to-date (and identity) cast is not pre-classified as lossy; per-value exactness is enforced downstream. - Add `scale_date_literal` with exact-only semantics: `Date32 -> Date64` multiplies by `MILLISECONDS_IN_DAY` (overflow-guarded with checked arithmetic); `Date64 -> Date32` divides only on a whole-day boundary and otherwise returns `None`. This mirrors the existing Decimal scaling path in the same function. - Add `is_date_narrowing_cast` and block the narrowing `Date64 -> Date32` column cast in the two logical-optimizer gates (comparison and in-list) **and** in the physical-expr simplifier, mirroring `is_timestamp_precision_narrowing_cast`. The physical-expr guard is required for soundness on the pruning / row-group-filter path (verified by a unit test that fails without it); the widening `Date32 -> Date64` column cast is injective and stays supported. Scope is intentionally limited to `Date32 <-> Date64` scaling, the narrowing gate, and tests. ## Are these changes tested? Yes, at two levels. **End-to-end (`datafusion/sqllogictest/test_files/simplify_expr.slt`)** — the PR is structured as three commits so the behavior change is legible in the diff: 1. `test:` characterizes current behavior (passes on unmodified `main`): neither direction is unwrapped, results are correct. The fixture stores sub-day and pre-epoch `Date64` values on purpose. 2. `feat:` applies only the code change; the recorded widening `EXPLAIN` plans now fail intentionally. 3. `test:` regenerates the expectations. The commit-3 diff is exactly the widening plans flipping from `CAST(d32 AS Date64) <op> Date64(..)` to `d32 <op> Date32(..)`; **every result row and every narrowing plan is byte-identical**, which is the soundness proof. Coverage includes `=`/`<`/`<=`/`>`/`>=`/`IN`, whole-day vs sub-day literals (the latter yields zero rows and is left as-is), the narrowing soundness case (the noon row is still returned), pre-epoch dates (arrow's toward-zero truncation is pinned), and NULL three-valued logic. **Unit** — `scale_date_literal` exactness and `i32::MIN`/`i32::MAX` overflow, `is_date_narrowing_cast`, the relaxed `is_lossy_temporal_cast` date-pair behavior, and the physical-expr narrowing guard. `cargo fmt --check` is clean, `cargo clippy -p datafusion-expr-common -- -D warnings` passes, and the full sqllogictest suite passes. ## Are there any user-facing changes? No public API changes. The optimizer now additionally rewrites widening `Date32 -> Date64` cast comparisons where it previously left them untouched; results are unchanged, plans are simplified. Narrowing `Date64 -> Date32` cast comparisons are deliberately left as-is. ## Note for reviewers The open PR apache#23727 adds the `if from_type == to_type { return false }` identity guard to this same function. This PR is based independently on `main` and includes that identity line as part of the clean gate shape here, so depending on merge order the two may need a trivial rebase where those lines overlap. --------- Signed-off-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

github-actions Bot added logical-expr Logical plan and expressions optimizer Optimizer rules labels Jul 20, 2026

This was referenced Jul 20, 2026

fix: unwrap identity Date cast in comparison unwrapping pydantic/datafusion#64

Closed

Unwrap widening Date32 -> Date64 casts in comparison predicates #23729

Merged

adriangb added 3 commits July 20, 2026 19:13

adriangb force-pushed the fix-identity-date-cast-unwrap-upstream branch from 8a5e75d to 0ec7b72 Compare July 21, 2026 00:17

github-actions Bot added sqllogictest SQL Logic Tests (.slt) and removed optimizer Optimizer rules labels Jul 21, 2026

adriangb requested a review from kosiew July 21, 2026 00:37

kosiew approved these changes Jul 23, 2026

View reviewed changes

adriangb enabled auto-merge July 23, 2026 15:57

adriangb added this pull request to the merge queue Jul 23, 2026

Merged via the queue into apache:main with commit 0f2e137 Jul 23, 2026
40 checks passed

adriangb deleted the fix-identity-date-cast-unwrap-upstream branch July 23, 2026 16:59

Navigation Menu

Uh oh!

fix: unwrap identity Date cast in comparison unwrapping - #23727

fix: unwrap identity Date cast in comparison unwrapping#23727
adriangb merged 4 commits into
apache:mainfrom
pydantic:fix-identity-date-cast-unwrap-upstream

adriangb commented Jul 20, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 20, 2026 •

edited

Loading

Uh oh!

adriangb commented Jul 21, 2026

Uh oh!

kosiew left a comment •

edited

Loading

Uh oh!

kosiew Jul 23, 2026

Uh oh!

adriangb Jul 23, 2026

Uh oh!

Uh oh!

adriangb commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

adriangb commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

adriangb commented Jul 21, 2026

Uh oh!

kosiew left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

adriangb Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adriangb commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adriangb commented Jul 20, 2026 •

edited

Loading

codecov-commenter commented Jul 20, 2026 •

edited

Loading

kosiew left a comment •

edited

Loading