fix: Disable join dynamic filters for null-equal joins - #22965
Merged
Conversation
Contributor
Author
|
cc @adriangb |
mdashti
added a commit
to paradedb/datafusion
that referenced
this pull request
Jun 23, 2026
apache#22965 disabled dynamic filter pushdown for null-equal joins because the build-side predicate prunes a probe-side NULL that can null-match a build-side NULL. Push the filter with `OR key IS NULL` over the nullable probe keys instead, the way apache#23104 does for null-aware anti joins. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity.
mdashti
added a commit
to paradedb/datafusion
that referenced
this pull request
Jun 23, 2026
apache#22965 disabled dynamic filter pushdown for null-equal joins because the build-side predicate prunes a probe-side NULL that can null-match a build-side NULL. Push the filter with `OR key IS NULL` over the nullable probe keys instead, the way apache#23104 does for null-aware anti joins. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity.
mdashti
added a commit
to paradedb/datafusion
that referenced
this pull request
Jun 23, 2026
apache#22965 disabled dynamic filter pushdown for null-equal joins because the build-side predicate prunes a probe-side NULL that can null-match a build-side NULL. Push the filter with `OR key IS NULL` over the nullable probe keys instead, the way apache#23104 does for null-aware anti joins. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity.
pull Bot
pushed a commit
to TCeason/arrow-datafusion
that referenced
this pull request
Aug 9, 2026
…ate (apache#23106) ## Which issue does this close? Re-enables the dynamic filter that apache#22965 disabled (apache#22964), with the proper null-equal semantics. ## Rationale for this change apache#22965 disabled hash-join dynamic filter pushdown for null-equal joins: the build-side bounds and membership predicates evaluate to NULL for a probe-side NULL key, so they prune rows that should null-match a build-side NULL. Its description already named the better fix, "generate a predicate with `OR IS NULL`". apache#23104 does that for null-aware anti joins; this re-enables the null-equal case the same way. ## What changes are included in this PR? - Revert the null-equal `return false` in `allow_join_dynamic_filter_pushdown`. - Generalize the shared probe-NULL helper to cover both null-aware (single-key) and null-equal (multi-key) joins: OR `key IS NULL` for every nullable probe key. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity. ## Are these changes tested? Yes. apache#22965's SLT now asserts the filter is back on the probe with the result unchanged, plus a multi-key null-equal case. The reject unit test flips to assert pushdown is allowed, and `preserve_probe_nulls` unit tests cover both the mixed nullable/NOT NULL case (only the nullable key widens) and the all-NOT-NULL case (no widening). ## Are there any user-facing changes? Null-equal joins regain dynamic filter pushdown, so they prune the probe scan again while returning correct results.
kosiew
pushed a commit
to kosiew/datafusion
that referenced
this pull request
Aug 12, 2026
…ate (apache#23106) ## Which issue does this close? Re-enables the dynamic filter that apache#22965 disabled (apache#22964), with the proper null-equal semantics. ## Rationale for this change apache#22965 disabled hash-join dynamic filter pushdown for null-equal joins: the build-side bounds and membership predicates evaluate to NULL for a probe-side NULL key, so they prune rows that should null-match a build-side NULL. Its description already named the better fix, "generate a predicate with `OR IS NULL`". apache#23104 does that for null-aware anti joins; this re-enables the null-equal case the same way. ## What changes are included in this PR? - Revert the null-equal `return false` in `allow_join_dynamic_filter_pushdown`. - Generalize the shared probe-NULL helper to cover both null-aware (single-key) and null-equal (multi-key) joins: OR `key IS NULL` for every nullable probe key. A NOT NULL key never widens the filter, so an all-NOT-NULL join keeps full selectivity. ## Are these changes tested? Yes. apache#22965's SLT now asserts the filter is back on the probe with the result unchanged, plus a multi-key null-equal case. The reject unit test flips to assert pushdown is allowed, and `preserve_probe_nulls` unit tests cover both the mixed nullable/NOT NULL case (only the nullable key widens) and the all-NOT-NULL case (no widening). ## Are there any user-facing changes? Null-equal joins regain dynamic filter pushdown, so they prune the probe scan again while returning correct results.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
We presently allow dynamic filter pushdown to be applied to null-equal hash joins. This might result in pushing a predicate down into the probe-side plan, where the predicate will not be evaluated with the null-equal semantics that are required.
Longer-term, we might consider supporting this case with the correct semantics (e.g., generate a predicate with
OR IS NULL ...), but for now disabling pushdown for null-equal joins seems much more practical.What changes are included in this PR?
Are these changes tested?
Yes.
Are there any user-facing changes?
No.