Optimize existence-only right semi and anti joins - #22828
Conversation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
fb316c6 to
72363e8
Compare
|
run benchmark hash_join_semi_anti tpch10 tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/existence-semi-join-kernel (72363e8) to 883c38e (merge-base) diff using: tpch10 File an issue against this benchmark runner |
|
🤖 Criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/existence-semi-join-kernel (72363e8) to 883c38e (merge-base) diff File an issue against this benchmark runner |
|
Benchmark for this request failed. Last 20 lines of output: Click to expandFile an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/existence-semi-join-kernel (72363e8) to 883c38e (merge-base) diff using: tpcds File an issue against this benchmark runner |
Seems similar to what I did for SMJ in #20806. There was a ton of wasted materialization work before. I later used mark to the same stream (#21184), can this optimization benefit mark as well? |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
72363e8 to
53bec28
Compare
|
run benchmark tpcds tpch10 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/existence-semi-join-kernel (53bec28) to 883c38e (merge-base) diff using: tpch10 File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
run benchmark tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/existence-semi-join-kernel (53bec28) to 883c38e (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
Is it similar to this PR: #21775 |
|
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days. |
Which issue does this PR close?
Rationale for this change
Right semi, right anti, and right mark joins only need to know whether each probe-side row has at least one build-side match. The existing hash join probe path can materialize every build-side match before the semi/anti/mark output projection, which is unnecessarily expensive for high-fanout build keys.
What changes are included in this PR?
JoinHashMapandArrayMapthat return at most one build-side match per probe row.RightSemi,RightAnti, andRightMarkhash joins.JoinHashMapprobes on the existing path to avoid comparator overhead when there are no chains to skip.ArrayMapduplicate-key behavior.RightMarkhigh-fanout benchmark case.Are these changes tested?
cargo fmt --allcargo test -p datafusion-physical-plan --lib join_right -- --nocapturecargo test -p datafusion-physical-plan --lib matching_indices -- --nocapturecargo clippy --all-targets --all-features -- -D warningscargo bench -p datafusion-physical-plan --features test_utils --bench hash_join_semi_anti -- --baseline main-inplace --sample-size 10 --warm-up-time 1 --measurement-time 2cargo bench -p datafusion-physical-plan --features test_utils --bench hash_join_semi_anti -- right_mark_fanout100_h1 --sample-size 10 --warm-up-time 1 --measurement-time 2Selected Criterion results:
Are there any user-facing changes?
No.