IN LIST: centralize primitive filters and optimize Decimal128 - #24283
Open
geoffreyclaude wants to merge 5 commits into
Open
IN LIST: centralize primitive filters and optimize Decimal128#24283geoffreyclaude wants to merge 5 commits into
geoffreyclaude wants to merge 5 commits into
Conversation
This was referenced Aug 12, 2026
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24283 +/- ##
==========================================
+ Coverage 80.91% 81.14% +0.23%
==========================================
Files 1102 1112 +10
Lines 377102 386940 +9838
Branches 377102 386940 +9838
==========================================
+ Hits 305143 313998 +8855
- Misses 53769 54439 +670
- Partials 18190 18503 +313 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geoffreyclaude
force-pushed
the
codex/in-list-primitive-selector-decimal
branch
7 times, most recently
from
August 13, 2026 10:14
00885e0 to
65b1e9f
Compare
Use PrimitiveHashSetFilter<T, K> for integer and floating-point arrays. Integer filters keep their native key type by default; Float32 and Float64 use their existing bitwise wrapper keys, preserving signed-zero and NaN-payload behavior. The key type and conversion are selected statically, with no function pointer or dynamic dispatch in the lookup loop. Filter selection, dictionary handling, null handling, IN, and NOT IN behavior remain unchanged. Decimal128 routing remains in the following commit.
Route Decimal128 lists above the branchless cutoff through PrimitiveHashSetFilter, replacing the ArrayStaticFilter fallback.
geoffreyclaude
force-pushed
the
codex/in-list-primitive-selector-decimal
branch
from
August 13, 2026 10:46
65b1e9f to
510bf3e
Compare
geoffreyclaude
marked this pull request as ready for review
August 14, 2026 11:15
Contributor
Author
|
run benchmark in_list_strategy |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing codex/in-list-primitive-selector-decimal (510bf3e) to 8649e80 diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "8649e80c239c68d141af4ec655c5132ea2660169"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing codex/in-list-primitive-selector-decimal (510bf3e) to 8649e80 diff Run configurationrun benchmark in_list_strategy
baseline:
ref: "8649e80c239c68d141af4ec655c5132ea2660169"CPU Details (lscpu)Details
Resource Usagein_list_strategy — base (merge-base)
in_list_strategy — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
#23014 adds fast direct comparisons for short primitive
INlists. For longer lists, DataFusion uses a bitmap, a hash set, or another specialized filter when one exists, and otherwise uses the general filter.#24102 needs to make that same choice after reading
FixedSizeBinaryvalues as same-width primitive keys. Today, the choice is part of the native primitive path, so #24102 would have to repeat its limits and fallback rules.This PR moves that choice beside the primitive filters and makes it reusable. The limits count only non-null list values. A separate commit uses one
PrimitiveHashSetFilter<T, K>for the existing integer and floating-point hash-set paths. Integers keep their native keys;Float32andFloat64keep their bitwise keys, including distinct signed zeros and NaN encodings. Filter selection and SQL behavior are unchanged.The only new lookup path is for
Decimal128. Lists with at most four non-null values still use direct comparisons. Larger lists now use a hash set over the storedi128values instead of the general Arrow filter. Values are not rescaled, and the existing precision, scale, and null rules are unchanged. #24102 also uses this path for 16-byteFixedSizeBinaryvalues.What changes are included in this PR?
PrimitiveHashSetFilter<Decimal128Type>forDecimal128lists with more than four non-null values.Decimal128benchmark cases at list sizes 5 and 64.Are these changes tested?
Tests cover the direct-comparison boundary, an all-null list, and bitwise float hash keys, including signed zero and distinct NaN encodings. Existing end-to-end Decimal tests cover hits, misses, nulls,
IN, andNOT IN.Are there any user-facing changes?
No. SQL results and public APIs are unchanged.
Benchmark snapshot
The benchmark uses list size 5, the first size above the direct-comparison limit, and a larger size of 64. Each size is measured with 0% and 50% matches. Filter construction is outside the timed loop. Lower is better.
An automated run compared this PR's benchmark-only first commit with the completed implementation. The benchmark source is identical in both states, so the comparison isolates the production changes.
All four cases improved, with a 34.2% geometric-mean reduction in execution time.
primitive/decimal128/large_list/list=5/match=0%primitive/decimal128/large_list/list=5/match=50%primitive/decimal128/large_list/list=64/match=0%primitive/decimal128/large_list/list=64/match=50%