POC: validate current DataFusion browser WASM stack - #2
Draft
ethan-tyler wants to merge 109 commits into
Draft
Conversation
… hooks (apache#23784) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23501 . ## Rationale for this change Migrates the `EmptyExec` and `PlaceholderRowExec` leaf plans while preserving the existing protobuf wire format. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Implement `ExecutionPlan::try_to_proto` for `EmptyExec` and `PlaceholderRowExec`. - Add plan-specific `try_from_proto` implementations. - Route protobuf encoding and decoding through the new hooks. - Keep the deprecated helper methods as compatibility shims. ## Are these changes tested? yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Hi @andygrove , would you be willing to review this PR when you have time? Thanks!
…ws (apache#23707) ## Which issue does this PR close? - Closes apache#22666 ## Rationale for this change Under SQL aggregate `FILTER` semantics, a row passes only when the predicate evaluates to `true`; rows where the predicate is `null` must be excluded. Grouped `first_value` / `last_value` checked only `BooleanArray::value(idx)`, without checking validity, so a NULL predicate row whose underlying value bit is set (as produced by comparison kernels, e.g. `null::int < 1`) was treated as passing: ```sql SELECT g, first_value(a ORDER BY a) FILTER (WHERE b < 1) AS fv FROM (VALUES (0, 10, CAST(NULL AS INT)), (0, 20, 2)) AS t(g, a, b) GROUP BY g; -- returned fv = 10, must return fv = NULL -- (row 1: b < 1 is NULL; row 2: b < 1 is FALSE — no row satisfies `true`) ``` ## What changes are included in this PR? `datafusion/functions-aggregate/src/first_last.rs`, in `FirstLastGroupsAccumulator::get_filtered_extreme_of_each_group` (shared by `first_value` and `last_value`, and by both the `update_batch` and `merge_batch` paths): - `passed_filter` now requires `is_valid(idx) && value(idx)` (the `Some(true)` semantics), matching the convention already used by `variance.rs` / `correlation.rs`. - The `is_set_arr` read gets the same validity check. This is *not* only an internal bitmap: `convert_to_state` stores the user FILTER clause (including its nulls) in the last state column, so on the merge path (e.g. skip-partial-aggregation) NULL predicate rows were likewise treated as set. Verified with a forced skip-partial run (100k unique groups, all-NULL predicates): 83,616 groups were incorrectly assigned non-NULL values before the fix, 0 after. For genuine internal bitmaps (no nulls) the added check is trivially true, so behavior there is unchanged. Regression coverage: - sqllogictest (`aggregate.slt`): the issue reproducer, the `last_value` counterpart, mixed TRUE/FALSE/NULL predicates, all-TRUE and no-FILTER controls, the (already correct) non-grouped path, and a window-function no-regression case. - Unit tests: `test_group_acc_filter_null_predicate` (update path) and `test_group_acc_merge_null_is_set` (merge path via `convert_to_state` → `merge_batch`), both constructing `BooleanArray`s whose null slots carry a set value bit. ## Are these changes tested? Yes — see above. Verified `./dev/rust_lint.sh`, `cargo test -p datafusion-functions-aggregate --lib`, the `aggregate`/`window` sqllogictest files, and `datafusion-cli` end-to-end (grouped first/last_value now return NULL for the issue reproducer; mixed-predicate and non-grouped results unchanged). Also audited the rest of `functions-aggregate` for the same validity-blind pattern: shared helpers (`nulls.rs::filter_to_validity`, `accumulate.rs`, `prim_op.rs`, `count.rs`, `array_agg.rs`, `variance.rs`, `correlation.rs`) already handle validity correctly, and non-grouped paths pre-filter with arrow's `filter` kernel (which drops NULL predicate rows), so no other aggregate needs changes. ## Performance `functions-aggregate/benches/first_last.rs` was run against `main`. The added checks are one validity-bit test per row on the grouped path; measured deltas were within the machine's noise floor (±5% on unchanged `filter=false` cases). A variant hoisting the null check out of the row loop showed no measurable benefit beyond noise, so the simple idiomatic form is kept. ## Are there any user-facing changes? Only the bug fix: grouped `first_value`/`last_value` with a nullable `FILTER` predicate now correctly exclude NULL-predicate rows, matching SQL semantics and the behavior of other aggregates. No API or configuration changes. --------- Co-authored-by: Claude <noreply@anthropic.com>
…he#23729) ## Which issue does this PR close? - N/A. Small, self-contained enhancement to `unwrap_cast_in_comparison`; happy to file a tracking issue if preferred. ## Rationale for this change `unwrap_cast_in_comparison` could not unwrap a cast between the two date types, so a predicate like `CAST(date32_col AS Date64) <op> <date64 literal>` never folded onto the bare column. Folding it lets the literal be compared against the unmodified column, keeping predicate pushdown / pruning effective on date columns. There were two coupled causes in `try_cast_literal_to_type` (`datafusion/expr-common/src/casts.rs`): 1. `try_cast_numeric_literal` scaled both `Date32` and `Date64` literals by the same target multiplier (`mul = 1`). But `Date32` counts **days** since the epoch while `Date64` counts **milliseconds**, so a cross conversion needs a factor of `MILLISECONDS_IN_DAY` (86_400_000). 2. `is_lossy_temporal_cast` classified every `Date <-> temporal` pair as lossy, which swept in `Date32 <-> Date64` and blocked the unwrap outright. The reverse direction is subtle and unsound if handled naively: narrowing a `Date64` **column** down to `Date32` truncates milliseconds to the day (many-to-one), so `CAST(date64 AS Date32) = <day>` matches any millisecond within that day. arrow-rs does not require `Date64` values to be whole-day (apache/arrow-rs#5288), so the column may carry sub-day values the planner cannot see, and unwrapping would drop those rows. That direction is therefore explicitly blocked. ## What changes are included in this PR? - Relax `is_lossy_temporal_cast` so a date-to-date (and identity) cast is not pre-classified as lossy; per-value exactness is enforced downstream. - Add `scale_date_literal` with exact-only semantics: `Date32 -> Date64` multiplies by `MILLISECONDS_IN_DAY` (overflow-guarded with checked arithmetic); `Date64 -> Date32` divides only on a whole-day boundary and otherwise returns `None`. This mirrors the existing Decimal scaling path in the same function. - Add `is_date_narrowing_cast` and block the narrowing `Date64 -> Date32` column cast in the two logical-optimizer gates (comparison and in-list) **and** in the physical-expr simplifier, mirroring `is_timestamp_precision_narrowing_cast`. The physical-expr guard is required for soundness on the pruning / row-group-filter path (verified by a unit test that fails without it); the widening `Date32 -> Date64` column cast is injective and stays supported. Scope is intentionally limited to `Date32 <-> Date64` scaling, the narrowing gate, and tests. ## Are these changes tested? Yes, at two levels. **End-to-end (`datafusion/sqllogictest/test_files/simplify_expr.slt`)** — the PR is structured as three commits so the behavior change is legible in the diff: 1. `test:` characterizes current behavior (passes on unmodified `main`): neither direction is unwrapped, results are correct. The fixture stores sub-day and pre-epoch `Date64` values on purpose. 2. `feat:` applies only the code change; the recorded widening `EXPLAIN` plans now fail intentionally. 3. `test:` regenerates the expectations. The commit-3 diff is exactly the widening plans flipping from `CAST(d32 AS Date64) <op> Date64(..)` to `d32 <op> Date32(..)`; **every result row and every narrowing plan is byte-identical**, which is the soundness proof. Coverage includes `=`/`<`/`<=`/`>`/`>=`/`IN`, whole-day vs sub-day literals (the latter yields zero rows and is left as-is), the narrowing soundness case (the noon row is still returned), pre-epoch dates (arrow's toward-zero truncation is pinned), and NULL three-valued logic. **Unit** — `scale_date_literal` exactness and `i32::MIN`/`i32::MAX` overflow, `is_date_narrowing_cast`, the relaxed `is_lossy_temporal_cast` date-pair behavior, and the physical-expr narrowing guard. `cargo fmt --check` is clean, `cargo clippy -p datafusion-expr-common -- -D warnings` passes, and the full sqllogictest suite passes. ## Are there any user-facing changes? No public API changes. The optimizer now additionally rewrites widening `Date32 -> Date64` cast comparisons where it previously left them untouched; results are unchanged, plans are simplified. Narrowing `Date64 -> Date32` cast comparisons are deliberately left as-is. ## Note for reviewers The open PR apache#23727 adds the `if from_type == to_type { return false }` identity guard to this same function. This PR is based independently on `main` and includes that identity line as part of the clean gate shape here, so depending on merge order the two may need a trivial rebase where those lines overlap. --------- Signed-off-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Which issue does this PR close? - Part of apache#22710. ## Rationale for this change There's almost no slt coverage for grouped aggregation under a memory limit, where the aggregate spills to disk and re-groups the spilled state. apache#23657 covers the ordered path & this covers the unordered/hash path. ## What changes are included in this PR? - Adds `aggregate_memory_limit.slt` in which the high-cardinality "GROUP BY" is run under a 1M limit, so the hash aggregate spills. The group key is scrambled with `(v * 7) % 100000` (a bijection, so still 100000 groups) to keep the input unsorted; otherwise it takes the streaming path and never spills. - Scoped to a single partition (`target_partitions = 1`), so the spill happens in one aggregate operator with no repartition. - Cases cover different accumulator states: single-column, multi-column, count(DISTINCT), sum/min/max, avg (widening), array_agg (growable). ## Are these changes tested? This PR is tests. All pass locally. ## Are there any user-facing changes? No.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#22342. ## Rationale for this change `BinaryExpr` already uses pre-selection for `AND` when only a small set of LHS rows can affect the final result. This adds the matching optimization for `OR` when most LHS rows are already true. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> ## What changes are included in this PR? This PR extends pre-selection short-circuiting to `OR`. For `OR`, the RHS is evaluated only for rows where the LHS is false. Rows where the LHS is true are filled directly as true. The existing `AND` path is kept and the scatter logic is shared. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? Yes <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> ## Are there any user-facing changes? No Public API Change <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? N/A ## Rationale for this change Improve performance of existing expression. ## What changes are included in this PR? Replace per-row O(set_len) linear scan in find_in_set's constant-list path with a one-time HashMap lookup (threshold-guarded so short lists keep the linear scan), giving O(1) per-row probing for large sets. ## Are these changes tested? Existing tests + new tests Benchmark (criterion): - long_list_256: 95.827% faster (base 1246412ns -> cand 52009ns) - ~24x faster - short_list_4: 2.13% faster (base 64343ns -> cand 62972ns) - long_list_64: 88.562% faster (base 422083ns -> cand 48276ns) ## Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…oin (apache#22810) ## Problem `NOT IN (subquery)` is a null-aware anti join: when the subquery yields a NULL the predicate is never TRUE, so the query must return zero rows. With `prefer_hash_join = false` and multiple partitions, the planner routed the null-aware anti join to `SortMergeJoinExec`, which is not null-aware, so it returned wrong results. HashJoin (the default) was already correct. ## Proof ```sql set datafusion.optimizer.prefer_hash_join = false; create table t1(x int) as values (1); create table t2(y int) as values (NULL); select x from t1 where x not in (select y from t2); ``` Expected 0 rows (the subquery contains a NULL). Before this change it returned `1`. With `prefer_hash_join = true` it correctly returned 0 rows. `EXPLAIN` showed the wrong config selecting `SortMergeJoinExec: join_type=LeftAnti`. ## Solution The planner already requires null-aware joins to use the CollectLeft HashJoin, and the HashJoin branch guards on `!null_aware`. The SortMergeJoin branch was missing the same guard, so this adds `&& !*null_aware` to it. Null-aware anti joins now fall through to the CollectLeft HashJoin regardless of `prefer_hash_join`. `SortMergeJoinExec` has no `null_aware` parameter and cannot honor these semantics. Added a regression test in `subquery.slt` (under `prefer_hash_join = false`) covering both a null-containing subquery (zero rows) and a null-free subquery (normal anti join). All 61 SortMergeJoin unit tests pass.
…nvert_to_state` (apache#23489) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23081. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Following apache#23275, all `GroupsAccumulator` implementations now provide `convert_to_state`. The `supports_convert_to_state` capability flag is therefore no longer needed. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Make `GroupsAccumulator::convert_to_state` a required trait method. - Remove `GroupsAccumulator::supports_convert_to_state` and its implementations. - Remove the corresponding capability checks from hash aggregation. - Simplify skip-partial aggregation to use the required `convert_to_state` implementation directly. - Add a regression test covering the partial hash aggregation skip path. - Document the breaking trait change in the 55.0.0 upgrading guide. - Remove `FFI_GroupsAccumulator::supports_convert_to_state`. This changes the FFI ABI layout, so providers and consumers must be rebuilt against DataFusion 55. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. Added a regression test verifying that skip-partial aggregation uses the required `convert_to_state` implementation without a capability flag. Existing physical-plan and FFI tests continue to pass. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> Yes. This is a breaking Rust API change for external `GroupsAccumulator` implementations: - `convert_to_state` must now be implemented. - `supports_convert_to_state` should be removed. The migration is documented in the 55.0.0 upgrading guide. The `FFI_GroupsAccumulator` layout has changed. FFI providers and consumers must be rebuilt against DataFusion 55 and must not exchange this struct with older major versions.
…t/datafusion-wasm-app (apache#23866) Bumps [ws](https://github.com/websockets/ws) from 8.18.2 to 8.21.1. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/websockets/ws/releases">ws's releases</a>.</em></p> <blockquote> <h2>8.21.1</h2> <h1>Bug fixes</h1> <ul> <li>Empty fragments are now counted toward the limit (a2f4e7c0).</li> <li>The default values of the <code>maxBufferedChunks</code> and <code>maxFragments</code> options have been reduced (f197ac65).</li> </ul> <h2>8.21.0</h2> <h1>Features</h1> <ul> <li>Introduced the <code>maxBufferedChunks</code> and <code>maxFragments</code> options (2b2abd45).</li> </ul> <h1>Bug fixes</h1> <ul> <li>Fixed a remote memory exhaustion DoS vulnerability (2b2abd45).</li> </ul> <p>A high volume of tiny fragments and data chunks could be sent by a peer, using modest network traffic, to crash a <code>ws</code> server or client due to OOM.</p> <pre lang="js"><code>import { WebSocket, WebSocketServer } from 'ws'; <p>const wss = new WebSocketServer({ port: 0 }, function () { const data = Buffer.alloc(1); const options = { fin: false }; const { port } = wss.address(); const ws = new WebSocket(<code>ws://localhost:${port}</code>);</p> <p>ws.on('open', function () { (function send() { ws.send(data, options, function (err) { if (err) return; send(); }); })(); });</p> <p>ws.on('error', console.error); ws.on('close', function (code, reason) { console.log(<code>client close - code: ${code} reason: ${reason.toString()}</code>); }); });</p> <p>wss.on('connection', function (ws) { ws.on('error', console.error); ws.on('close', function (code, reason) { console.log(<code>server close - code: ${code} reason: ${reason.toString()}</code>); }); }); </code></pre></p> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/websockets/ws/commit/ae1de54330cef77e487548890fabfeb9aae1d83d"><code>ae1de54</code></a> [dist] 8.21.1</li> <li><a href="https://github.com/websockets/ws/commit/8e9511b86b3fc6deebbd97dd9af7c9056deea8d1"><code>8e9511b</code></a> [ci] Trust Coveralls Homebrew tap</li> <li><a href="https://github.com/websockets/ws/commit/f197ac65140920bdcecdab74bfc69c2d7858e55d"><code>f197ac6</code></a> [fix] Lower default values of <code>maxBufferedChunks</code> and <code>maxFragments</code></li> <li><a href="https://github.com/websockets/ws/commit/8df8265c2f63fd44af3193a98e23cf38888cd991"><code>8df8265</code></a> [ci] Update actions/checkout action to v7</li> <li><a href="https://github.com/websockets/ws/commit/a2f4e7c046c2112bbce6fef39a083dac77d6f0d2"><code>a2f4e7c</code></a> [fix] Count empty fragments toward the limit (<a href="https://redirect.github.com/websockets/ws/issues/2329">#2329</a>)</li> <li><a href="https://github.com/websockets/ws/commit/e79f912cb3f492ae04c28feb9459a209e186b0ad"><code>e79f912</code></a> [pkg] Approve install scripts for bufferutil and utf-8-validate</li> <li><a href="https://github.com/websockets/ws/commit/4ea355d6d3069394994f82ca1b6d38c32ba208fb"><code>4ea355d</code></a> [doc] Document 32-bit signed integer coercion for option values</li> <li><a href="https://github.com/websockets/ws/commit/2120f4c8c625a76316792680a231496e1b615252"><code>2120f4c</code></a> [example] Remove uuid dependency</li> <li><a href="https://github.com/websockets/ws/commit/4c534a6b8a5224a563af116e85c6ced7d4ca60cf"><code>4c534a6</code></a> [security] Add latest vulnerability to SECURITY.md</li> <li><a href="https://github.com/websockets/ws/commit/bca91adf15677e47dbe4f959653452727be28b94"><code>bca91ad</code></a> [dist] 8.21.0</li> <li>Additional commits viewable in <a href="https://github.com/websockets/ws/compare/8.18.2...8.21.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…datafusion/wasmtest/datafusion-wasm-app (apache#23865) Bumps [http-proxy-middleware](https://github.com/chimurai/http-proxy-middleware) from 2.0.9 to 2.0.10. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/chimurai/http-proxy-middleware/releases">http-proxy-middleware's releases</a>.</em></p> <blockquote> <h2>v2.0.10-beta.0</h2> <h2>What's Changed</h2> <ul> <li>fix: harden proxy-table matching to prevent routing bypass by <a href="https://github.com/G-Rath"><code>@G-Rath</code></a> in <a href="https://redirect.github.com/chimurai/http-proxy-middleware/pull/1268">chimurai/http-proxy-middleware#1268</a></li> <li>ci(github-actions): update publish.yml by <a href="https://github.com/chimurai"><code>@chimurai</code></a> in <a href="https://redirect.github.com/chimurai/http-proxy-middleware/pull/1270">chimurai/http-proxy-middleware#1270</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/G-Rath"><code>@G-Rath</code></a> made their first contribution in <a href="https://redirect.github.com/chimurai/http-proxy-middleware/pull/1268">chimurai/http-proxy-middleware#1268</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/chimurai/http-proxy-middleware/compare/v2.0.9...v2.0.10-beta.0">https://github.com/chimurai/http-proxy-middleware/compare/v2.0.9...v2.0.10-beta.0</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/chimurai/http-proxy-middleware/blob/v2.0.10/CHANGELOG.md">http-proxy-middleware's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v2.0.10">v2.0.10</a></h2> <ul> <li>fix(router): harden proxy-table matching (exact host for host+path keys, prefix-only path matching) to prevent routing bypass</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/chimurai/http-proxy-middleware/commit/f0be839066ba14f2598c6c1aef10ff54b020babb"><code>f0be839</code></a> chore(package.json): v2.0.10 (<a href="https://redirect.github.com/chimurai/http-proxy-middleware/issues/1271">#1271</a>)</li> <li><a href="https://github.com/chimurai/http-proxy-middleware/commit/19c860de9854988351c4c9bf5fe79687942e1833"><code>19c860d</code></a> ci(github-actions): update publish.yml (<a href="https://redirect.github.com/chimurai/http-proxy-middleware/issues/1270">#1270</a>)</li> <li><a href="https://github.com/chimurai/http-proxy-middleware/commit/d0f7d6368ac9cf7dac5638c466d820370a4ae98f"><code>d0f7d63</code></a> fix: harden proxy-table matching to prevent routing bypass (<a href="https://redirect.github.com/chimurai/http-proxy-middleware/issues/1268">#1268</a>)</li> <li>See full diff in <a href="https://github.com/chimurai/http-proxy-middleware/compare/v2.0.9...v2.0.10">compare view</a></li> </ul> </details> <details> <summary>Maintainer changes</summary> <p>This version was pushed to npm by <a href="https://www.npmjs.com/~GitHub%20Actions">GitHub Actions</a>, a new releaser for http-proxy-middleware since your current version.</p> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
add badge; easier to see at a glance what the coverage is, and also easier to access our codecov page <img width="789" height="145" alt="image" src="https://github.com/user-attachments/assets/ed44f56c-8daa-48c7-809a-679823084e0f" />
## Rationale for this change Logical `CASE` nullability unwraps null preserving casts before analyzing guarded branches, but physical `CASE` nullability did not. Type coercion could therefore produce conflicting schemas and cause valid aggregation queries to fail during planning --------- Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…driven optimizations (apache#23821) ## Which issue does this PR close? This PR adds test coverage rather than closing an issue. It documents the current behavior of these bugs so that fixing them shows up as a test change: - apache#23634 - apache#23818 - apache#23819 - apache#23820 ## Rationale for this change While reviewing apache#23636 @neilconway and I kept coming up with more examples of bad plans, and it was not obvious which of the surrounding wrong answers were pre-existing and which the PR introduced. ## What changes are included in this PR? A new `datafusion/sqllogictest/test_files/functional_dependencies.slt` with one section per consumer of functional dependencies: 1. `ReplaceDistinctWithAggregate` — removing `DISTINCT` 2. `eliminate_duplicated_expr` — dropping trailing `ORDER BY` keys 3. `optimize_projections` — dropping `GROUP BY` expressions 4. `add_group_by_exprs_from_dependencies` — selecting non-grouped columns 5. `GROUP BY` derived keys on the NULL-padded side of an outer join Cases that currently return wrong answers are labelled `BUG` with the expected result and a link to the issue: | Case | Symptom | Issue | | --- | --- | --- | | 1.2 | `DISTINCT` over a nullable `UNIQUE` column returns both `NULL` rows | apache#23634 | | 2.2 | `ORDER BY x, y` drops the `y` key, so the `NULL` rows come back unordered | apache#23818 | | 3.2 | `GROUP BY x, y` drops `y`, merging the two `NULL` groups and losing a row | apache#23819 | | 4.2 | `SELECT x, y ... GROUP BY x` returns two rows for the `x = NULL` group | apache#23820 | ## Are these changes tested? CI ## Are there any user-facing changes? No. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…3679) ## Which issue does this PR close? - Closes: N/A ## Rationale for this change There are a reasonable number of places where we have functions marked as `async` that don't need to be so. Fix this by enabling the `unused_async` lint, and fixing up the resulting breakage. There are a handful of spots that need an `expect(clippy::unused_async)` -- mostly mocks of `async` methods and example code. ## What changes are included in this PR? * Enable `unused_async` lint * Remove unnecessary `async` annotations in a bunch of places * Add `expect(clippy::unused_async)` where necessary ## Are these changes tested? Yes, covered by existing tests (no behavioral change expected). ## Are there any user-facing changes? Yes: this PR updates a few public APIs: - `datafusion_cli::command::OutputFormat::execute` - `datafusion::test_util::parquet::TestParquetFile::create_scan` - `datafusion_datasource_csv::file_format::CsvFormat::read_to_delimited_chunks_from_stream` - `datafusion_substrait::serializer::deserialize_bytes` Migration is mostly straightforward (e.g., removing `await` from calling code).
…fusion/wasmtest/datafusion-wasm-app (apache#23868) Bumps [webpack-dev-server](https://github.com/webpack/webpack-dev-server) from 5.2.6 to 6.0.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/webpack/webpack-dev-server/releases">webpack-dev-server's releases</a>.</em></p> <blockquote> <h2>v6.0.0</h2> <h3>Major Changes</h3> <ul> <li> <p>Bump Express to v5. See the <a href="https://expressjs.com/en/guide/migrating-5.html">Express 5 migration guide</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Bump the <code>webpack</code> peer dependency range from <code>^5.0.0</code> to <code>^5.101.0</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Drop support for Node.js < 22.15.0. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Convert the source to native ES modules. The package keeps <code>"type": "module"</code> and now exposes both an ESM and a CommonJS build via the <code>exports</code> field: ESM consumers import the native <code>lib/</code>, while CommonJS consumers <code>require()</code> a transpiled <code>dist/</code> build, allowing the package to be consumed from both ESM and CommonJS without relying on <code>require(ESM)</code> for CommonJS consumers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5674">#5674</a>)</p> </li> <li> <p>Remove CLI flags. Use the <code>serve</code> command from <code>webpack-cli</code> together with a configuration file or the programmatic API instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>internalIP</code> and <code>internalIPSync</code> static methods from <code>Server</code>. Resolve the local IP yourself if you need it. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>bypass</code> option from proxy configuration. Use the <code>router</code> or <code>context</code> options provided by <code>http-proxy-middleware</code> instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove SockJS support. The <code>webSocketServer</code> option no longer accepts <code>"sockjs"</code>; use the default <code>"ws"</code> transport instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>spdy</code> dependency. Use the built-in <code>node:http2</code> module via the <code>server</code> option for HTTP/2 support. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>http-proxy-middleware</code> to v4. See the <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v3.0.0">http-proxy-middleware v3 release notes</a> and <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v4.0.0">v4 release notes</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>webpack-dev-middleware</code> to v8 and sync <code>originalUrl</code> for middleware compatibility. <code>server.middleware.getFilenameFromUrl()</code> is now asynchronous and resolves to <code>{ filename, extra: { stats, outputFileSystem } }</code>. See the <a href="https://github.com/webpack/webpack-dev-middleware/releases/tag/v8.0.0">webpack-dev-middleware v8 release notes</a> for details. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Minor Changes</h3> <ul> <li> <p>Add plugin support. <code>webpack-dev-server</code> can now be used as a webpack plugin, integrating with the compiler lifecycle without explicitly passing a compiler, preventing multiple server starts on recompilation, ensuring clean shutdown, and supporting <code>MultiCompiler</code> setups with multiple independent plugin servers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Enable the compression middleware for HTTP/2 connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>colorette</code> dependency in favor of native ANSI styling. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>chokidar</code> to v5 and extend <code>watchFiles.options.ignored</code> to support glob string patterns via <code>tinyglobby</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use <code>compiler.platform</code> to determine the target environment instead of inspecting the resolved <code>target</code> string. Universal targets (<code>"universal"</code> or <code>["web", "node"]</code>, where <code>compiler.platform.universal</code> is <code>true</code> since webpack <code>5.108.0</code>) are treated as web targets so the client runtime is injected. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use the WHATWG <code>URL</code> API instead of the deprecated <code>url.parse</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Patch Changes</h3> <ul> <li> <p>Bump production dependencies, notably <code>open</code> to v11 and <code>p-retry</code> to v8. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Reject cross-site requests to the internal <code>open-editor</code> and <code>invalidate</code> endpoints. They performed state-changing actions (opening a file in the editor, forcing a recompilation) on any GET request, so a page the developer visited could trigger them. They now require a same-origin request, validated via <code>Sec-Fetch-Site</code> with an <code>Origin</code>/<code>Host</code> fallback. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5691">#5691</a>)</p> </li> <li> <p>Treat loopback aliases (<code>127.0.0.1</code>, <code>::1</code>, <code>localhost</code>) as equivalent in <code>isSameOrigin</code> so the WebSocket client does not reject valid same-origin connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Migrate the test suite from Jest to <code>node:test</code> and set up the jsdom environment. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/webpack/webpack-dev-server/blob/main/CHANGELOG.md">webpack-dev-server's changelog</a>.</em></p> <blockquote> <h2>6.0.0</h2> <h3>Major Changes</h3> <ul> <li> <p>Bump Express to v5. See the <a href="https://expressjs.com/en/guide/migrating-5.html">Express 5 migration guide</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Bump the <code>webpack</code> peer dependency range from <code>^5.0.0</code> to <code>^5.101.0</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Drop support for Node.js < 22.15.0. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Convert the source to native ES modules. The package keeps <code>"type": "module"</code> and now exposes both an ESM and a CommonJS build via the <code>exports</code> field: ESM consumers <code>import</code> the native <code>lib/</code>, while CommonJS consumers <code>require()</code> a transpiled <code>dist/</code> build — so the package works from both ESM and CommonJS, including environments where <code>require(ESM)</code> is not supported. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove CLI flags. Use the <code>serve</code> command from <code>webpack-cli</code> together with a configuration file or the programmatic API instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>internalIP</code> and <code>internalIPSync</code> static methods from <code>Server</code>. Resolve the local IP yourself if you need it. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>bypass</code> option from proxy configuration. Use the <code>router</code> or <code>context</code> options provided by <code>http-proxy-middleware</code> instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove SockJS support. The <code>webSocketServer</code> option no longer accepts <code>"sockjs"</code>; use the default <code>"ws"</code> transport instead. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>spdy</code> dependency. Use the built-in <code>node:http2</code> module via the <code>server</code> option for HTTP/2 support. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>http-proxy-middleware</code> to v4. See the <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v3.0.0">http-proxy-middleware v3 release notes</a> and <a href="https://github.com/chimurai/http-proxy-middleware/releases/tag/v4.0.0">v4 release notes</a> for the full list of breaking changes. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>webpack-dev-middleware</code> to v8 and sync <code>originalUrl</code> for middleware compatibility. <code>server.middleware.getFilenameFromUrl()</code> is now asynchronous and resolves to <code>{ filename, extra: { stats, outputFileSystem } }</code>. See the <a href="https://github.com/webpack/webpack-dev-middleware/releases/tag/v8.0.0">webpack-dev-middleware v8 release notes</a> for details. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Minor Changes</h3> <ul> <li> <p>Add plugin support. <code>webpack-dev-server</code> can now be used as a webpack plugin, integrating with the compiler lifecycle without explicitly passing a compiler, preventing multiple server starts on recompilation, ensuring clean shutdown, and supporting <code>MultiCompiler</code> setups with multiple independent plugin servers. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Enable the compression middleware for HTTP/2 connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Remove the <code>colorette</code> dependency in favor of native ANSI styling. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Update <code>chokidar</code> to v5 and extend <code>watchFiles.options.ignored</code> to support glob string patterns via <code>tinyglobby</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use <code>compiler.platform</code> to determine the target environment instead of inspecting the resolved <code>target</code> string. Universal targets (<code>"universal"</code> or <code>["web", "node"]</code>, where <code>compiler.platform.universal</code> is <code>true</code> since webpack <code>5.108.0</code>) are treated as web targets so the client runtime is injected. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Use the WHATWG <code>URL</code> API instead of the deprecated <code>url.parse</code>. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <h3>Patch Changes</h3> <ul> <li> <p>Bump production dependencies, notably <code>open</code> to v11 and <code>p-retry</code> to v8. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Reject cross-site requests to the internal <code>open-editor</code> and <code>invalidate</code> endpoints. They performed state-changing actions (opening a file in the editor, forcing a recompilation) on any GET request, so a page the developer visited could trigger them. They now require a same-origin request, validated via <code>Sec-Fetch-Site</code> with an <code>Origin</code>/<code>Host</code> fallback. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5691">#5691</a>)</p> </li> <li> <p>Treat loopback aliases (<code>127.0.0.1</code>, <code>::1</code>, <code>localhost</code>) as equivalent in <code>isSameOrigin</code> so the WebSocket client does not reject valid same-origin connections. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> <li> <p>Migrate the test suite from Jest to <code>node:test</code> and set up the jsdom environment. (by <a href="https://github.com/bjohansebas"><code>@bjohansebas</code></a> in <a href="https://redirect.github.com/webpack/webpack-dev-server/pull/5674">#5674</a>)</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/webpack/webpack-dev-server/commit/05cb7921b2cd216f8caa74d2bdcc39b3f0d05bea"><code>05cb792</code></a> chore(release): new release (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5692">#5692</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/a451839256733b37c8b71d15b9e22a55db3290d1"><code>a451839</code></a> fix: handle middleware teardown in plugin mode (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5703">#5703</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/c2d23a7e9f8562c634b3f3cabf8cee18123a402b"><code>c2d23a7</code></a> fix: load ESM-only dependencies with native import() in the CommonJS build (#...</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/ba54764c65464b43ab695ab8e7f20ca8e6c92e8d"><code>ba54764</code></a> fix: reject cross-site requests to open-editor and invalidate endpoints (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5691">#5691</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/2b369b39d537331fc44df3d3c1d62f2ac4598016"><code>2b369b3</code></a> fixup!</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/08a0ea7e92b2f954d2ab88e4b9669a45108cf0c8"><code>08a0ea7</code></a> fix: ensure undefined options default to an empty object in Server constructor</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/797b9e7d780eb5202895021bbd47253774a0135d"><code>797b9e7</code></a> fix: handle undefined options in Server constructor</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/e90221cd422dbaa71b3a870538be1c88caf98361"><code>e90221c</code></a> feat: plugin support (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5650">#5650</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/4c351e1b45f6a0bcc5fb3ef1e6bfad2d0c4fdd8b"><code>4c351e1</code></a> feat: support universal platform as a web target (<a href="https://redirect.github.com/webpack/webpack-dev-server/issues/5690">#5690</a>)</li> <li><a href="https://github.com/webpack/webpack-dev-server/commit/2236aa4b490a0b6abfd9e234beb3da5a32da91e7"><code>2236aa4</code></a> chore: update http-proxy-middleware to version 4.1.1 and add tests for pathRe...</li> <li>Additional commits viewable in <a href="https://github.com/webpack/webpack-dev-server/compare/v5.2.6...v6.0.0">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/datafusion/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
## Which issue does this PR close? - Closes apache#23507. ## Rationale for this change Part of epic apache#23494. Moves `HashJoinExec` protobuf serialization from central dispatch. ## What changes are included in this PR? Add protobuf serialization and deserialization to the `HashJoinExec` physical plan implementation and deprecate the corresponding central proto methods. The wire format remains unchanged. ## Are these changes tested? Yes, existing HashJoin round-trip tests cover this plan. ## Are there any user-facing changes? Existing methods are deprecated with no immediate API change.
…che#23825) Closes apache#23514. Part of apache#23494. Migrate `AsyncFuncExec` proto encode/decode into the plan itself via `try_to_proto` / `try_from_proto`, removing its central-arm handling in `physical_plan/mod.rs`. ## Rationale `datafusion-proto` currently downcasts `AsyncFuncExec` in the central encode match and rebuilds it inline on decode. apache#23495 introduced self-serializing hooks so each plan owns its own wire format. This moves `AsyncFuncExec` onto that pattern. ## Changes - Added `try_to_proto` / `AsyncFuncExec::try_from_proto`, wired into the decode dispatch - Removed the central encode downcast branch - Old helper methods kept as `#[deprecated]` stubs, per existing convention - Wire format unchanged ## Testing Existing `roundtrip_async_func_exec` integration test now exercises the new hooks (old path deleted, so it's the only path left). `cargo fmt` + `cargo clippy --all-targets --features proto -- -D warnings` clean. ## User-facing changes No Co-authored-by: Matthew Patton <matthewpatton@macbookpro.mynetworksettings.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23513. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Window plans still relied on centralized protobuf dispatch, keeping serialization separate from the execution plans that own their state. Both window executors share one protobuf variant. Decoding must inspect `input_order_mode` before selecting the concrete plan. Add plan-local encoders for both executors and a decoder for the shared node. Keep the legacy helpers as deprecated delegates while preserving window frames and UDF payloads on the existing wire format. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23511. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Physical plan protobuf serialization is being moved from the central dispatch module into each `ExecutionPlan` implementation. Co-locating this logic with the corresponding execution plan makes the serialization code easier to maintain and incrementally reduces the central downcast chain. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Implement plan-local protobuf serialization and deserialization for `ExplainExec`. - Implement plan-local protobuf serialization and deserialization for `AnalyzeExec`. - Remove both plans from the live central serialization dispatch. - Retain the deprecated compatibility methods as delegates to the new implementations. - Preserve the existing protobuf wire format. - Strengthen roundtrip tests to cover all stored fields, including every `StringifiedPlan` variant, metric categories, and explain formats. Each execution plan migration is kept in a separate commit. ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. The following checks pass: - `cargo fmt --all` - `cargo clippy --all-targets --all-features -- -D warnings` - Full extended workspace test suite - `cargo test -p datafusion-proto --test proto_integration` The proto integration suite passes all 209 tests after rebasing onto the latest `main`. ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> No. This is an internal refactor and does not change the protobuf wire format or user-facing behavior. --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23512. ## Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Aggregate protobuf conversion lived in the central proto crate, which prevented AggregateExec from owning its function-codec serialization. Move encoding and decoding into AggregateExec and keep the deprecated helpers as delegates so existing callers retain wire-compatible behavior. ## What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> ## Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. --> <!-- If there are any breaking changes to public APIs, please add the `api change` label. --> --------- Signed-off-by: Jiawei Zhao <Phoenix500526@163.com>
…nparser (apache#23789) ## Which issue does this PR close? - Closes apache#23668. ## Rationale for this change - When printing a GROUP BY query, the unparser sometimes wraps the aggregate's input in an inner subquery (`... FROM (SELECT ...)`). - Table aliases like `cs` only exist inside that subquery. Any clause outside it (SELECT, GROUP BY, HAVING, QUALIFY, ORDER BY) must use the subquery's output columns, not the aliases, so the unparser drops the alias. - Each clause does that dropping on its own, so it is easy to miss one. - ORDER BY on an aggregate that is not in the SELECT list was missed: it printed `ORDER BY round(sum("cs"."total_revenue"), 2)`, while the SELECT list in the same query correctly printed `sum("total_revenue")`. - DataFusion reads that SQL back fine, but stricter databases reject it because `cs` is out of scope there. ## What changes are included in this PR? I moved the rule into one helper, `UnparserAggScope`, that checks once per aggregate whether the input is an inner subquery and then prepares expressions for each clause. SELECT, GROUP BY, HAVING, QUALIFY, and both ORDER BY paths now go through it, which fixes the ORDER BY case. The window-over-aggregate path is left as-is with a comment: it is only reachable from hand-built plans, since a window in SQL always sits inside a SELECT that already drops the alias. Only ORDER BY output changes; every already-correct case stays the same. ## Are these changes tested? Yes. New tests in `datafusion/core/tests/sql/unparser.rs` cover the inner-subquery shape for a window sorting by an aggregate, ORDER BY on an unselected aggregate (the fixed case), and a top-level ORDER BY (the second sort path). Existing unparser tests and the TPC-H/Clickbench roundtrips still pass. ## Are there any user-facing changes? ORDER BY over an aggregate is now printed without an out-of-scope table qualifier, so the generated SQL is valid for stricter databases. No API changes.
…che#22980) ## Which issue does this PR close? * Part of apache#20835 ## Rationale for this change `FixedSizeList` containing `Struct` values was not handled by the existing recursive nested adaptation logic used for schema evolution. As a result, planner-time compatibility checks, nested cast detection, and runtime casting did not support additive struct evolution within `FixedSizeList` containers. This change adds `FixedSizeList` support and verifies planner/runtime parity so that planning allows exactly the cases runtime can adapt while continuing to reject incompatible schema changes. ## What changes are included in this PR? * Extend `cast_column` to support recursive casting of `FixedSizeList` values when source and target list sizes match. * Add `FixedSizeList` handling to: * `requires_nested_struct_cast` * `validate_data_type_compatibility` * Implement recursive casting of nested `Struct` values contained in `FixedSizeList`. * Preserve planner/runtime parity by validating child type compatibility before runtime fallback logic is applied. * Add handling for null-parent `FixedSizeList` entries by masking hidden child values before retrying casts, avoiding failures caused by semantically inaccessible child data. * Refactor list and list-view casting helpers to use Arrow `AsArray` accessors. ## Are these changes tested? Yes. The following tests were added: * `test_cast_fixed_size_list_struct` * `test_validate_fixed_size_list_struct_compatibility` * `test_validate_fixed_size_list_struct_missing_non_nullable_field_rejected` * `test_validate_fixed_size_list_struct_size_mismatch_rejected` * `test_cast_fixed_size_list_struct_all_null` * `test_fixed_size_list_struct_planner_runtime_parity_on_incompatible_type` * `test_cast_fixed_size_list_struct_missing_non_nullable_field_runtime_rejected` * `test_cast_fixed_size_list_struct_ignores_hidden_child_values_for_null_parent` Existing coverage in `test_requires_nested_struct_cast` was also extended to include `FixedSizeList` cases. These tests cover: * Additive nullable nested-field evolution * All-null and partially null list cases * Incompatible nested type changes * Non-nullable field addition rejection * Planner/runtime parity validation ## Are there any user-facing changes? No user-facing changes. This is an internal enhancement to nested schema adaptation and casting behavior for `FixedSizeList<Struct>` types. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
… spark (apache#23766) ## Which issue does this PR close? N/A ## Rationale for this change Hex encoding was implemented six times across the workspace, at two different levels of optimization. Three copies used a fast byte-pair lookup table: - `datafusion/spark/src/function/math/hex.rs` - `datafusion/functions/src/string/to_hex.rs` - `datafusion/functions/src/encoding/inner.rs` (via the `hex` crate) The other three used a slower nibble-at-a-time loop pushing one character at a time: - `datafusion/functions/src/crypto/md5.rs` - `datafusion/spark/src/function/hash/sha1.rs` - `datafusion/spark/src/function/hash/sha2.rs` Beyond the duplication, this split meant the digest functions were paying for a slower encoder than the one already sitting elsewhere in the tree. Consolidating on a single implementation removes the duplication and moves `md5`, `sha1`, and `sha2` onto the fast path. ## What changes are included in this PR? A new `datafusion_common::utils::hex` module holds the only hex encoder in the workspace: ```rust pub enum HexCase { Lower, Upper } pub fn encode_bytes_into(bytes: &[u8], case: HexCase, out: &mut Vec<u8>); pub fn encode_bytes(bytes: &[u8], case: HexCase) -> String; pub fn encode_bytes_to_slice(bytes: &[u8], case: HexCase, out: &mut [u8]); pub fn encode_u64(v: u64, case: HexCase, buf: &mut [u8; 16]) -> &[u8]; ``` Four entry points rather than one because the call sites have genuinely different output needs, and forcing them through a single shape would cost an allocation somewhere: `to_hex` writes straight into a `StringArray` values buffer, Spark's `hex` appends to a reused scratch `Vec`, `encode` writes into a pre-sized slice, and the digest functions want an owned `String`. Migrated call sites, all producing bit-identical output: | File | Change | | --- | --- | | `functions/src/string/to_hex.rs` | local table and two write helpers deleted; eight trait impls collapsed to two macros | | `spark/src/function/math/hex.rs` | two nibble tables, two lookup tables, `build_hex_lookup` and `hex_int64` deleted | | `functions/src/crypto/md5.rs` | local table and `hex_encode` deleted | | `spark/src/function/hash/sha1.rs` | local table and inline nibble loop deleted | | `spark/src/function/hash/sha2.rs` | local table and `hex_encode` deleted; eight call sites migrated | | `functions/src/encoding/inner.rs` | both encode sites moved off the `hex` crate | Deliberately left alone: - `hex::decode` / `hex::decode_to_slice` in `encoding/inner.rs`, and Spark's `unhex`. The decode direction has different semantics — Spark's `unhex` left-pads odd-length input, the `hex` crate does not — so unifying it is a separate question. The `hex` dependency stays for those. - `ScalarValue`'s binary `Display` impl in `common/src/scalar/mod.rs`, which writes to a `fmt::Formatter` rather than a byte buffer, and is a cold path. Three details worth a reviewer's attention: - The two ancestors of `encode_u64` disagreed on zero: `to_hex` wrote `'0'` into the caller's buffer, Spark's `hex_int64` returned a `'static` `b"0"` that never touched it. The shared version always writes into the buffer and returns a subslice of it, so the lifetime is uniform. Both callers still produce `"0"`. - Spark's `hex_encode_bytes` guards large binary input with `checked_mul(2)` + `try_reserve`, returning a `DataFusionError` rather than aborting on allocation failure. That guard stays at the call site; `encode_bytes_into` performs no reservation of its own, so behaviour is unchanged. - The encoders are `#[inline(always)]`, not `#[inline]`. They are called once per row from two other crates, and plain `#[inline]` left them out-of-line across the crate boundary. Measured cost of that: +3% on Spark's byte paths and up to +18% on `to_hex`'s i32 path, i.e. the refactor was a net regression on those benchmarks until the attribute changed. ## Are these changes tested? Every migrated function keeps its existing unit and sqllogictest coverage, which is what pins bit-identical output — in particular `spark/hash/sha1.slt` and `sha2.slt` assert concrete digest strings for all four SHA-2 bit lengths across both the scalar and array paths, and `expr.slt` covers `to_hex` and `md5`. New unit tests in `common/src/utils/hex.rs` cover zero, `u64::MAX`, single-nibble values, the odd/even digit-count boundary, two's complement of negative input, empty input, all 256 byte values in both cases, appending into a non-empty buffer, and that a reused scratch buffer never leaks stale digits between calls. Tests cross-check against `format!("{:x}")` rather than restating the implementation. Two Spark tests changed. `test_hex_int64` now drives `hex_encode_int64` instead of the deleted private `hex_int64`, keeping all ten cases including `i64::MIN`, `-1`, and the uppercase expectations. `test_hex_lookup_table_covers_all_bytes` was deleted — it only cross-checked the raw lookup tables, and `encode_bytes_covers_every_byte_value` now does that exhaustively through the public API. A test was added for Spark's lowercase byte path, which previously had no coverage. ### Benchmarks Criterion, `apache/main` @ `eef101769` as baseline. Median of the reported change interval. `datafusion/functions/benches/to_hex.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `i32_random` | −13.8% | −12.8% | −10.2% | | `i64_random` | −8.8% | −9.5% | −8.1% | | `i64_large_values` | −9.9% | −9.8% | −7.9% | `scalar_i32` −2.3%, `scalar_i64` −1.8%. `datafusion/spark/benches/sha2.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `array_binary_256` | −20.7% | −18.2% | −19.1% | | `array_scalar_binary_256` | −14.6% | −13.2% | −13.1% | `scalar/size=1` −3.3%. `datafusion/spark/benches/hex.rs`: | Benchmark | 1024 | 4096 | 8192 | | --- | --- | --- | --- | | `hex_int64` | −2.9% | −3.6% | −3.5% | | `hex_int64_dict` | −3.2% | −1.5% | −0.2% | | `hex_utf8` | −0.2% | −0.3% | +0.7% | | `hex_binary` | −0.4% | −0.6% | −0.3% | The `hex_utf8` and `hex_binary` paths already used the byte-pair table before this PR, so they are expected to be flat; they are. `datafusion/functions/benches/crypto.rs`: `md5_array` −4.0%, `md5_scalar` −3.7%. The `sha224` / `sha256` / `sha384` / `sha512` cases in that file range from −0.1% to +2.2%, but they exercise `crypto/basic.rs`, which this PR does not modify and which contains no hex encoding — those numbers are run-to-run variance, not an effect of this change. No number is quoted for Spark `sha1`: it has no benchmark, and its change is the same substitution applied to `md5` and `sha2`. ## Are there any user-facing changes? No behaviour change — all migrated functions produce byte-identical output. `datafusion_common::utils::hex` is new public API on `datafusion-common`. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
…t_batch` memory (apache#23873) ## Which issue does this PR close? - related to apache#23716 ## Rationale for this change It adds test coverage for two gaps found while reviewing apache#23716. ## What changes are included in this PR? Tests only, no functional change. - Dictionary inputs - memory usage on retract (make sure memory is released) Note I moved `array_agg` cases out `aggregate.slt` as it is already more than 9k lines long ## Are these changes tested? They are only tests ## Are there any user-facing changes? No. Tests only, no public API changes. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…3881) ## Which issue does this PR close? N/A ## Rationale for this change Two Spark functions allocated a `String` per row purely to render a small, bounded amount of text: - `bin` called `format!("{value:b}")` for every row. - `char` called `ch.to_string()` for every row — a heap allocation for a single character. In both cases the output has a known upper bound (64 binary digits for an `i64`, 4 bytes for a UTF-8 character), so the rendering fits in a stack buffer and the result can be appended straight to a pre-sized builder. ## What changes are included in this PR? `math/bin.rs`: - `spark_bin` now writes digits right-aligned into a caller-supplied `[u8; 64]` and returns a `&str` borrowed from it, instead of returning an owned `String`. - The `collect::<StringArray>()` becomes an explicit loop over a `StringBuilder::with_capacity`, sized at 8 digits per row. - Negative values still render as their two's-complement bit pattern, matching `{:b}`. The digit loop is a `loop`, not a `while`, so zero renders as `"0"` rather than the empty string. `string/char.rs`: - `ch.to_string()` becomes `ch.encode_utf8(&mut encoded)` against a `[u8; 4]` hoisted out of the loop. Output is unchanged in both cases. ## Are these changes tested? Existing coverage pins the behaviour. `spark/math/bin.slt` asserts concrete output for the cases the rewrite had to get right: zero, negative values, `i64::MIN` (`-9223372036854775808`), `i64::MAX`, and `-2147483648` / `-32768` widened from narrower integer types. `spark/string/char.slt` covers the negative-input empty string, the null path, and characters on both sides of the ASCII boundary (`char(256)` and above wrap via `% 256`). All 119 `spark/math` and `spark/string` sqllogictest files pass, along with the 258 `datafusion-spark` unit tests. The `bin` benchmark used below, `datafusion/spark/benches/bin.rs`, is added separately in apache#23882 so the baseline can be measured on `main` before this change lands. It covers 1024 and 8192 rows with 20% nulls over two value distributions: small values that render to a handful of digits, and full-range values that render to the maximum 64. `char` already had `datafusion/spark/benches/char.rs` on `main`, so no benchmark change is needed for it. ### Benchmarks Criterion, `apache/main` @ `f1ab86dad` as baseline. Median of the reported change interval. | Benchmark | 1024 | 8192 | | --- | --- | --- | | `bin/small` | −75.7% | −73.7% | | `bin/wide` | −47.8% | −49.8% | `char` (1024 rows): −76.1%. ## Are there any user-facing changes? No. Both functions produce byte-identical output; this is purely an allocation change.
…ts enabled (apache#23848) ## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Closes apache#23847 . ## Rationale for this change Fixes a correctness bug, not sure when that config used/desirable. This is another thing I ran into during apache#21585 ## What changes are included in this PR? 1. New SLT test 2. Fix for bug in `datafusion-optimizer` ## Are these changes tested? Existing tests, and a new SLT test verifying the config change doesn't affect the query result and basic unit test. ## Are there any user-facing changes? None --------- Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
…lter (apache#23901) ## Which issue does this PR close? - Closes apache#23900. ## Rationale for this change `push_down_filter` infers equi-key predicates across a join's ON keys and pushes them to the opposite side. For a null-aware join (the `LeftAnti` join produced by `NOT IN` with a nullable subquery), an outer predicate on the left key like `outer.id > 5` is rewritten to `sub.id > 5` and pushed onto the subquery input. Since the inferred predicate must be null-rejecting to be pushed, this drops the subquery's NULL rows and breaks the three-valued `NOT IN` semantics — a NULL in the subquery key must reach the join so the result is empty. Same class of bug as apache#23848, in a different rule. ## What changes are included in this PR? - Skip predicate inference in `infer_join_predicates` when `join.null_aware` is set (mirrors the apache#23848 guard on `FilterNullJoinKeys`). - A `push_down_filter` unit test asserting no predicate is inferred onto the subquery side of a null-aware `LeftAnti` join. - SLT coverage for the failing query, plus a `prefer_hash_join = false` / multi-partition variant. ## Are these changes tested? Yes — new unit test (verified it fails without the guard) and SLT cases. The full optimizer lib suite passes. ## Are there any user-facing changes? No, aside from the correctness fix.
## Which issue does this PR close? - Part of apache#19241. - Stacked on [apache#23311](apache#23311). - Next in stack: apache#23015. - Extracted from apache#19390. ## Rationale for this change For very small `IN` lists, building or probing a hash table can be more work than just comparing the input value with each constant. For example, for `x IN (10, 20, 30)`, the fast path can behave like: ```text x == 10 OR x == 20 OR x == 30 ``` Because the list is tiny, those comparisons are cheap. The implementation stores the constants in a fixed-size array and checks them with a compact comparison chain. “Branchless” here means the comparisons are combined without stopping at the first match. That can be faster for these small fixed-width lists because the CPU gets a predictable sequence of simple operations instead of hash-table setup and probe logic. For primitive values that are not already plain unsigned integers, this PR keeps the logical Arrow type explicit and uses a matching same-width comparison representation only inside the branchless filter. For example, `Float16` uses `UInt16` storage, `Float32` uses `UInt32` storage, and `TimestampNanosecond` uses `UInt64` storage. `Decimal128` and `IntervalMonthDayNano` use their own 16-byte native representation. This preserves bit-pattern equality while relying on Arrow's native primitive compatibility rules: timestamp timezone metadata and Decimal128 precision/scale metadata may differ, while incompatible primitive representations remain rejected. ## What changes are included in this PR? - Adds a const-generic `BranchlessFilter` for small primitive `IN` lists. - Adds thresholds for when this path is used: - up to 16 values for 1-byte types - up to 8 values for 2-byte types - up to 32 values for 4-byte types - up to 16 values for 8-byte types - up to 4 values for 16-byte types - Keeps dispatch concrete and explicit in `strategy.rs`. - Maps each optimized logical type to the comparison representation used by the branchless filter: - `Int8` -> `UInt8` - `Int16`, `Float16` -> `UInt16` - `Int32`, `Float32`, `Date32`, `Time32` -> `UInt32` - `Int64`, `Float64`, `Date64`, `Time64`, `Timestamp`, `Duration` -> `UInt64` - `Decimal128`, `IntervalMonthDayNano` -> their native 16-byte representation - Leaves larger 1-byte and 2-byte lists on the existing bitmap filters. - Leaves larger 4-byte and 8-byte lists on the existing hash/generic paths. - Leaves wider primitive types such as `Decimal256` and unsupported complex types on the generic path. - Keeps the same `IN` / `NOT IN` null behavior as the rest of the stack. - Adds focused coverage for branchless null handling, signed boundary values, slices, Float16/Float32/Float64 bit patterns, compatible timestamp/Decimal128 metadata, incompatible timestamp units, IntervalMonthDayNano values, and same-width wrong-type probe rejection. ## Are these changes tested? Yes. - `cargo fmt --all -- --check` - `cargo test -p datafusion-physical-expr expressions::in_list --lib` - `cargo test -p datafusion-physical-expr --bench in_list_strategy --no-run` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? No. This is an internal performance optimization only. ## Local benchmark snapshot Built and run with `release-nonlto`, filtered to the relevant small primitive-list rows: ```bash cargo bench -p datafusion-physical-expr --profile release-nonlto --bench in_list_strategy -- <filter> --save-baseline <baseline> ``` Filters used: `narrow_integer`, `primitive/i32/small_list`, `primitive/i64/small_list`, `f32/small_list`, `timestamp_ns/small_list`, and `interval_month_day_nano/small_list`. Method: directly compared Criterion's raw sample minima (`min(time / iterations)`) from `sample.json`. Lower is better; changes within +/-5% are treated as noise. Compared baselines: [apache#23311](apache#23311) -> [apache#23014](apache#23014) Relevant scope: small primitive-list rows. Summary: 39 relevant rows, 28 faster, 0 slower, 11 within +/-5%. Largest relevant deltas: | Benchmark | Before | After | Change | |---|---:|---:|---:| | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | <details> <summary>Full relevant table (39 rows)</summary> | Benchmark | Before | After | Change | |---|---:|---:|---:| | `narrow_integer/u8/list=4/match=0%` | 3.86 us | 2.79 us | -27.8% (1.38x faster) | | `narrow_integer/u8/list=4/match=50%` | 3.84 us | 2.78 us | -27.7% (1.38x faster) | | `narrow_integer/u8/list=16/match=0%` | 3.88 us | 3.85 us | -0.8% (within noise) | | `narrow_integer/u8/list=16/match=50%` | 3.84 us | 3.86 us | +0.5% (within noise) | | `narrow_integer/i16/list=4/match=0%` | 3.93 us | 3.18 us | -19.1% (1.24x faster) | | `narrow_integer/i16/list=4/match=50%` | 3.92 us | 3.16 us | -19.5% (1.24x faster) | | `narrow_integer/i16/list=64/match=0%` | 3.96 us | 3.82 us | -3.5% (within noise) | | `narrow_integer/i16/list=64/match=50%` | 3.91 us | 3.80 us | -2.9% (within noise) | | `narrow_integer/i16/list=256/match=0%` | 3.90 us | 3.81 us | -2.5% (within noise) | | `narrow_integer/i16/list=256/match=50%` | 3.97 us | 3.81 us | -4.1% (within noise) | | `narrow_integer/f16/list=4/match=0%` | 3.87 us | 3.16 us | -18.5% (1.23x faster) | | `narrow_integer/f16/list=4/match=50%` | 3.94 us | 3.15 us | -20.2% (1.25x faster) | | `narrow_integer/f16/list=64/match=0%` | 3.87 us | 3.84 us | -0.6% (within noise) | | `narrow_integer/f16/list=64/match=50%` | 3.93 us | 3.85 us | -1.9% (within noise) | | `narrow_integer/f16/list=256/match=0%` | 3.90 us | 3.84 us | -1.5% (within noise) | | `narrow_integer/f16/list=256/match=50%` | 3.87 us | 3.91 us | +1.2% (within noise) | | `nulls/narrow_integer/u8/list=16/match=50%/nulls=20%` | 3.92 us | 4.02 us | +2.5% (within noise) | | `primitive/i32/small_list/list=4/match=0%` | 17.00 us | 3.04 us | -82.1% (5.59x faster) | | `primitive/i32/small_list/list=4/match=50%` | 32.63 us | 3.08 us | -90.5% (10.58x faster) | | `primitive/i32/small_list/list=32/match=0%` | 16.34 us | 13.33 us | -18.5% (1.23x faster) | | `primitive/i32/small_list/list=32/match=50%` | 31.17 us | 13.31 us | -57.3% (2.34x faster) | | `primitive/i32/small_list/list=16/match=50%/NOT_IN` | 31.98 us | 7.26 us | -77.3% (4.41x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%` | 29.35 us | 7.32 us | -75.1% (4.01x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=20%/NOT_IN` | 26.05 us | 7.42 us | -71.5% (3.51x faster) | | `nulls/primitive/i32/small_list/list=16/match=50%/nulls=50%` | 25.89 us | 7.31 us | -71.8% (3.54x faster) | | `primitive/i64/small_list/list=4/match=0%` | 17.12 us | 3.22 us | -81.2% (5.31x faster) | | `primitive/i64/small_list/list=4/match=50%` | 33.55 us | 3.18 us | -90.5% (10.54x faster) | | `primitive/i64/small_list/list=16/match=0%` | 16.34 us | 11.93 us | -27.0% (1.37x faster) | | `primitive/i64/small_list/list=16/match=50%` | 29.46 us | 11.76 us | -60.1% (2.50x faster) | | `f32/small_list/list=4/match=0%` | 18.14 us | 3.05 us | -83.2% (5.95x faster) | | `f32/small_list/list=4/match=50%` | 33.93 us | 3.04 us | -91.0% (11.15x faster) | | `f32/small_list/list=32/match=0%` | 22.05 us | 13.43 us | -39.1% (1.64x faster) | | `f32/small_list/list=32/match=50%` | 38.78 us | 13.27 us | -65.8% (2.92x faster) | | `timestamp_ns/small_list/list=4/match=0%` | 19.57 us | 3.18 us | -83.8% (6.16x faster) | | `timestamp_ns/small_list/list=4/match=50%` | 46.55 us | 3.17 us | -93.2% (14.69x faster) | | `timestamp_ns/small_list/list=16/match=0%` | 19.73 us | 12.07 us | -38.8% (1.63x faster) | | `timestamp_ns/small_list/list=16/match=50%` | 45.32 us | 11.79 us | -74.0% (3.84x faster) | | `interval_month_day_nano/small_list/list=4/match=0%` | 20.12 us | 13.20 us | -34.4% (1.52x faster) | | `interval_month_day_nano/small_list/list=4/match=50%` | 52.94 us | 15.52 us | -70.7% (3.41x faster) | </details> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
## Which issue does this PR close? - Closes apache#23770 - Part of apache#15914 ## Rationale for this change Spark provides [`hypot(expr1, expr2)`](https://spark.apache.org/docs/latest/api/sql/#hypot), which returns `sqrt(expr1^2 + expr2^2)` computed without intermediate overflow or underflow. It was not yet implemented in `datafusion-spark` — only an auto-generated test stub existed at `spark/math/hypot.slt` with its query commented out. ## What changes are included in this PR? - Add `SparkHypot` (implementing `ScalarUDFImpl`) in `datafusion/spark/src/function/math/hypot.rs`, backed by Rust's `f64::hypot` — the same overflow-safe algorithm as Java/Spark's `Math.hypot`. - Register it in `datafusion/spark/src/function/math/mod.rs`. - Enable the `hypot.slt` sqllogictest. The signature is `exact(Float64, Float64) -> Float64`, following the `datafusion-spark` convention of only accepting types Spark supports. Computation uses the Arrow `binary` kernel so NULL in either argument propagates to a NULL result, matching Spark. ## Are these changes tested? Yes — `datafusion/sqllogictest/test_files/spark/math/hypot.slt` covers: - scalar Pythagorean triples (`hypot(3, 4)` → 5, `hypot(5, 12)` → 13), - double inputs, - NULL propagation when either argument is NULL, - the array path (including a NULL row), - overflow-safety: `hypot(3e200, 4e200)` stays finite, whereas a naive `sqrt(a^2 + b^2)` would overflow to `Infinity`. ## Are there any user-facing changes? Yes — adds the Spark-compatible `hypot` scalar function to `datafusion-spark`. No breaking changes to public APIs.
## Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes apache#123` indicates that this PR will close issue apache#123. --> - Part of apache#23393 . ## Rationale for this change `SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` only reported the stack size of their `ScalarValue` field plus its heap payload, ignoring the memory held by the underlying `MovingMin` / `MovingMax` sliding-window buffers. For windowed `MIN`/`MAX` over string or list data, the two per-element stacks can hold megabytes of `ScalarValue` payload that the memory pool was never told about, understating accumulator memory usage. ## What changes are included in this PR? - Add a private `heap_size(elem_heap)` method to `MovingMin<T>` and `MovingMax<T>` that reports the two stack buffers' capacity in bytes plus each stored element's heap payload. - Factor the shared implementation into a `moving_stacks_heap_size` free helper so the two types stay in sync. - Include the buffer bytes in `SlidingMinAccumulator::size` and `SlidingMaxAccumulator::size` via the new method. ## Are these changes tested? Yes. Two new unit tests in `datafusion/functions-aggregate/src/min_max.rs`: - `moving_min_max_heap_size_i32` — fixed-width `T`, verifies buffer-only accounting with and without pushed elements. - `moving_min_max_heap_size_counts_elems` — `T = String`, verifies each of the two slots in a `(T, T)` pair contributes independently to the heap payload (mirroring the two independent `Clone`s made by `push`).
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com> (cherry picked from commit 717ff90)
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com> (cherry picked from commit bedca33)
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com> (cherry picked from commit db3fb06)
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com> (cherry picked from commit 2d7bac5)
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com> (cherry picked from commit 693aa0b)
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Signed-off-by: Ethan Urbanski <ethan@urbanskitech.com>
Comment on lines
+27
to
+62
| name: No native compiler | ||
| runs-on: ubuntu-latest | ||
| container: | ||
| image: debian:bookworm-slim | ||
| env: | ||
| CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER: ${{ github.workspace }}/ci/scripts/compiler-free-linker.sh | ||
| steps: | ||
| - name: Install transport tools | ||
| run: | | ||
| apt-get update | ||
| apt-get install --yes --no-install-recommends ca-certificates curl git libc6-dev libgcc-12-dev lld | ||
| rm -rf /var/lib/apt/lists/* | ||
| - uses: actions/checkout@v6 | ||
| - name: Install Rust and wasm32 | ||
| run: | | ||
| curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | | ||
| sh -s -- -y --profile minimal --default-toolchain 1.94.0 | ||
| "$HOME/.cargo/bin/rustup" target add wasm32-unknown-unknown | ||
| echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" | ||
| - name: Verify minimal toolchain | ||
| run: | | ||
| test -z "${RUSTFLAGS:-}" | ||
| for tool in cc gcc clang c++ g++ cmake; do | ||
| if command -v "$tool"; then | ||
| echo "unexpected native build tool: $tool" >&2 | ||
| exit 1 | ||
| fi | ||
| done | ||
| command -v ld.lld | ||
| test -x "$CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER" | ||
| - name: Check browser graph | ||
| run: | | ||
| cargo check -p datafusion --target wasm32-unknown-unknown --locked | ||
| ci/scripts/check_wasm_dependency_policy.sh | ||
|
|
||
| browser-runtime: |
Comment on lines
+63
to
+97
| name: ${{ matrix.browser }} | ||
| runs-on: ubuntu-24.04 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| browser: | ||
| - chrome | ||
| - firefox | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| - run: rustup target add wasm32-unknown-unknown | ||
| - uses: taiki-e/install-action@v2 | ||
| with: | ||
| tool: wasm-pack | ||
| - name: Run browser runtime profile | ||
| env: | ||
| RUSTFLAGS: -C debuginfo=none | ||
| run: | | ||
| node --test datafusion/wasmtest/tests/browser-server.test.mjs | ||
| node datafusion/wasmtest/tests/browser-server.mjs > browser-server.log 2>&1 & | ||
| server_pid=$! | ||
| trap 'kill "$server_pid"' EXIT | ||
| for attempt in $(seq 1 30); do | ||
| if curl --fail --silent --head \ | ||
| http://127.0.0.1:9876/tpch_region_small.parquet >/dev/null; then | ||
| break | ||
| fi | ||
| if [ "$attempt" -eq 30 ]; then | ||
| cat browser-server.log | ||
| exit 1 | ||
| fi | ||
| sleep 1 | ||
| done | ||
| DATAFUSION_WASM_HTTP_PARQUET_URL=http://127.0.0.1:9876/tpch_region_small.parquet \ | ||
| wasm-pack test --headless --${{ matrix.browser }} datafusion/wasmtest |
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Axon's native browser execution boundary needs a current-main DataFusion proof that does not silently enable filesystem-only behavior on
wasm32-unknown-unknown. This draft preserves the upstream-shaped feature and runtime commits, then pins the separately validated Daxis Arrow and object_store fork heads only in the final proof commit.What changes are included in this PR?
daxis-io/arrow-rs@3ef75b2ac57fb1cbd40a98c2bf29adb86fbdb578daxis-io/object_store@d0066c218eaf3336bc6b5e5ca3141fe78e4fea8dThe upstream-shaped runtime candidate is separately preserved at
upstream/2026-07-30/wasm32-browser-runtimewithout Daxis dependency pins. The smaller feature-ownership seam is preserved atupstream/2026-07-30/wasm32-feature-ownership.Are these changes tested?
Yes, on exact head
aebdb2958b5beb94f7891e3e8efbee169ac6f220:cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningscargo check -p datafusion --target wasm32-unknown-unknown --lockedbash ci/scripts/check_wasm_dependency_policy.shRUST_BACKTRACE=1, including 499/499 SQL logic filesAre there any user-facing changes?
Browser WASM consumers gain an explicit supported profile for in-memory execution and HTTP Parquet reads. Filesystem spilling and unavailable compression codecs return explicit target-aware errors. Native defaults remain intact.
This is a Daxis fork integration draft and is not a canonical Apache upstream submission.