fix(deps): upgrade Apache Arrow to 19.0.0 and Netty to 4.2.15.Final#6185
Conversation
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6185 +/- ##
============================================
- Coverage 59.94% 59.92% -0.02%
+ Complexity 3310 3306 -4
============================================
Files 1131 1131
Lines 43805 43805
Branches 4745 4745
============================================
- Hits 26257 26251 -6
- Misses 16125 16128 +3
- Partials 1423 1426 +3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
✅ No material benchmark regressions detected🟢 0 better · 🔴 0 worse · ⚪ 15 noise (<±5%) · 0 without baseline
Baseline detailsLatest main
Raw CSVconfig_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,487.48,200,128000,410,0.250,23401.26,32778.88,32778.88
1,100,10,64,20,2465.18,2000,1280000,811,0.495,120941.35,152892.88,152892.88
2,1000,10,64,20,21501.18,20000,12800000,930,0.568,1067991.88,1131797.63,1131797.63 |
Bumps [io.netty:netty-transport](https://github.com/netty/netty) from 4.1.96.Final to 4.2.15.Final. - [Release notes](https://github.com/netty/netty/releases) - [Commits](netty/netty@netty-4.1.96.Final...netty-4.2.15.Final) --- updated-dependencies: - dependency-name: io.netty:netty-transport dependency-version: 4.2.15.Final dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
815f937 to
810d9b5
Compare
There was a problem hiding this comment.
Copilot wasn't able to review any files in this pull request.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Dependabot bumped only netty-transport and netty-transport-native-unix-common to 4.2.15.Final, which split the Netty family across the 4.1/4.2 line. arrow-memory-netty reaches into Netty allocator internals, so the split broke the Arrow Flight transport to the Python workers at runtime (NoClassDefFoundError: io.netty.util.concurrent.ThreadAwareExecutor), failing amber-integration and Bench, and the direct-dep LICENSE-binary drift check failed the amber build. Apache Arrow 15.0.2 requires Netty 4.1.x; Arrow 19.0.0 is the first released version that targets the Netty 4.2 line. Coordinate the whole upgrade instead: - Arrow 15.0.2 -> 19.0.0 (common/workflow-core/build.sbt) - entire Netty family -> 4.2.15.Final in both override sites (root build.sbt amber module + common/workflow-core), pinned via a shared nettyVersion so the family can never split again - sync amber/LICENSE-binary-java for the changed netty and arrow jars
Arrow discontinued the standalone org.apache.arrow:flight-grpc artifact after 15.0.2 (its gRPC transport now lives in flight-core), so requesting flight-grpc:19.0.0 failed dependency resolution: not found: .../org/apache/arrow/flight-grpc/19.0.0/flight-grpc-19.0.0.pom Texera only uses org.apache.arrow.flight._ (provided by flight-core), so drop flight-grpc and keep flight-core. Sync the arrow jar bullets to 19.0.0 in the LICENSE-binary of every module whose dist bundles Arrow: amber, file-service, workflow-compiling-service, computing-unit-managing-service.
|
this is related to CVE. needs to backport to v1.2 |
|
Heads up — the plain ( Arrow
Test results on the latest push:
The runtime crash is gone — the Python-worker Arrow Flight transport comes up cleanly on both OSes and the arrow-flight benchmark runs end to end. Remaining red is |
…license/notice Arrow 19's transitive cascade (grpc 1.62->1.79, protobuf 3.25->4.33, netty 4.1->4.2 with netty-codec split into netty-codec-base/-compression, the new arrow-memory-netty-buffer-patch module, guava ->33.5.0-android, and eclipse-collections dropped) drifts LICENSE-binary/NOTICE-binary for every module whose dist bundles Arrow, not just amber. - Extract the Netty family override into a shared nettyDependencyOverrides val and apply it to amber + the three Arrow-bundling platform services (computing-unit-managing-service, file-service, workflow-compiling-service) so none of them can split across the 4.1/4.2 line. - Re-sync each module's LICENSE-binary (versions + added/removed jars) and regenerate NOTICE-binary with bin/licensing/generate_notice_binary.py. Verified locally against the resolved runtime classpaths: check_binary_deps.py passes for all four modules and each NOTICE-binary matches generator output. amber-integration (ubuntu + macos) and Bench already pass on the prior commit.
…erator Arrow 19's transitive dependencies evict jackson-databind up to 2.21.0, but jackson-module-scala stays pinned at 2.18.8, so the WorkflowCore and WorkflowOperator test suites aborted at initialization: Scala module 2.18.8 requires Jackson Databind version >= 2.18.0 and < 2.19.0 - Found jackson-databind version 2.21.0 amber already overrides jackson-databind; add the same jackson-core / -databind / -annotations override to the other two Arrow-bearing modules whose tests exercise jackson-module-scala. Verified locally: both resolve jackson-databind back to 2.18.8.
…15.Final Resolves conflicts in the LICENSE-binary / NOTICE-binary files between main's Apache Hadoop 3.4.3 upgrade (#6201) and this PR's Arrow 19 / Netty 4.2 upgrade. Both sides rewrote overlapping regions of the bundled-jar lists. Regenerated each file against the merged runtime classpath (Hadoop 3.4.3 + Arrow 19.0.0 + Netty 4.2.15) with bin/licensing tooling; check_binary_deps.py passes for amber and the three Arrow-bundling platform services.
…service apache#6185 (Arrow 19 upgrade) evicts jackson-databind up to 2.21.0, past the range jackson-module-scala 2.18.8 allows. The pin was applied to workflow-core/workflow-operator and to file-service/workflow-compiling-service, but computing-unit-managing-service only overrode jackson-module-scala, so its dist bundled jackson-databind-2.21.0.jar and crashed at startup with "Scala module 2.18.8 requires Jackson Databind version >= 2.18.0 and < 2.19.0". Add the jackson-databind override to ComputingUnitManagingService (matching file-service and workflow-compiling-service) and re-sync its LICENSE-binary. Fixes apache#6206
…service (apache#6209) ### What changes were proposed in this PR? `computing-unit-managing-service` crashes at startup on `main` (since apache#6185, the Arrow 19 upgrade) with: ``` Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Scala module 2.18.8 requires Jackson Databind version >= 2.18.0 and < 2.19.0 - Found jackson-databind version 2.21.0 at com.fasterxml.jackson.module.scala.JacksonModule.setupModule at org.apache.texera.service.ComputingUnitManagingService.initialize ``` **Root cause.** apache#6185 counters Arrow 19 evicting `jackson-databind` up to `2.21.0` (past the range `jackson-module-scala` allows) by pinning the Jackson core family — in `workflow-core`/`workflow-operator`, and by pinning `jackson-databind` directly in `file-service` and `workflow-compiling-service`. `ComputingUnitManagingService` was the one Arrow-bundling service left overriding **only** `jackson-module-scala`, so nothing held `jackson-databind` down. Its dist bundled `jackson-databind-2.21.0.jar` next to `jackson-module-scala_2.13-2.18.8.jar` and aborted at init. **Before → after** (resolved `Runtime/managedClasspath` of `ComputingUnitManagingService`): | artifact | before | after | | --- | --- | --- | | jackson-annotations | 2.21 | 2.21 | | jackson-core | 2.21.0 | 2.21.0 | | **jackson-databind** | **2.21.0** ❌ | **2.18.8** ✅ | The fix mirrors `file-service` / `workflow-compiling-service` exactly: add `"com.fasterxml.jackson.core" % "jackson-databind" % jacksonVersion` to the module's `dependencyOverrides`, and re-sync the one drifted line in `computing-unit-managing-service/LICENSE-binary` (`2.21.0` → `2.18.8`). `NOTICE-binary` is version-less by design (the generator strips versions so routine bumps don't churn it) and the jar *set* is unchanged, so it needs no edit. ### Any related issues, documentation, discussions? Fixes apache#6206. Follow-up to apache#6185. ### How was this PR tested? Reproduced the CI binary-license check locally against the module's resolved runtime classpath (per the repo's `check_binary_deps.py` recipe): - `sbt "print ComputingUnitManagingService/Runtime/managedClasspath"` now resolves `jackson-databind-2.18.8.jar` (was `2.21.0`) — the only delta; no other dep drifted. - `check_binary_deps.py jar <staged-lib> --license-binary computing-unit-managing-service/LICENSE-binary` → `OK: 339 JVM jars match LICENSE-binary.` ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context])
…inal (#6212) ### What changes were proposed in this PR? Backport of #6185 to `release/v1.2`: upgrade Apache Arrow `15.0.2 → 19.0.0` and the Netty family `4.1.x → 4.2.15.Final` as one coordinated move. `arrow-memory-netty` reaches into Netty's allocator internals, so the 4.1/4.2 line cannot split — a split throws `NoClassDefFoundError: io/netty/util/concurrent/ThreadAwareExecutor` on the Arrow Flight transport to the Python workers. Arrow 19 is the first release targeting the Netty 4.2 line, so the two move together. The standalone `flight-grpc` artifact (discontinued after 15.0.2) is dropped — its gRPC transport now lives inside `flight-core`. The Netty family is extracted into a shared `nettyDependencyOverrides` val applied to amber and the three Arrow-bundling platform services, and the Jackson core family is pinned to `2.18.8` in `workflow-core`/`workflow-operator` (Arrow 19 otherwise evicts `jackson-databind` past the `>= 2.18.0, < 2.19.0` range `jackson-module-scala` allows). **Update:** this PR also folds in the `computing-unit-managing-service` startup fix from #6209 (issue #6206), so the Arrow upgrade lands on `release/v1.2` without ever breaking that service's startup. #6185 pinned `jackson-databind` in `workflow-core`/`workflow-operator` and in `file-service`/`workflow-compiling-service`, but not in `computing-unit-managing-service`, whose dist consequently bundled `jackson-databind-2.21.0.jar` next to `jackson-module-scala 2.18.8` and aborted at boot with `Scala module 2.18.8 requires Jackson Databind version >= 2.18.0 and < 2.19.0 - Found jackson-databind version 2.21.0`. The fix adds the `jackson-databind` override to that service too (matching `file-service`/`workflow-compiling-service`). Notes on how the backport was produced: - **Source changes were hand-applied, not cherry-picked.** `release/v1.2`'s `build.sbt` is structurally different from `main`'s (no `Resource` module or `NotebookMigrationService`, `asfLicensingSettings` rather than `commonModuleSettings`, hadoop 3.3.1, no log4j-2 bridges), so a raw cherry-pick of #6185's squash would drag in unrelated structure. Only the Arrow/Netty/Jackson semantics from #6185 + #6209 were ported; the `common/workflow-operator/build.sbt` hunk is byte-identical to #6185's. - **The 8 `LICENSE-binary`(-java) / `NOTICE-binary` manifests were regenerated from `release/v1.2`'s own resolved runtime classpath**, not copied from `main`. `main`'s manifests reflect deps this branch does not have (hadoop 3.4.3, the log4j-2 bridges), so their transitive tails differ. On this branch the cascade resolves to grpc `1.62 → 1.79.0`, protobuf-java `3.25 → 4.33.4` (+ new `protobuf-java-util`), the netty `netty-codec → netty-codec-base`/`netty-codec-compression` split, new `arrow-memory-netty-buffer-patch` and `org.jspecify:jspecify 1.0.0`, `guava → 33.5.0-android`; `flight-grpc` and `eclipse-collections` drop out. ### Any related issues, documentation, discussions? Backports #6185 (merged to `main`). Also includes the fix from #6209 for #6206 (startup regression introduced by #6185). No `release/*` label is added — this PR *is* the backport. ### How was this PR tested? The binary-licensing manifests were verified the same way CI checks them, against each module's resolved `Runtime/managedClasspath` on this branch (which equals the dist's third-party jar set): - `check_binary_deps.py jar --license-binary <manifest> <lib>` passes for all four services in both strict (nightly) mode and `--ignore-transitive-version` (PR) mode: `OK: 407 / 311 / 323 / 350 JVM jars match LICENSE-binary` for amber / file-service / workflow-compiling-service / computing-unit-managing-service. - `generate_notice_binary.py` reproduces each committed `NOTICE-binary` byte-for-byte (regenerate-and-diff is clean for all four), matching the CI NOTICE check. - `sbt export <Service>/Runtime/managedClasspath` resolves cleanly for all four services with Arrow 19 + Netty 4.2 (no eviction or resolution errors); `computing-unit-managing-service` now resolves `jackson-databind-2.18.8.jar` (was `2.21.0`). ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8 [1M context]) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
What changes were proposed in this PR?
Dependabot proposed bumping only
io.netty:netty-transport(andnetty-transport-native-unix-common) to4.2.15.Final. Netty can't move tothe 4.2 line in isolation here: the family is pinned as a unit because
arrow-memory-nettyreaches into Netty's allocator internals, and Apache Arrow15.0.2requires the 4.1 line. Bumping two modules split the family across the4.1/4.2 boundary, which broke the Arrow Flight transport to the Python workers
at runtime:
(
ThreadAwareExecutoris new innetty-common4.2, butnetty-commonwasstill pinned at 4.1.96.)
Apache Arrow
19.0.0is the first released Arrow that targets the Netty 4.2line, so this PR turns the bump into the coordinated upgrade it needs:
15.0.2 → 19.0.0flight-grpcwas discontinued after 15.0.2; its gRPC transport now lives inflight-core4.2.15.FinalnettyDependencyOverridesval, applied to amber and the three Arrow-bundling platform services so the family can never split again2.18.8jackson-databindup to 2.21.0, past the rangejackson-module-scalaallows; pinned inworkflow-core/workflow-operatorLICENSE-binary/NOTICE-binary1.62→1.79, protobuf3.25→4.33, the netty 4.2netty-codec→netty-codec-base/-compressionsplit, newarrow-memory-netty-buffer-patch,guava → 33.5.0-android,eclipse-collectionsdropped)Any related issues, documentation, discussions?
Supersedes the standalone
netty-transportbump this PR started as — Netty 4.2is only reachable by moving Apache Arrow forward at the same time.
How was this PR tested?
Validated through the project's CI on this branch:
build / amber-integration(ubuntu-latest & macos-latest) — the Python-workerArrow Flight transport comes up cleanly; the prior
NoClassDefFoundError: ThreadAwareExecutorcrash is gone.Bench(arrow-flight-e2e) — runs end to end.build / platform (*)— all six services build and pass their binaryLICENSE-binary/NOTICE-binarychecks.build / amber— compiles, builds the Universal dist, and passes the binaryLICENSE-binary/NOTICE-binarychecks; the amber + common unit testsinitialize correctly once the Jackson core family is pinned.
The
LICENSE-binaryandNOTICE-binaryedits were verified by reproducing theCI checks locally (
bin/licensing/check_binary_deps.pyandgenerate_notice_binary.py) against each module's resolved runtime classpath.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8 [1M context])