fix(deps, v1.2): upgrade Apache Arrow to 19.0.0 and Netty to 4.2.15.Final

[aglinxinyuan]

What changes were proposed in this PR?

Backport of #6185 to release/v1.2: upgrade Apache Arrow 15.0.2 → 19.0.0 and the Netty family 4.1.x → 4.2.15.Final as one coordinated move. arrow-memory-netty reaches into Netty's allocator internals, so the 4.1/4.2 line cannot split — a split throws NoClassDefFoundError: io/netty/util/concurrent/ThreadAwareExecutor on the Arrow Flight transport to the Python workers. Arrow 19 is the first release targeting the Netty 4.2 line, so the two move together. The standalone flight-grpc artifact (discontinued after 15.0.2) is dropped — its gRPC transport now lives inside flight-core. The Netty family is extracted into a shared nettyDependencyOverrides val applied to amber and the three Arrow-bundling platform services, and the Jackson core family is pinned to 2.18.8 in workflow-core/workflow-operator (Arrow 19 otherwise evicts jackson-databind past the >= 2.18.0, < 2.19.0 range jackson-module-scala allows).

Update: this PR also folds in the computing-unit-managing-service startup fix from #6209 (issue #6206), so the Arrow upgrade lands on release/v1.2 without ever breaking that service's startup. #6185 pinned jackson-databind in workflow-core/workflow-operator and in file-service/workflow-compiling-service, but not in computing-unit-managing-service, whose dist consequently bundled jackson-databind-2.21.0.jar next to jackson-module-scala 2.18.8 and aborted at boot with Scala module 2.18.8 requires Jackson Databind version >= 2.18.0 and < 2.19.0 - Found jackson-databind version 2.21.0. The fix adds the jackson-databind override to that service too (matching file-service/workflow-compiling-service).

Notes on how the backport was produced:

• Source changes were hand-applied, not cherry-picked. release/v1.2's build.sbt is structurally different from main's (no Resource module or NotebookMigrationService, asfLicensingSettings rather than commonModuleSettings, hadoop 3.3.1, no log4j-2 bridges), so a raw cherry-pick of fix(deps): upgrade Apache Arrow to 19.0.0 and Netty to 4.2.15.Final #6185's squash would drag in unrelated structure. Only the Arrow/Netty/Jackson semantics from fix(deps): upgrade Apache Arrow to 19.0.0 and Netty to 4.2.15.Final #6185 + fix(deps): pin jackson-databind to 2.18.8 in computing-unit-managing-service #6209 were ported; the common/workflow-operator/build.sbt hunk is byte-identical to fix(deps): upgrade Apache Arrow to 19.0.0 and Netty to 4.2.15.Final #6185's.
• The 8 LICENSE-binary(-java) / NOTICE-binary manifests were regenerated from release/v1.2's own resolved runtime classpath, not copied from main. main's manifests reflect deps this branch does not have (hadoop 3.4.3, the log4j-2 bridges), so their transitive tails differ. On this branch the cascade resolves to grpc 1.62 → 1.79.0, protobuf-java 3.25 → 4.33.4 (+ new protobuf-java-util), the netty netty-codec → netty-codec-base/netty-codec-compression split, new arrow-memory-netty-buffer-patch and org.jspecify:jspecify 1.0.0, guava → 33.5.0-android; flight-grpc and eclipse-collections drop out.

Any related issues, documentation, discussions?

Backports #6185 (merged to main). Also includes the fix from #6209 for #6206 (startup regression introduced by #6185). No release/* label is added — this PR is the backport.

How was this PR tested?

The binary-licensing manifests were verified the same way CI checks them, against each module's resolved Runtime/managedClasspath on this branch (which equals the dist's third-party jar set):

• check_binary_deps.py jar --license-binary <manifest> <lib> passes for all four services in both strict (nightly) mode and --ignore-transitive-version (PR) mode: OK: 407 / 311 / 323 / 350 JVM jars match LICENSE-binary for amber / file-service / workflow-compiling-service / computing-unit-managing-service.
• generate_notice_binary.py reproduces each committed NOTICE-binary byte-for-byte (regenerate-and-diff is clean for all four), matching the CI NOTICE check.
• sbt export <Service>/Runtime/managedClasspath resolves cleanly for all four services with Arrow 19 + Netty 4.2 (no eviction or resolution errors); computing-unit-managing-service now resolves jackson-databind-2.18.8.jar (was 2.21.0).

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8 [1M context])

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @Yicong-Huang
  You can notify them by mentioning @Yicong-Huang in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.06%. Comparing base (0059bea) to head (b695528).

Additional details and impacted files

@@                Coverage Diff                 @@
##             release/v1.2    #6212      +/-   ##
==================================================
- Coverage           52.07%   52.06%   -0.01%     
+ Complexity           2490     2486       -4     
==================================================
  Files                1065     1065              
  Lines               41315    41315              
  Branches             4424     4424              
==================================================
- Hits                21514    21511       -3     
- Misses              18536    18539       +3     
  Partials             1265     1265              

Flag	Coverage Δ		*Carryforward flag
access-control-service	64.61% <ø> (ø)
agent-service	34.36% <ø> (ø)		Carriedforward from 8f2faef
amber	52.41% <ø> (-0.03%)	⬇️
computing-unit-managing-service	1.65% <ø> (ø)
config-service	56.06% <ø> (ø)
file-service	58.59% <ø> (ø)
frontend	46.32% <ø> (+<0.01%)	⬆️	Carriedforward from 8f2faef
pyamber	90.77% <ø> (ø)		Carriedforward from 8f2faef
python	90.75% <ø> (ø)		Carriedforward from 8f2faef
workflow-compiling-service	58.69% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Backport to release/v1.2 that upgrades the Arrow Flight stack to Apache Arrow 19.0.0 together with a coordinated Netty family move to 4.2.15.Final, while adding dependency overrides to avoid known runtime incompatibilities (notably the Arrow/Netty coupling and the jackson-module-scala vs jackson-databind version range issue). It also refreshes the binary licensing manifests to match the new resolved runtime classpaths.

Changes:

• Upgrade Arrow Flight dependencies to 19.0.0 and remove the deprecated flight-grpc usage in favor of flight-core.
• Centralize/pin Netty 4.2.15.Final via shared overrides across Amber + Arrow-bundling services.
• Regenerate LICENSE-binary / NOTICE-binary manifests for the affected distributions to reflect the new dependency graph.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
build.sbt	Introduces shared Netty override list; applies Netty/Jackson overrides across Amber and platform services.
common/workflow-core/build.sbt	Updates Arrow/Netty versions and adds dependency overrides for Netty + Jackson core family.
common/workflow-operator/build.sbt	Adds Jackson core-family overrides to keep jackson-module-scala compatible under Arrow 19 transitive deps.
amber/NOTICE-binary	Refreshes third-party notices for the new resolved runtime dependencies.
amber/LICENSE-binary-java	Updates jar inventory for Amber’s Java distribution after dependency upgrades.
file-service/NOTICE-binary	Refreshes third-party notices for updated resolved runtime dependencies.
file-service/LICENSE-binary	Updates jar inventory for FileService distribution after dependency upgrades.
workflow-compiling-service/NOTICE-binary	Refreshes third-party notices for updated resolved runtime dependencies.
workflow-compiling-service/LICENSE-binary	Updates jar inventory for WorkflowCompilingService distribution after dependency upgrades.
computing-unit-managing-service/NOTICE-binary	Refreshes third-party notices for updated resolved runtime dependencies.
computing-unit-managing-service/LICENSE-binary	Updates jar inventory for ComputingUnitManagingService distribution after dependency upgrades.

Comments suppressed due to low confidence (1)

common/workflow-core/build.sbt:136

• The Netty pinning list in workflow-core has the same gap as the root nettyDependencyOverrides: netty-codec-base and netty-codec-compression are present in service runtime jars, but aren’t included in dependencyOverrides here. If any transitive brings these at a different version, the Netty family can still split.

  "io.netty" % "netty-buffer" % nettyVersion,
  "io.netty" % "netty-codec" % nettyVersion,
  "io.netty" % "netty-codec-http" % nettyVersion,
  "io.netty" % "netty-codec-http2" % nettyVersion,
  "io.netty" % "netty-codec-socks" % nettyVersion,

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Yicong-Huang]

LGTM. also needs @xuang7 's stamp.

[xuang7]

LGTM! There are some conflicts.