chore(deps): upgrade Apache Hadoop to 3.4.3 to resolve hadoop-common CVEs#6201
Merged
Merged
Conversation
…preduce-client-core Resolves Dependabot alerts for CVE-2022-25168, CVE-2022-26612, CVE-2021-37404 (critical, fixed in 3.3.3) and CVE-2024-23454 (low, fixed in 3.4.0) by bumping both hadoop-common pins (workflow-core 3.3.1, amber 3.3.3) to 3.4.3. hadoop-mapreduce-client-core is removed since no code or runtime path references org.apache.hadoop.mapreduce. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #6201 +/- ##
============================================
+ Coverage 59.93% 59.95% +0.02%
- Complexity 3307 3310 +3
============================================
Files 1131 1131
Lines 43805 43738 -67
Branches 4745 4735 -10
============================================
- Hits 26253 26222 -31
+ Misses 16128 16095 -33
+ Partials 1424 1421 -3
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
6 tasks
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 390 | 0.238 | 24,176/38,058/38,058 us | 🔴 +22.1% / 🔴 +150.1% |
| 🟢 | bs=100 sw=10 sl=64 | 910 | 0.555 | 109,085/130,367/130,367 us | 🟢 -6.2% / 🔴 +20.8% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,026 | 0.626 | 974,860/1,100,093/1,100,093 us | 🔴 +12.1% / 🔴 +7.2% |
Baseline details
Latest main 5bab776 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 390 tuples/sec | 450 tuples/sec | 777.63 tuples/sec | -13.3% | -49.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.238 MB/s | 0.275 MB/s | 0.475 MB/s | -13.5% | -49.9% |
| bs=10 sw=10 sl=64 | p50 | 24,176 us | 19,800 us | 12,603 us | +22.1% | +91.8% |
| bs=10 sw=10 sl=64 | p95 | 38,058 us | 35,559 us | 15,215 us | +7.0% | +150.1% |
| bs=10 sw=10 sl=64 | p99 | 38,058 us | 35,559 us | 18,617 us | +7.0% | +104.4% |
| bs=100 sw=10 sl=64 | throughput | 910 tuples/sec | 903 tuples/sec | 987.11 tuples/sec | +0.8% | -7.8% |
| bs=100 sw=10 sl=64 | MB/s | 0.555 MB/s | 0.551 MB/s | 0.602 MB/s | +0.7% | -7.9% |
| bs=100 sw=10 sl=64 | p50 | 109,085 us | 107,832 us | 101,428 us | +1.2% | +7.5% |
| bs=100 sw=10 sl=64 | p95 | 130,367 us | 138,943 us | 107,959 us | -6.2% | +20.8% |
| bs=100 sw=10 sl=64 | p99 | 130,367 us | 138,943 us | 114,777 us | -6.2% | +13.6% |
| bs=1000 sw=10 sl=64 | throughput | 1,026 tuples/sec | 1,062 tuples/sec | 1,021 tuples/sec | -3.4% | +0.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.626 MB/s | 0.648 MB/s | 0.623 MB/s | -3.4% | +0.4% |
| bs=1000 sw=10 sl=64 | p50 | 974,860 us | 944,315 us | 986,363 us | +3.2% | -1.2% |
| bs=1000 sw=10 sl=64 | p95 | 1,100,093 us | 980,992 us | 1,026,594 us | +12.1% | +7.2% |
| bs=1000 sw=10 sl=64 | p99 | 1,100,093 us | 980,992 us | 1,054,559 us | +12.1% | +4.3% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,512.26,200,128000,390,0.238,24175.78,38058.15,38058.15
1,100,10,64,20,2197.76,2000,1280000,910,0.555,109084.78,130366.58,130366.58
2,1000,10,64,20,19492.98,20000,12800000,1026,0.626,974859.52,1100092.80,1100092.80…3.4.3 hadoop-common 3.4.3 changes the bundled jar set: the mapreduce/yarn/ kerby-server/htrace/log4j-1.x/netty-3.x baggage is gone, hadoop-thirdparty shaded jars move to 1.5.0 (protobuf_3_25), and hadoop's transitive netty raises the resolved netty line to 4.1.127. LICENSE-binary files updated via check_binary_deps.py findings; NOTICE-binary files regenerated with generate_notice_binary.py from freshly built dists. Both checks (PR mode and strict) pass locally for all three platform services. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same treatment as the platform services' manifests, against amber's java manifest (LICENSE-binary-java): drop the jars that left the dist with hadoop-mapreduce-client-core, bump drifted versions, and add the jars hadoop 3.4.3 newly pulls in. Both check_binary_deps.py modes pass locally against a freshly built amber dist. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t at runtime) CI amber tests failed with NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/input/FileInputFormat — iceberg-parquet's read path references parquet-hadoop classes that extend mapreduce types, so the jar is required at runtime even with no compile-time references. Restore it at 3.4.3 with the exclusion list it had before, plus io.netty: hadoop 3.4 adds a direct netty-all dependency that would otherwise drag the whole netty fat-jar set into every dist (the netty artifacts this module needs are declared explicitly). Binary LICENSE/NOTICE manifests refreshed from rebuilt dists; strict and PR-mode checks pass for all four services. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chenlica
approved these changes
Jul 5, 2026
Resolve conflicts with apache#6199 (log4j 1.2 -> log4j 2.26.1 bridges): - common/workflow-core/build.sbt: keep both the hadoop excludeNetty rule and the log4j bridge dependencies. - LICENSE-binary / NOTICE-binary manifests: regenerated from freshly built dists (hadoop 3.4.3 + log4j 2.26.1 bridges); check_binary_deps.py passes in both PR and strict mode for all four services. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… merge of an older main; no content changes)
aglinxinyuan
added a commit
that referenced
this pull request
Jul 6, 2026
…15.Final Resolves conflicts in the LICENSE-binary / NOTICE-binary files between main's Apache Hadoop 3.4.3 upgrade (#6201) and this PR's Arrow 19 / Netty 4.2 upgrade. Both sides rewrote overlapping regions of the bundled-jar lists. Regenerated each file against the merged runtime classpath (Hadoop 3.4.3 + Arrow 19.0.0 + Netty 4.2.15) with bin/licensing tooling; check_binary_deps.py passes for amber and the three Arrow-bundling platform services.
aglinxinyuan
pushed a commit
to aglinxinyuan/texera
that referenced
this pull request
Jul 6, 2026
…ce startup (apache#6205) ### What changes were proposed in this PR? After apache#6201 upgraded Hadoop to 3.4.3, `texera-web` crashes on boot with `NoClassDefFoundError: com/sun/jersey/core/provider/jaxb/AbstractRootElementProvider`. Hadoop 3.4 ships Jersey 1's JSON module via the fork `com.github.pjfanning:jersey-json` instead of `com.sun.jersey:jersey-json`, so the existing `ExclusionRule(organization = "com.sun.jersey")` rules no longer catch it and the jar lands on the runtime classpath. Its `META-INF/services/javax.ws.rs.ext.MessageBodyReader|Writer` entries register Jersey 1.x providers (e.g. `JSONRootElementProvider`) whose superclasses live in jersey-core 1.x — which *is* excluded — so Dropwizard's Jersey 2 `ServiceFinder` dies loading them during startup. This PR: - adds an exclusion for `com.github.pjfanning:jersey-json` to the `hadoop-common` (and `hadoop-mapreduce-client-core`) dependencies in `common/workflow-core/build.sbt` and `amber/build.sbt`. The module only serves hadoop's own web endpoints; the Iceberg `HadoopCatalog`/`HadoopFileIO` and `HDFSRecordStorage` code paths don't use it. - removes the two jars that consequently leave the dists — `com.github.pjfanning.jersey-json-1.22.0.jar` and its transitive `com.sun.xml.bind.jaxb-impl-2.2.3-1.jar` — from the `LICENSE-binary`(-java) manifests of the four affected services (amber, file-service, workflow-compiling-service, computing-unit-managing-service). `jettison` stays: hadoop-common declares it directly. ### Any related issues, documentation, discussions? Fixes apache#6204. Regression introduced by apache#6201. ### How was this PR tested? - Rebuilt all four affected dists (`sbt <svc>/dist`) and verified via `unzip -l` that `jersey-json` and `jaxb-impl` are no longer bundled in any of them (before the fix, all four bundled both). - `./bin/licensing/check_binary_deps.py --ignore-transitive-version jar --license-binary ...` (the CI PR-mode check) passes for all four services against the freshly built dists. - `./bin/licensing/generate_notice_binary.py` regenerated all four `NOTICE-binary` files from the same dists: no drift (the removed jars carry no NOTICE entries). - Brought the stack up from this branch with `bin/local-dev.sh`: `file-service` and `workflow-compiling-service` — Dropwizard services that previously bundled the jar via workflow-core and exhibit the same startup path — start cleanly (Jetty up, graceful shutdown on `down`). On unmodified `main`, `texera-web` reproducibly crashes with the `NoClassDefFoundError` above; with the jar excluded there is nothing left on the classpath for `ServiceFinder` to trip over. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
aglinxinyuan
added a commit
to Yicong-Huang/texera
that referenced
this pull request
Jul 6, 2026
Resolve conflicts between the Hadoop 3.4.3 upgrade (apache#6201 backport) and the log4j 2.x bridge change (apache#6202) in dependency-metadata files: - common/workflow-core/build.sbt: keep both the excludeNetty rule (Hadoop upgrade) and the log4jVersion val (log4j bridges). - LICENSE-binary / NOTICE-binary (amber + 3 platform services): union of both dependency sets. Regenerated from the actual merged runtime classpath so the files match generate_notice_binary.py / check_binary_deps.py output exactly. Net effect vs the Hadoop branch: add the three log4j 2.26.1 bridge jars, and (services only) bump the transitively-resolved slf4j-api 2.0.16 -> 2.0.17. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Yicong-Huang
added a commit
that referenced
this pull request
Jul 6, 2026
…mon CVEs (#6203) ### What changes were proposed in this PR? Backport of #6201 to `release/v1.2`: upgrade `org.apache.hadoop:hadoop-common` to 3.4.3 in both places it is pinned (`common/workflow-core/build.sbt` was at 3.3.1, `amber/build.sbt` at 3.3.3), and upgrade `hadoop-mapreduce-client-core` to 3.4.3 alongside it with hadoop 3.4's new direct `netty-all` dependency excluded. This resolves the same 4 Dependabot alerts on `hadoop-common` for the release line: - [#656](https://github.com/apache/texera/security/dependabot/656) CVE-2022-25168 (critical, argument injection in `FileUtil.unTar`, fixed in 3.3.3) - [#732](https://github.com/apache/texera/security/dependabot/732) CVE-2022-26612 (critical, fixed in 3.3.3) - [#654](https://github.com/apache/texera/security/dependabot/654) CVE-2021-37404 (critical, heap overflow in libhdfs, fixed in 3.3.2) - [#700](https://github.com/apache/texera/security/dependabot/700) CVE-2024-23454 (low, fixed in 3.4.0) Cherry-pick of 1094a65 with conflicts resolved: - `common/workflow-core/build.sbt`: took only this PR's `excludeNetty` addition; the adjacent `log4jVersion` line in the conflict hunk belongs to #6199, which is not on `release/v1.2` (its backport #6202 is separate and still open). - The 8 `LICENSE-binary`(-java) / `NOTICE-binary` manifests were **regenerated from freshly built `release/v1.2` dists** rather than taken from `main`, because `main`'s manifests reflect deps this branch does not have: the log4j 2.26.1 bridge entries (from #6199) are removed, and pekko stays 1.2.1 (main: 1.6.0), ssl-config-core 0.6.1 (main: 0.7.1), slf4j-api 2.0.16 (main: 2.0.17). `NOTICE-binary` files were regenerated with `generate_notice_binary.py` (amber with `--extras amber/NOTICE-binary-python`). **Update:** this PR also folds in the startup-regression fix from #6205 (issue #6204), so the hadoop upgrade lands on `release/v1.2` without ever breaking service startup. Hadoop 3.4 ships Jersey 1's JSON module via the fork `com.github.pjfanning:jersey-json`, which the existing `com.sun.jersey` exclusion rules do not catch; its `META-INF/services` entries register Jersey 1.x providers whose superclasses live in the (excluded) jersey-core 1.x, so Dropwizard's Jersey 2 `ServiceFinder` dies with `NoClassDefFoundError: com/sun/jersey/core/provider/jaxb/AbstractRootElementProvider` on boot. The fix excludes `com.github.pjfanning:jersey-json` from the hadoop dependencies in `common/workflow-core/build.sbt` and `amber/build.sbt`, and removes the two jars that consequently leave the dists (`jersey-json-1.22.0`, its transitive `jaxb-impl-2.2.3-1`) from the four `LICENSE-binary`(-java) manifests, regenerated against dists rebuilt on this branch. ### Any related issues, documentation, discussions? Backports #6201 (merged to `main`), which resolves #6200. Also includes the fix from #6205 for #6204 (startup regression introduced by #6201). No `release/*` label is added — this PR *is* the backport. ### How was this PR tested? Binary licensing manifests were verified the same way CI checks them: built the four affected dists on this branch (`sbt WorkflowExecutionService/dist FileService/dist ComputingUnitManagingService/dist WorkflowCompilingService/dist`), then ran `check_binary_deps.py` in strict (nightly) mode against each unzipped dist — all four pass (`OK: 395/300/337/312 JVM jars match LICENSE-binary`). For the folded-in #6205 fix: rebuilt all four dists on this branch after adding the exclusion and verified via `unzip -l` that `jersey-json` and `jaxb-impl` are no longer bundled in any of them; `check_binary_deps.py` re-run against the fresh dists passes for all four services (`OK: 393/298/310/335 JVM jars match LICENSE-binary`), and `generate_notice_binary.py` reports no `NOTICE-binary` drift (the removed jars carry no NOTICE entries). On `main` the same exclusion was verified end to end: `texera-web` reproducibly crashes on boot without it and starts cleanly with it (Jetty up on :8080). Tested with existing test cases: - `sbt WorkflowCore/test`: 294 tests passed, 0 failed. (4 suites — `S3StorageClientSpec` and the `LargeBinary*` specs — abort because they need an external MinIO service; verified they abort identically on unmodified `release/v1.2`, unrelated to this change.) - `sbt WorkflowExecutionService/Test/compile` passes (95 test sources; `SequentialRecordStorageSpec`, run on `main` for the `HDFSRecordStorage` path, does not exist on `release/v1.2` — `HDFSRecordStorage` itself compiles against hadoop 3.4.3). - `sbt "WorkflowCore/testOnly *IcebergUtilSpec"` (Iceberg + hadoop `Configuration` usage): 14 tests passed. (`DocumentFactorySpec` from the original PR's test list also does not exist on `release/v1.2`.) ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Xinyuan Lin <xinyual3@uci.edu>
This was referenced Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Upgrade
org.apache.hadoop:hadoop-commonto 3.4.3 in both places it is pinned (common/workflow-core/build.sbtwas at 3.3.1,amber/build.sbtat 3.3.3), and removehadoop-mapreduce-client-corefrom workflow-core.This resolves 4 open Dependabot alerts on
hadoop-common:FileUtil.unTar, fixed in 3.3.3)The hadoop dependency itself cannot be removed: Iceberg's
HadoopCatalog/HadoopFileIO(used byIcebergUtilin workflow-core) andHDFSRecordStorage(amber) require hadoop-common at runtime.hadoop-mapreduce-client-corehas no compile-time references in the codebase, but iceberg-parquet's read path loads parquet-hadoop classes that extendorg.apache.hadoop.mapreducetypes (amber e2e tests fail withNoClassDefFoundError: org/apache/hadoop/mapreduce/lib/input/FileInputFormatwithout it), so it is upgraded to 3.4.3 alongside hadoop-common rather than dropped. hadoop 3.4 gives it a directnetty-alldependency, which is excluded — the netty artifacts workflow-core needs are already declared explicitly, and this keeps the netty fat-jar set out of every dist.Side effects visible in the binary manifests (all previously bundled jars, now newer): the hadoop 3.3 baggage leaves the dists (log4j 1.2.17, netty 3.10.6, htrace, kerby 1.x server jars, okhttp 2.7.5), and the per-service
LICENSE-binary(-java) /NOTICE-binaryfiles are refreshed from freshly built dists using the CI scripts.Any related issues, documentation, discussions?
Resolves #6200. Dependabot alert https://github.com/apache/texera/security/dependabot/656 (also 732, 654, 700).
How was this PR tested?
Binary licensing manifests were refreshed the same way CI checks them: built the four affected dists locally (
sbt <svc>/dist), updated each service'sLICENSE-binaryfromcheck_binary_deps.pyfindings, and regeneratedNOTICE-binarywithgenerate_notice_binary.py(amber with--extras amber/NOTICE-binary-python).check_binary_deps.pynow passes locally for all three platform services in both PR mode and strict (nightly) mode.Tested with existing test cases:
sbt WorkflowCore/test: 458 tests passed, 0 failed. (5 suites — S3/LakeFS/LargeBinary specs — abort because they need external MinIO/LakeFS services; verified they abort identically on unmodifiedmain, unrelated to this change.)sbt WorkflowExecutionService/Test/compileandWorkflowExecutionService/testOnly ...SequentialRecordStorageSpec(covers theHDFSRecordStoragecode path): 9 tests passed.IcebergUtilSpecandDocumentFactorySpec(Iceberg + hadoopConfigurationusage) pass, confirming nothing neededhadoop-mapreduce-client-core.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)