Skip to content

chore(deps): upgrade Apache Hadoop to 3.4.3 to resolve hadoop-common CVEs#6201

Merged
Yicong-Huang merged 7 commits into
apache:mainfrom
Yicong-Huang:yicong/6200-hadoop-cve
Jul 6, 2026
Merged

chore(deps): upgrade Apache Hadoop to 3.4.3 to resolve hadoop-common CVEs#6201
Yicong-Huang merged 7 commits into
apache:mainfrom
Yicong-Huang:yicong/6200-hadoop-cve

Conversation

@Yicong-Huang

@Yicong-Huang Yicong-Huang commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Upgrade org.apache.hadoop:hadoop-common to 3.4.3 in both places it is pinned (common/workflow-core/build.sbt was at 3.3.1, amber/build.sbt at 3.3.3), and remove hadoop-mapreduce-client-core from workflow-core.

This resolves 4 open Dependabot alerts on hadoop-common:

The hadoop dependency itself cannot be removed: Iceberg's HadoopCatalog/HadoopFileIO (used by IcebergUtil in workflow-core) and HDFSRecordStorage (amber) require hadoop-common at runtime. hadoop-mapreduce-client-core has no compile-time references in the codebase, but iceberg-parquet's read path loads parquet-hadoop classes that extend org.apache.hadoop.mapreduce types (amber e2e tests fail with NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/input/FileInputFormat without it), so it is upgraded to 3.4.3 alongside hadoop-common rather than dropped. hadoop 3.4 gives it a direct netty-all dependency, which is excluded — the netty artifacts workflow-core needs are already declared explicitly, and this keeps the netty fat-jar set out of every dist.

Side effects visible in the binary manifests (all previously bundled jars, now newer): the hadoop 3.3 baggage leaves the dists (log4j 1.2.17, netty 3.10.6, htrace, kerby 1.x server jars, okhttp 2.7.5), and the per-service LICENSE-binary(-java) / NOTICE-binary files are refreshed from freshly built dists using the CI scripts.

Any related issues, documentation, discussions?

Resolves #6200. Dependabot alert https://github.com/apache/texera/security/dependabot/656 (also 732, 654, 700).

How was this PR tested?

Binary licensing manifests were refreshed the same way CI checks them: built the four affected dists locally (sbt <svc>/dist), updated each service's LICENSE-binary from check_binary_deps.py findings, and regenerated NOTICE-binary with generate_notice_binary.py (amber with --extras amber/NOTICE-binary-python). check_binary_deps.py now passes locally for all three platform services in both PR mode and strict (nightly) mode.

Tested with existing test cases:

  • sbt WorkflowCore/test: 458 tests passed, 0 failed. (5 suites — S3/LakeFS/LargeBinary specs — abort because they need external MinIO/LakeFS services; verified they abort identically on unmodified main, unrelated to this change.)
  • sbt WorkflowExecutionService/Test/compile and WorkflowExecutionService/testOnly ...SequentialRecordStorageSpec (covers the HDFSRecordStorage code path): 9 tests passed.
  • IcebergUtilSpec and DocumentFactorySpec (Iceberg + hadoop Configuration usage) pass, confirming nothing needed hadoop-mapreduce-client-core.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Fable 5)

…preduce-client-core

Resolves Dependabot alerts for CVE-2022-25168, CVE-2022-26612,
CVE-2021-37404 (critical, fixed in 3.3.3) and CVE-2024-23454 (low,
fixed in 3.4.0) by bumping both hadoop-common pins (workflow-core
3.3.1, amber 3.3.3) to 3.4.3. hadoop-mapreduce-client-core is removed
since no code or runtime path references org.apache.hadoop.mapreduce.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added engine dependencies Pull requests that update a dependency file common labels Jul 5, 2026
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Committers with relevant context: @pjfanning
    You can request their reviews formally with /request-review @pjfanning.

@codecov-commenter

codecov-commenter commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.95%. Comparing base (5bab776) to head (68b5fb5).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #6201      +/-   ##
============================================
+ Coverage     59.93%   59.95%   +0.02%     
- Complexity     3307     3310       +3     
============================================
  Files          1131     1131              
  Lines         43805    43738      -67     
  Branches       4745     4735      -10     
============================================
- Hits          26253    26222      -31     
+ Misses        16128    16095      -33     
+ Partials       1424     1421       -3     
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø)
agent-service 44.59% <ø> (ø) Carriedforward from 096926a
amber 65.14% <ø> (+0.02%) ⬆️
computing-unit-managing-service 0.00% <ø> (ø)
config-service 52.30% <ø> (ø)
file-service 62.81% <ø> (ø)
frontend 51.47% <ø> (-0.02%) ⬇️ Carriedforward from 096926a
notebook-migration-service 78.57% <ø> (ø)
pyamber 91.19% <ø> (+0.01%) ⬆️ Carriedforward from 096926a
workflow-compiling-service 55.14% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Yicong-Huang Yicong-Huang changed the title chore(deps): upgrade hadoop-common to 3.4.3 and drop unused hadoop-mapreduce-client-core fix(deps): upgrade hadoop-common to 3.4.3 and drop unused hadoop-mapreduce-client-core Jul 5, 2026
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 7 worse · ⚪ 6 noise (<±5%) · 0 without baseline

Compared against main 5bab776 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 390 0.238 24,176/38,058/38,058 us 🔴 +22.1% / 🔴 +150.1%
🟢 bs=100 sw=10 sl=64 910 0.555 109,085/130,367/130,367 us 🟢 -6.2% / 🔴 +20.8%
🔴 bs=1000 sw=10 sl=64 1,026 0.626 974,860/1,100,093/1,100,093 us 🔴 +12.1% / 🔴 +7.2%
Baseline details

Latest main 5bab776 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 390 tuples/sec 450 tuples/sec 777.63 tuples/sec -13.3% -49.8%
bs=10 sw=10 sl=64 MB/s 0.238 MB/s 0.275 MB/s 0.475 MB/s -13.5% -49.9%
bs=10 sw=10 sl=64 p50 24,176 us 19,800 us 12,603 us +22.1% +91.8%
bs=10 sw=10 sl=64 p95 38,058 us 35,559 us 15,215 us +7.0% +150.1%
bs=10 sw=10 sl=64 p99 38,058 us 35,559 us 18,617 us +7.0% +104.4%
bs=100 sw=10 sl=64 throughput 910 tuples/sec 903 tuples/sec 987.11 tuples/sec +0.8% -7.8%
bs=100 sw=10 sl=64 MB/s 0.555 MB/s 0.551 MB/s 0.602 MB/s +0.7% -7.9%
bs=100 sw=10 sl=64 p50 109,085 us 107,832 us 101,428 us +1.2% +7.5%
bs=100 sw=10 sl=64 p95 130,367 us 138,943 us 107,959 us -6.2% +20.8%
bs=100 sw=10 sl=64 p99 130,367 us 138,943 us 114,777 us -6.2% +13.6%
bs=1000 sw=10 sl=64 throughput 1,026 tuples/sec 1,062 tuples/sec 1,021 tuples/sec -3.4% +0.5%
bs=1000 sw=10 sl=64 MB/s 0.626 MB/s 0.648 MB/s 0.623 MB/s -3.4% +0.4%
bs=1000 sw=10 sl=64 p50 974,860 us 944,315 us 986,363 us +3.2% -1.2%
bs=1000 sw=10 sl=64 p95 1,100,093 us 980,992 us 1,026,594 us +12.1% +7.2%
bs=1000 sw=10 sl=64 p99 1,100,093 us 980,992 us 1,054,559 us +12.1% +4.3%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,512.26,200,128000,390,0.238,24175.78,38058.15,38058.15
1,100,10,64,20,2197.76,2000,1280000,910,0.555,109084.78,130366.58,130366.58
2,1000,10,64,20,19492.98,20000,12800000,1026,0.626,974859.52,1100092.80,1100092.80

…3.4.3

hadoop-common 3.4.3 changes the bundled jar set: the mapreduce/yarn/
kerby-server/htrace/log4j-1.x/netty-3.x baggage is gone, hadoop-thirdparty
shaded jars move to 1.5.0 (protobuf_3_25), and hadoop's transitive netty
raises the resolved netty line to 4.1.127. LICENSE-binary files updated
via check_binary_deps.py findings; NOTICE-binary files regenerated with
generate_notice_binary.py from freshly built dists. Both checks (PR mode
and strict) pass locally for all three platform services.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions github-actions Bot added the platform Non-amber Scala service paths label Jul 5, 2026
Yicong-Huang and others added 2 commits July 5, 2026 14:19
Same treatment as the platform services' manifests, against amber's
java manifest (LICENSE-binary-java): drop the jars that left the dist
with hadoop-mapreduce-client-core, bump drifted versions, and add the
jars hadoop 3.4.3 newly pulls in. Both check_binary_deps.py modes pass
locally against a freshly built amber dist.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t at runtime)

CI amber tests failed with NoClassDefFoundError:
org/apache/hadoop/mapreduce/lib/input/FileInputFormat — iceberg-parquet's
read path references parquet-hadoop classes that extend mapreduce types,
so the jar is required at runtime even with no compile-time references.
Restore it at 3.4.3 with the exclusion list it had before, plus io.netty:
hadoop 3.4 adds a direct netty-all dependency that would otherwise drag
the whole netty fat-jar set into every dist (the netty artifacts this
module needs are declared explicitly). Binary LICENSE/NOTICE manifests
refreshed from rebuilt dists; strict and PR-mode checks pass for all
four services.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Yicong-Huang Yicong-Huang changed the title fix(deps): upgrade hadoop-common to 3.4.3 and drop unused hadoop-mapreduce-client-core chore(deps): upgrade Apache Hadoop to 3.4.3 to resolve hadoop-common CVEs Jul 5, 2026
@chenlica chenlica self-requested a review July 5, 2026 21:56
@Yicong-Huang Yicong-Huang added the release/v1.2 back porting to release/v1.2 label Jul 5, 2026
@Yicong-Huang Yicong-Huang removed the release/v1.2 back porting to release/v1.2 label Jul 5, 2026
@Yicong-Huang Yicong-Huang enabled auto-merge July 5, 2026 22:12
@Yicong-Huang Yicong-Huang added this pull request to the merge queue Jul 5, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Jul 5, 2026
@Yicong-Huang Yicong-Huang enabled auto-merge July 5, 2026 23:55
Yicong-Huang and others added 2 commits July 5, 2026 18:19
Resolve conflicts with apache#6199 (log4j 1.2 -> log4j 2.26.1 bridges):
- common/workflow-core/build.sbt: keep both the hadoop excludeNetty rule
  and the log4j bridge dependencies.
- LICENSE-binary / NOTICE-binary manifests: regenerated from freshly
  built dists (hadoop 3.4.3 + log4j 2.26.1 bridges);
  check_binary_deps.py passes in both PR and strict mode for all four
  services.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… merge of an older main; no content changes)
@Yicong-Huang Yicong-Huang added this pull request to the merge queue Jul 6, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 6, 2026
@Yicong-Huang Yicong-Huang added this pull request to the merge queue Jul 6, 2026
Merged via the queue into apache:main with commit 1094a65 Jul 6, 2026
23 checks passed
@Yicong-Huang Yicong-Huang deleted the yicong/6200-hadoop-cve branch July 6, 2026 01:54
aglinxinyuan added a commit that referenced this pull request Jul 6, 2026
…15.Final

Resolves conflicts in the LICENSE-binary / NOTICE-binary files between main's
Apache Hadoop 3.4.3 upgrade (#6201) and this PR's Arrow 19 / Netty 4.2 upgrade.
Both sides rewrote overlapping regions of the bundled-jar lists. Regenerated
each file against the merged runtime classpath (Hadoop 3.4.3 + Arrow 19.0.0 +
Netty 4.2.15) with bin/licensing tooling; check_binary_deps.py passes for amber
and the three Arrow-bundling platform services.
aglinxinyuan pushed a commit to aglinxinyuan/texera that referenced this pull request Jul 6, 2026
…ce startup (apache#6205)

### What changes were proposed in this PR?

After apache#6201 upgraded Hadoop to 3.4.3, `texera-web` crashes on boot with
`NoClassDefFoundError:
com/sun/jersey/core/provider/jaxb/AbstractRootElementProvider`.

Hadoop 3.4 ships Jersey 1's JSON module via the fork
`com.github.pjfanning:jersey-json` instead of
`com.sun.jersey:jersey-json`, so the
existing `ExclusionRule(organization = "com.sun.jersey")` rules no
longer catch it and
the jar lands on the runtime classpath. Its
`META-INF/services/javax.ws.rs.ext.MessageBodyReader|Writer`
entries register Jersey 1.x providers (e.g. `JSONRootElementProvider`)
whose superclasses
live in jersey-core 1.x — which *is* excluded — so Dropwizard's Jersey 2
`ServiceFinder`
dies loading them during startup.

This PR:
- adds an exclusion for `com.github.pjfanning:jersey-json` to the
`hadoop-common`
(and `hadoop-mapreduce-client-core`) dependencies in
`common/workflow-core/build.sbt`
and `amber/build.sbt`. The module only serves hadoop's own web
endpoints; the
Iceberg `HadoopCatalog`/`HadoopFileIO` and `HDFSRecordStorage` code
paths don't use it.
- removes the two jars that consequently leave the dists —
  `com.github.pjfanning.jersey-json-1.22.0.jar` and its transitive
`com.sun.xml.bind.jaxb-impl-2.2.3-1.jar` — from the
`LICENSE-binary`(-java) manifests
of the four affected services (amber, file-service,
workflow-compiling-service,
computing-unit-managing-service). `jettison` stays: hadoop-common
declares it directly.

### Any related issues, documentation, discussions?

Fixes apache#6204. Regression introduced by apache#6201.

### How was this PR tested?

- Rebuilt all four affected dists (`sbt <svc>/dist`) and verified via
`unzip -l` that
`jersey-json` and `jaxb-impl` are no longer bundled in any of them
(before the fix,
  all four bundled both).
- `./bin/licensing/check_binary_deps.py --ignore-transitive-version jar
--license-binary ...`
(the CI PR-mode check) passes for all four services against the freshly
built dists.
- `./bin/licensing/generate_notice_binary.py` regenerated all four
`NOTICE-binary` files
from the same dists: no drift (the removed jars carry no NOTICE
entries).
- Brought the stack up from this branch with `bin/local-dev.sh`:
`file-service` and
`workflow-compiling-service` — Dropwizard services that previously
bundled the jar via
workflow-core and exhibit the same startup path — start cleanly (Jetty
up, graceful
shutdown on `down`). On unmodified `main`, `texera-web` reproducibly
crashes with the
`NoClassDefFoundError` above; with the jar excluded there is nothing
left on the
  classpath for `ServiceFinder` to trip over.

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Fable 5)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
aglinxinyuan added a commit to Yicong-Huang/texera that referenced this pull request Jul 6, 2026
Resolve conflicts between the Hadoop 3.4.3 upgrade (apache#6201 backport) and the
log4j 2.x bridge change (apache#6202) in dependency-metadata files:

- common/workflow-core/build.sbt: keep both the excludeNetty rule (Hadoop
  upgrade) and the log4jVersion val (log4j bridges).
- LICENSE-binary / NOTICE-binary (amber + 3 platform services): union of both
  dependency sets. Regenerated from the actual merged runtime classpath so the
  files match generate_notice_binary.py / check_binary_deps.py output exactly.
  Net effect vs the Hadoop branch: add the three log4j 2.26.1 bridge jars, and
  (services only) bump the transitively-resolved slf4j-api 2.0.16 -> 2.0.17.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Yicong-Huang added a commit that referenced this pull request Jul 6, 2026
…mon CVEs (#6203)

### What changes were proposed in this PR?

Backport of #6201 to `release/v1.2`: upgrade
`org.apache.hadoop:hadoop-common` to 3.4.3 in both places it is pinned
(`common/workflow-core/build.sbt` was at 3.3.1, `amber/build.sbt` at
3.3.3), and upgrade `hadoop-mapreduce-client-core` to 3.4.3 alongside it
with hadoop 3.4's new direct `netty-all` dependency excluded.

This resolves the same 4 Dependabot alerts on `hadoop-common` for the
release line:
- [#656](https://github.com/apache/texera/security/dependabot/656)
CVE-2022-25168 (critical, argument injection in `FileUtil.unTar`, fixed
in 3.3.3)
- [#732](https://github.com/apache/texera/security/dependabot/732)
CVE-2022-26612 (critical, fixed in 3.3.3)
- [#654](https://github.com/apache/texera/security/dependabot/654)
CVE-2021-37404 (critical, heap overflow in libhdfs, fixed in 3.3.2)
- [#700](https://github.com/apache/texera/security/dependabot/700)
CVE-2024-23454 (low, fixed in 3.4.0)

Cherry-pick of 1094a65 with conflicts resolved:
- `common/workflow-core/build.sbt`: took only this PR's `excludeNetty`
addition; the adjacent `log4jVersion` line in the conflict hunk belongs
to #6199, which is not on `release/v1.2` (its backport #6202 is separate
and still open).
- The 8 `LICENSE-binary`(-java) / `NOTICE-binary` manifests were
**regenerated from freshly built `release/v1.2` dists** rather than
taken from `main`, because `main`'s manifests reflect deps this branch
does not have: the log4j 2.26.1 bridge entries (from #6199) are removed,
and pekko stays 1.2.1 (main: 1.6.0), ssl-config-core 0.6.1 (main:
0.7.1), slf4j-api 2.0.16 (main: 2.0.17). `NOTICE-binary` files were
regenerated with `generate_notice_binary.py` (amber with `--extras
amber/NOTICE-binary-python`).

**Update:** this PR also folds in the startup-regression fix from
#6205 (issue #6204), so the hadoop upgrade lands on
`release/v1.2` without ever breaking service startup. Hadoop 3.4 ships
Jersey 1's JSON module via the fork `com.github.pjfanning:jersey-json`,
which the existing `com.sun.jersey` exclusion rules do not catch; its
`META-INF/services` entries register Jersey 1.x providers whose
superclasses live in the (excluded) jersey-core 1.x, so Dropwizard's
Jersey 2 `ServiceFinder` dies with `NoClassDefFoundError:
com/sun/jersey/core/provider/jaxb/AbstractRootElementProvider` on boot.
The fix excludes `com.github.pjfanning:jersey-json` from the hadoop
dependencies in `common/workflow-core/build.sbt` and `amber/build.sbt`,
and removes the two jars that consequently leave the dists
(`jersey-json-1.22.0`, its transitive `jaxb-impl-2.2.3-1`) from the four
`LICENSE-binary`(-java) manifests, regenerated against dists rebuilt on
this branch.

### Any related issues, documentation, discussions?

Backports #6201 (merged to `main`), which resolves #6200.
Also includes the fix from #6205 for #6204 (startup
regression introduced by #6201). No `release/*` label is added — this PR
*is* the backport.

### How was this PR tested?

Binary licensing manifests were verified the same way CI checks them:
built the four affected dists on this branch (`sbt
WorkflowExecutionService/dist FileService/dist
ComputingUnitManagingService/dist WorkflowCompilingService/dist`), then
ran `check_binary_deps.py` in strict (nightly) mode against each
unzipped dist — all four pass (`OK: 395/300/337/312 JVM jars match
LICENSE-binary`).

For the folded-in #6205 fix: rebuilt all four dists on this branch after
adding the exclusion and verified via `unzip -l` that `jersey-json` and
`jaxb-impl` are no longer bundled in any of them; `check_binary_deps.py`
re-run against the fresh dists passes for all four services (`OK:
393/298/310/335 JVM jars match LICENSE-binary`), and
`generate_notice_binary.py` reports no `NOTICE-binary` drift (the
removed jars carry no NOTICE entries). On `main` the same exclusion was
verified end to end: `texera-web` reproducibly crashes on boot without
it and starts cleanly with it (Jetty up on :8080).

Tested with existing test cases:
- `sbt WorkflowCore/test`: 294 tests passed, 0 failed. (4 suites —
`S3StorageClientSpec` and the `LargeBinary*` specs — abort because they
need an external MinIO service; verified they abort identically on
unmodified `release/v1.2`, unrelated to this change.)
- `sbt WorkflowExecutionService/Test/compile` passes (95 test sources;
`SequentialRecordStorageSpec`, run on `main` for the `HDFSRecordStorage`
path, does not exist on `release/v1.2` — `HDFSRecordStorage` itself
compiles against hadoop 3.4.3).
- `sbt "WorkflowCore/testOnly *IcebergUtilSpec"` (Iceberg + hadoop
`Configuration` usage): 14 tests passed. (`DocumentFactorySpec` from the
original PR's test list also does not exist on `release/v1.2`.)

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Fable 5)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Xinyuan Lin <xinyual3@uci.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common dependencies Pull requests that update a dependency file engine platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade Apache Hadoop dependencies to 3.4.3 to resolve critical CVEs

3 participants