Oliver daemon db#656
Closed
jowdy wants to merge 7 commits into
Closed
Conversation
Contributor
|
High-level comments: 1) Add a description about this PR; 2) Use a more appropriate subject; 3) Is it the "smallest" PR? |
sshiv012
pushed a commit
to sshiv012/texera
that referenced
this pull request
Jul 6, 2026
…CVEs (apache#6201) ### What changes were proposed in this PR? Upgrade `org.apache.hadoop:hadoop-common` to 3.4.3 in both places it is pinned (`common/workflow-core/build.sbt` was at 3.3.1, `amber/build.sbt` at 3.3.3), and remove `hadoop-mapreduce-client-core` from workflow-core. This resolves 4 open Dependabot alerts on `hadoop-common`: - [apache#656](https://github.com/apache/texera/security/dependabot/656) CVE-2022-25168 (critical, argument injection in `FileUtil.unTar`, fixed in 3.3.3) - [apache#732](https://github.com/apache/texera/security/dependabot/732) CVE-2022-26612 (critical, fixed in 3.3.3) - [apache#654](https://github.com/apache/texera/security/dependabot/654) CVE-2021-37404 (critical, heap overflow in libhdfs, fixed in 3.3.2) - [apache#700](https://github.com/apache/texera/security/dependabot/700) CVE-2024-23454 (low, fixed in 3.4.0) The hadoop dependency itself cannot be removed: Iceberg's `HadoopCatalog`/`HadoopFileIO` (used by `IcebergUtil` in workflow-core) and `HDFSRecordStorage` (amber) require hadoop-common at runtime. `hadoop-mapreduce-client-core` has no compile-time references in the codebase, but iceberg-parquet's read path loads parquet-hadoop classes that extend `org.apache.hadoop.mapreduce` types (amber e2e tests fail with `NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/input/FileInputFormat` without it), so it is upgraded to 3.4.3 alongside hadoop-common rather than dropped. hadoop 3.4 gives it a direct `netty-all` dependency, which is excluded — the netty artifacts workflow-core needs are already declared explicitly, and this keeps the netty fat-jar set out of every dist. Side effects visible in the binary manifests (all previously bundled jars, now newer): the hadoop 3.3 baggage leaves the dists (log4j 1.2.17, netty 3.10.6, htrace, kerby 1.x server jars, okhttp 2.7.5), and the per-service `LICENSE-binary`(-java) / `NOTICE-binary` files are refreshed from freshly built dists using the CI scripts. ### Any related issues, documentation, discussions? Resolves apache#6200. Dependabot alert https://github.com/apache/texera/security/dependabot/656 (also 732, 654, 700). ### How was this PR tested? Binary licensing manifests were refreshed the same way CI checks them: built the four affected dists locally (`sbt <svc>/dist`), updated each service's `LICENSE-binary` from `check_binary_deps.py` findings, and regenerated `NOTICE-binary` with `generate_notice_binary.py` (amber with `--extras amber/NOTICE-binary-python`). `check_binary_deps.py` now passes locally for all three platform services in both PR mode and strict (nightly) mode. Tested with existing test cases: - `sbt WorkflowCore/test`: 458 tests passed, 0 failed. (5 suites — S3/LakeFS/LargeBinary specs — abort because they need external MinIO/LakeFS services; verified they abort identically on unmodified `main`, unrelated to this change.) - `sbt WorkflowExecutionService/Test/compile` and `WorkflowExecutionService/testOnly ...SequentialRecordStorageSpec` (covers the `HDFSRecordStorage` code path): 9 tests passed. - `IcebergUtilSpec` and `DocumentFactorySpec` (Iceberg + hadoop `Configuration` usage) pass, confirming nothing needed `hadoop-mapreduce-client-core`. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Yicong-Huang
added a commit
that referenced
this pull request
Jul 6, 2026
…mon CVEs (#6203) ### What changes were proposed in this PR? Backport of #6201 to `release/v1.2`: upgrade `org.apache.hadoop:hadoop-common` to 3.4.3 in both places it is pinned (`common/workflow-core/build.sbt` was at 3.3.1, `amber/build.sbt` at 3.3.3), and upgrade `hadoop-mapreduce-client-core` to 3.4.3 alongside it with hadoop 3.4's new direct `netty-all` dependency excluded. This resolves the same 4 Dependabot alerts on `hadoop-common` for the release line: - [#656](https://github.com/apache/texera/security/dependabot/656) CVE-2022-25168 (critical, argument injection in `FileUtil.unTar`, fixed in 3.3.3) - [#732](https://github.com/apache/texera/security/dependabot/732) CVE-2022-26612 (critical, fixed in 3.3.3) - [#654](https://github.com/apache/texera/security/dependabot/654) CVE-2021-37404 (critical, heap overflow in libhdfs, fixed in 3.3.2) - [#700](https://github.com/apache/texera/security/dependabot/700) CVE-2024-23454 (low, fixed in 3.4.0) Cherry-pick of 1094a65 with conflicts resolved: - `common/workflow-core/build.sbt`: took only this PR's `excludeNetty` addition; the adjacent `log4jVersion` line in the conflict hunk belongs to #6199, which is not on `release/v1.2` (its backport #6202 is separate and still open). - The 8 `LICENSE-binary`(-java) / `NOTICE-binary` manifests were **regenerated from freshly built `release/v1.2` dists** rather than taken from `main`, because `main`'s manifests reflect deps this branch does not have: the log4j 2.26.1 bridge entries (from #6199) are removed, and pekko stays 1.2.1 (main: 1.6.0), ssl-config-core 0.6.1 (main: 0.7.1), slf4j-api 2.0.16 (main: 2.0.17). `NOTICE-binary` files were regenerated with `generate_notice_binary.py` (amber with `--extras amber/NOTICE-binary-python`). **Update:** this PR also folds in the startup-regression fix from #6205 (issue #6204), so the hadoop upgrade lands on `release/v1.2` without ever breaking service startup. Hadoop 3.4 ships Jersey 1's JSON module via the fork `com.github.pjfanning:jersey-json`, which the existing `com.sun.jersey` exclusion rules do not catch; its `META-INF/services` entries register Jersey 1.x providers whose superclasses live in the (excluded) jersey-core 1.x, so Dropwizard's Jersey 2 `ServiceFinder` dies with `NoClassDefFoundError: com/sun/jersey/core/provider/jaxb/AbstractRootElementProvider` on boot. The fix excludes `com.github.pjfanning:jersey-json` from the hadoop dependencies in `common/workflow-core/build.sbt` and `amber/build.sbt`, and removes the two jars that consequently leave the dists (`jersey-json-1.22.0`, its transitive `jaxb-impl-2.2.3-1`) from the four `LICENSE-binary`(-java) manifests, regenerated against dists rebuilt on this branch. ### Any related issues, documentation, discussions? Backports #6201 (merged to `main`), which resolves #6200. Also includes the fix from #6205 for #6204 (startup regression introduced by #6201). No `release/*` label is added — this PR *is* the backport. ### How was this PR tested? Binary licensing manifests were verified the same way CI checks them: built the four affected dists on this branch (`sbt WorkflowExecutionService/dist FileService/dist ComputingUnitManagingService/dist WorkflowCompilingService/dist`), then ran `check_binary_deps.py` in strict (nightly) mode against each unzipped dist — all four pass (`OK: 395/300/337/312 JVM jars match LICENSE-binary`). For the folded-in #6205 fix: rebuilt all four dists on this branch after adding the exclusion and verified via `unzip -l` that `jersey-json` and `jaxb-impl` are no longer bundled in any of them; `check_binary_deps.py` re-run against the fresh dists passes for all four services (`OK: 393/298/310/335 JVM jars match LICENSE-binary`), and `generate_notice_binary.py` reports no `NOTICE-binary` drift (the removed jars carry no NOTICE entries). On `main` the same exclusion was verified end to end: `texera-web` reproducibly crashes on boot without it and starts cleanly with it (Jetty up on :8080). Tested with existing test cases: - `sbt WorkflowCore/test`: 294 tests passed, 0 failed. (4 suites — `S3StorageClientSpec` and the `LargeBinary*` specs — abort because they need an external MinIO service; verified they abort identically on unmodified `release/v1.2`, unrelated to this change.) - `sbt WorkflowExecutionService/Test/compile` passes (95 test sources; `SequentialRecordStorageSpec`, run on `main` for the `HDFSRecordStorage` path, does not exist on `release/v1.2` — `HDFSRecordStorage` itself compiles against hadoop 3.4.3). - `sbt "WorkflowCore/testOnly *IcebergUtilSpec"` (Iceberg + hadoop `Configuration` usage): 14 tests passed. (`DocumentFactorySpec` from the original PR's test list also does not exist on `release/v1.2`.) ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Xinyuan Lin <xinyual3@uci.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.