Skip to content

test(ci): add per-service boot smoke tests#6271

Closed
mengw15 wants to merge 3 commits into
apache:mainfrom
mengw15:test-6220-service-boot-smoke
Closed

test(ci): add per-service boot smoke tests#6271
mengw15 wants to merge 3 commits into
apache:mainfrom
mengw15:test-6220-service-boot-smoke

Conversation

@mengw15

@mengw15 mengw15 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

CI builds each service's distributable and runs its unit tests, but never starts the packaged service, so runtime classpath/linkage problems that compile and unit-test clean — but crash the moment main runs — slip through. Two recent dependency bumps did exactly this on main, caught only manually after merge: #6201 (Hadoop 3.4.3) crashed texera-web on boot with NoClassDefFoundError (fixed by #6205), and #6185 (Arrow 19) crashed computing-unit-managing-service at startup with a Jackson version conflict (fixed by #6209).

This PR adds a boot smoke test:

  • .github/scripts/smoke-boot.sh — launches a packaged service from its unpacked dist and asserts it either reaches a listening state (strict) or, for a service that fail-fasts on infra this job doesn't provision, exits without a linkage/module error (shallow). A NoClassDefFoundError / LinkageError / NoSuchMethodError / Jackson-module conflict fails the check in any mode.
  • platform job — one boot check per matrix service, against the postgres already provisioned there. file-service uses shallow (it fail-fasts on MinIO + LakeFS, not provisioned here); the rest use strict.
  • amber job — a strict boot check for texera-web.

Both past incidents crash early in init, so this catches that class of regression before merge without a fully provisioned environment.

Any related issues, documentation, discussions?

Closes #6220.
Incidents this would have caught: #6204 / #6205 (texera-web, Hadoop→Jersey), #6206 / #6209 (computing-unit-managing-service, Arrow→Jackson). Triggering upgrades: #6201, #6185.

How was this PR tested?

Validated locally against a CI-equivalent postgres (postgres:latest, postgres/postgres, schema from sql/texera_ddl.sql + sql/iceberg_postgres_catalog.sql + sql/texera_lakefs.sql):

  • config-service (representative of the postgres-only platform services) launched from its stage dist → smoke-boot.sh … 9094 strict reached LISTEN, exit 0.
  • With postgres stopped: strict correctly failed (exit 1, HikariCP connection-refused dumped); shallow correctly passed (exit 0, no linkage error) — validating the file-service path and that infra-connection failures are not mistaken for linkage errors.
  • texera-web launched from the amber stage dist → smoke-boot.sh … 8080 strict reached LISTEN, exit 0 (confirms amberHomePath config resolution and a postgres-only boot with no frontend-asset dependency).

The other services (access-control-service, computing-unit-managing-service, workflow-compiling-service, notebook-migration-service) follow the same Dropwizard + SqlServer.initConnection postgres-only boot pattern as config-service.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-opus-4-8)

CI builds each service's dist and runs its unit tests but never starts
the packaged service, so runtime classpath/linkage crashes on boot slip
through — e.g. apache#6204 (Hadoop 3.4 -> Jersey NoClassDefFoundError, fixed by
apache#6205) and apache#6206 (Arrow 19 -> Jackson version conflict, fixed by apache#6209),
both caught only after landing on main.

Add .github/scripts/smoke-boot.sh, which launches a packaged service from
its unpacked dist and asserts it reaches a listening state (strict) or
exits without a linkage/module error (shallow). Wire it into the platform
matrix (one check per service, against the postgres already provisioned
there; file-service uses shallow) and the amber job (texera-web, strict).

Closes apache#6220
@github-actions github-actions Bot added the ci changes related to CI label Jul 8, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • No candidates found from git blame history.

@codecov-commenter

codecov-commenter commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.64%. Comparing base (18ccda9) to head (7e425c1).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #6271      +/-   ##
============================================
- Coverage     60.67%   60.64%   -0.03%     
- Complexity     3368     3372       +4     
============================================
  Files          1133     1133              
  Lines         44141    44141              
  Branches       4825     4825              
============================================
- Hits          26782    26770      -12     
- Misses        15901    15913      +12     
  Partials       1458     1458              
Flag Coverage Δ
access-control-service 70.00% <ø> (ø)
agent-service 44.59% <ø> (ø)
amber 66.16% <ø> (+0.01%) ⬆️
computing-unit-managing-service 0.00% <ø> (ø)
config-service 52.30% <ø> (ø)
file-service 63.97% <ø> (ø)
frontend 52.37% <ø> (-0.07%) ⬇️
notebook-migration-service 78.57% <ø> (ø)
pyamber 91.18% <ø> (ø)
workflow-compiling-service 55.14% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

✅ No material benchmark regressions detected

🟢 2 better · 🔴 0 worse · ⚪ 13 noise (<±5%) · 0 without baseline

Compared against main 18ccda9 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
bs=10 sw=10 sl=64 411 0.251 23,244/37,371/37,371 us ⚪ within ±5% / 🔴 +138.8%
🟢 bs=100 sw=10 sl=64 943 0.576 104,040/123,576/123,576 us 🟢 -7.6% / 🔴 +13.2%
bs=1000 sw=10 sl=64 1,097 0.67 910,116/942,032/942,032 us ⚪ within ±5% / 🟢 -12.5%
Baseline details

Latest main 18ccda9 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 411 tuples/sec 419 tuples/sec 760.25 tuples/sec -1.9% -45.9%
bs=10 sw=10 sl=64 MB/s 0.251 MB/s 0.256 MB/s 0.464 MB/s -2.0% -45.9%
bs=10 sw=10 sl=64 p50 23,244 us 22,373 us 12,873 us +3.9% +80.6%
bs=10 sw=10 sl=64 p95 37,371 us 35,816 us 15,648 us +4.3% +138.8%
bs=10 sw=10 sl=64 p99 37,371 us 35,816 us 19,338 us +4.3% +93.3%
bs=100 sw=10 sl=64 throughput 943 tuples/sec 930 tuples/sec 974.75 tuples/sec +1.4% -3.3%
bs=100 sw=10 sl=64 MB/s 0.576 MB/s 0.568 MB/s 0.595 MB/s +1.4% -3.2%
bs=100 sw=10 sl=64 p50 104,040 us 101,943 us 102,470 us +2.1% +1.5%
bs=100 sw=10 sl=64 p95 123,576 us 133,676 us 109,144 us -7.6% +13.2%
bs=100 sw=10 sl=64 p99 123,576 us 133,676 us 115,530 us -7.6% +7.0%
bs=1000 sw=10 sl=64 throughput 1,097 tuples/sec 1,107 tuples/sec 1,000 tuples/sec -0.9% +9.7%
bs=1000 sw=10 sl=64 MB/s 0.67 MB/s 0.676 MB/s 0.61 MB/s -0.9% +9.8%
bs=1000 sw=10 sl=64 p50 910,116 us 898,936 us 1,005,545 us +1.2% -9.5%
bs=1000 sw=10 sl=64 p95 942,032 us 973,498 us 1,045,968 us -3.2% -9.9%
bs=1000 sw=10 sl=64 p99 942,032 us 973,498 us 1,076,800 us -3.2% -12.5%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,486.23,200,128000,411,0.251,23243.88,37371.43,37371.43
1,100,10,64,20,2119.86,2000,1280000,943,0.576,104039.68,123575.96,123575.96
2,1000,10,64,20,18232.74,20000,12800000,1097,0.670,910115.63,942031.90,942031.90

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds CI boot smoke tests to ensure each packaged Texera service can actually start from its built dist, catching runtime classpath/linkage regressions (e.g., NoClassDefFoundError, LinkageError, Jackson/Scala-module conflicts) that compile and unit-test clean.

Changes:

  • Introduces .github/scripts/smoke-boot.sh to launch a dist executable and assert either “LISTEN” (strict) or “no linkage error” (shallow) within a timeout.
  • Extends the platform job matrix to run a per-service boot check (strict for most services; shallow for file-service).
  • Adds an amber-job boot check for texera-web from the amber dist.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.github/workflows/build.yml Adds per-service (and amber) CI steps to boot the freshly built/unzipped dist artifacts.
.github/scripts/smoke-boot.sh New helper script implementing strict/shallow boot validation and linkage-error detection.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/scripts/smoke-boot.sh Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Meng Wang <mengw15@uci.edu>
@mengw15 mengw15 marked this pull request as ready for review July 8, 2026 17:42
@mengw15 mengw15 requested a review from Yicong-Huang July 8, 2026 17:43

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this smoke-boot script, or maybe local-dev.sh up <service> is enough?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the point is I want to have less scripts, so testing env can be closer to dev env, and closer to prod env. if we have separated scripts, it is easy to drift: something works in dev, but won't pass in CI, or something works in CI, won't actually work in prod.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can write the simple logic:

try local-dev.sh up
detect port licensing
timeout after 5s

@mengw15

mengw15 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #6274 — the platform-service boot checks moved into a dedicated platform-integration job that provisions real infra (postgres + MinIO + LakeFS), so file-service boots strict there instead of shallow. smoke-boot.sh carries over unchanged.

texera-web's boot check will follow with the amber-integration work (#6047); #6220 stays open to track it. Closing in favor of #6274.

@mengw15 mengw15 closed this Jul 8, 2026
tanishqgandhi1908 pushed a commit to tanishqgandhi1908/texera that referenced this pull request Jul 10, 2026
…fra (apache#6274)

### What changes were proposed in this PR?

The platform services' build jobs (the `platform` matrix) provision only
postgres. A boot smoke test there can't fully start a service that needs
more — `file-service` fail-fasts on **MinIO + LakeFS** during `run()`
(S3 bucket creation + `LakeFSStorageClient.healthCheck()`), so it can
only be checked shallowly.

This PR adds a **`platform-integration` job** (mirroring
`amber-integration`) that provisions **postgres + MinIO + LakeFS** and
boots each packaged platform service from its dist, asserting it reaches
a listening state without a runtime classpath/linkage crash (via
`.github/scripts/smoke-boot.sh`, from apache#6220). Every service has its real
dependencies here, so all run in **strict** mode — `file-service`
included.

- Dists are shipped in as **CI artifacts** from the `platform` job
(`upload-artifact` → `download-artifact`), not rebuilt.
- Infra is provisioned **once for the whole tier** (single job), like
`amber-integration`.
- **Ubuntu-only**, matching the `platform` job's OS scope.
(`amber-integration` is cross-OS because its Python-UDF / native specs
are arm64-mac-sensitive; these boot checks are pure-JVM and
OS-independent.)

This is the infra-provisioned home for the platform services'
integration checks (apache#6273); the boot smoke test is its first inhabitant,
and future platform integration tests (e.g. `file-service`'s S3 / LakeFS
paths) can live here too.

### Any related issues, documentation, discussions?

Closes apache#6273. The boot smoke test script is from apache#6220.

Follow-ups: the infra provisioning is now duplicated with
`amber-integration` — extracting it into a shared composite action is
tracked in apache#6275. (These separate `amber-integration` /
`platform-integration` jobs are deliberate — apache#6047, re-scoped, covers
the complementary sbt-side `@IntegrationTest` mechanism, not a single
unified job.) A `texera-web` boot check, out of this PR's platform-only
scope, is tracked in apache#6276 (under the apache#6220 umbrella).

### How was this PR tested?

- LakeFS provisioning recipe validated locally (postgres + MinIO +
LakeFS via docker) → LakeFS reaches healthy in ~9s.
- Confirmed `storage.conf` defaults align with the provisioned endpoints
(S3 `http://localhost:9000`, LakeFS `http://localhost:8000/api/v1`, JDBC
`localhost:5432`), so each service boots against this infra with no
extra env.
- `smoke-boot.sh` was validated end-to-end in apache#6271 (config-service +
texera-web reached LISTEN in CI; the negative case caught a reintroduced
apache#6206 Arrow / Jackson boot crash).
- `--network host` reachability, the artifact flow, and each service's
strict boot are exercised by this workflow's own CI run.

### Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-opus-4-8)

---------

Signed-off-by: Meng Wang <mengw15@uci.edu>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci changes related to CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CI smoke tests that each service boots successfully

4 participants