ci: add platform-integration job to boot each service against real infra

[mengw15]

What changes were proposed in this PR?

The platform services' build jobs (the platform matrix) provision only postgres. A boot smoke test there can't fully start a service that needs more — file-service fail-fasts on MinIO + LakeFS during run() (S3 bucket creation + LakeFSStorageClient.healthCheck()), so it can only be checked shallowly.

This PR adds a platform-integration job (mirroring amber-integration) that provisions postgres + MinIO + LakeFS and boots each packaged platform service from its dist, asserting it reaches a listening state without a runtime classpath/linkage crash (via .github/scripts/smoke-boot.sh, from #6220). Every service has its real dependencies here, so all run in strict mode — file-service included.

• Dists are shipped in as CI artifacts from the platform job (upload-artifact → download-artifact), not rebuilt.
• Infra is provisioned once for the whole tier (single job), like amber-integration.
• Ubuntu-only, matching the platform job's OS scope. (amber-integration is cross-OS because its Python-UDF / native specs are arm64-mac-sensitive; these boot checks are pure-JVM and OS-independent.)

This is the infra-provisioned home for the platform services' integration checks (#6273); the boot smoke test is its first inhabitant, and future platform integration tests (e.g. file-service's S3 / LakeFS paths) can live here too.

Any related issues, documentation, discussions?

Closes #6273. The boot smoke test script is from #6220.

Follow-ups: the infra provisioning is now duplicated with amber-integration — extracting it into a shared composite action is tracked in #6275. (These separate amber-integration / platform-integration jobs are deliberate — #6047, re-scoped, covers the complementary sbt-side @IntegrationTest mechanism, not a single unified job.) A texera-web boot check, out of this PR's platform-only scope, is tracked in #6276 (under the #6220 umbrella).

How was this PR tested?

• LakeFS provisioning recipe validated locally (postgres + MinIO + LakeFS via docker) → LakeFS reaches healthy in ~9s.
• Confirmed storage.conf defaults align with the provisioned endpoints (S3 http://localhost:9000, LakeFS http://localhost:8000/api/v1, JDBC localhost:5432), so each service boots against this infra with no extra env.
• smoke-boot.sh was validated end-to-end in test(ci): add per-service boot smoke tests #6271 (config-service + texera-web reached LISTEN in CI; the negative case caught a reintroduced computing-unit-managing-service fails to start: jackson-databind 2.21.0 incompatible with jackson-module-scala 2.18.8 #6206 Arrow / Jackson boot crash).
• --network host reachability, the artifact flow, and each service's strict boot are exercised by this workflow's own CI run.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-opus-4-8)

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @aglinxinyuan
  You can notify them by mentioning @aglinxinyuan in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.73%. Comparing base (18ccda9) to head (b197f29).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6274      +/-   ##
============================================
+ Coverage     60.67%   60.73%   +0.06%     
+ Complexity     3368     3363       -5     
============================================
  Files          1133     1135       +2     
  Lines         44141    44191      +50     
  Branches       4825     4830       +5     
============================================
+ Hits          26782    26839      +57     
+ Misses        15901    15892       -9     
- Partials       1458     1460       +2     

Flag	Coverage Δ
access-control-service	70.00% <ø> (ø)
agent-service	44.59% <ø> (ø)
amber	66.10% <ø> (-0.04%)	⬇️
computing-unit-managing-service	0.00% <ø> (ø)
config-service	52.30% <ø> (ø)
file-service	63.97% <ø> (ø)
frontend	52.65% <ø> (+0.21%)	⬆️
notebook-migration-service	78.94% <ø> (+0.37%)	⬆️
pyamber	91.18% <ø> (ø)
workflow-compiling-service	55.14% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 3 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 29233cc benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	355	0.217	29,159/34,055/34,055 us	🔴 +16.7% / 🔴 +132.3%
🟢	bs=100 sw=10 sl=64	768	0.469	132,771/137,663/137,663 us	🟢 -11.4% / 🔴 +32.9%
⚪	bs=1000 sw=10 sl=64	870	0.531	1,152,848/1,206,411/1,206,411 us	⚪ within ±5% / 🔴 +17.3%

Baseline details

Latest main 29233cc from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	355 tuples/sec	388 tuples/sec	779.42 tuples/sec	-8.5%	-54.5%
bs=10 sw=10 sl=64	MB/s	0.217 MB/s	0.237 MB/s	0.476 MB/s	-8.4%	-54.4%
bs=10 sw=10 sl=64	p50	29,159 us	24,979 us	12,550 us	+16.7%	+132.3%
bs=10 sw=10 sl=64	p95	34,055 us	38,751 us	15,115 us	-12.1%	+125.3%
bs=10 sw=10 sl=64	p99	34,055 us	38,751 us	18,652 us	-12.1%	+82.6%
bs=100 sw=10 sl=64	throughput	768 tuples/sec	765 tuples/sec	1,000 tuples/sec	+0.4%	-23.2%
bs=100 sw=10 sl=64	MB/s	0.469 MB/s	0.467 MB/s	0.61 MB/s	+0.4%	-23.2%
bs=100 sw=10 sl=64	p50	132,771 us	129,949 us	99,872 us	+2.2%	+32.9%
bs=100 sw=10 sl=64	p95	137,663 us	155,411 us	107,073 us	-11.4%	+28.6%
bs=100 sw=10 sl=64	p99	137,663 us	155,411 us	114,377 us	-11.4%	+20.4%
bs=1000 sw=10 sl=64	throughput	870 tuples/sec	877 tuples/sec	1,020 tuples/sec	-0.8%	-14.7%
bs=1000 sw=10 sl=64	MB/s	0.531 MB/s	0.535 MB/s	0.622 MB/s	-0.7%	-14.7%
bs=1000 sw=10 sl=64	p50	1,152,848 us	1,144,423 us	987,499 us	+0.7%	+16.7%
bs=1000 sw=10 sl=64	p95	1,206,411 us	1,184,948 us	1,028,784 us	+1.8%	+17.3%
bs=1000 sw=10 sl=64	p99	1,206,411 us	1,184,948 us	1,057,136 us	+1.8%	+14.1%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,563.47,200,128000,355,0.217,29158.58,34054.84,34054.84
1,100,10,64,20,2605.15,2000,1280000,768,0.469,132770.97,137663.07,137663.07
2,1000,10,64,20,22995.19,20000,12800000,870,0.531,1152848.49,1206410.55,1206410.55

[Copilot]

Pull request overview

Adds an infra-provisioned GitHub Actions job to boot each packaged platform service against a more complete “real” dependency stack (Postgres + MinIO + LakeFS), using dists produced by the existing platform matrix and shipped via artifacts. This extends CI coverage to catch runtime/linkage failures that only appear when executing the service launcher.

Changes:

• Upload each platform service’s dist zip from the platform matrix as a CI artifact for downstream consumption.
• Add a platform-integration job that provisions Postgres + MinIO + LakeFS once, downloads all service dists, and runs a strict boot smoke test for each service.
• Introduce .github/scripts/smoke-boot.sh to launch a packaged service, wait for TCP LISTEN, and fail on common runtime classpath/linkage errors.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
.github/workflows/build.yml	Upload platform service dists as artifacts; add platform-integration job that provisions Postgres/MinIO/LakeFS and boots each service from its dist.
.github/scripts/smoke-boot.sh	New boot-smoke helper script that launches a dist binary, waits for LISTEN, and detects common linkage/classpath failures.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mengw15]

Design note — why a dedicated smoke-boot.sh rather than reusing local-dev.sh up:

The script doesn't reimplement how a service starts — it runs bin/<service>, the exact sbt-native-packager launcher that the prod deploy image's CMD and local-dev.sh up both run, reading the same storage.conf. So the launch and config don't drift across dev / CI / prod; the packaged artifact is booted the prod way. What the script adds is only the test harness: start → wait for the port to LISTEN → timeout → grep the log for NoClassDefFoundError / LinkageError / Jackson-module errors → kill. (Same idea as local-dev's own tui_wait_panel port-wait, just standalone.)

local-dev.sh up <service> is the dev-stack orchestrator — it builds, brings up its own docker infra (the texera-local-dev compose), and exports dev creds (texera/password, HOST_LAN_IP). This job instead provisions infra via services: containers + this integration job's MinIO/LakeFS, and boots a pre-built dist shipped in as an artifact (postgres/postgres). Driving that through local-dev would mean overriding its env + --skip-build + bypassing its infra — arguably more glue (and more drift surface) than the thin harness. Booting the packaged artifact is, if anything, closer to prod than running the dev orchestrator.

[Yicong-Huang]

three changes needed:

1. need to declare what label triggers platform integration job. it should be similar to the triggering condition of platform job.
2. please split it into a CI matrix per service, similar to platform job which has a CI run for each service.
3. please do not make it upload + download artifacts: this makes platform-integration depends on platform job, and makes end to end CI time longer. instead, remove this dependency so that they can run in parallel. we can optimize this by extracting a build job as an upstream to both platform and platform-integration jobs depends on it.

[mengw15]

Thanks @Yicong-Huang — all three are addressed in b197f29:

1. Label trigger. Added a dedicated run_platform_integration input mirroring run_amber_integration end-to-end (LABEL_STACKS → decide output → precheck output → both build.yml callers → the job's if:), triggered by the same labels as platform (platform / common / ddl-change / ci). It's also a proper superset of platform's trigger, exactly as amber-integration is of amber: a dedicated platform-integration label fires only the integration stack, so a future spec-only change can run it without the full platform job. No labeler rule applies that label yet — platform has no integration specs today (#6047) — so it's a forward hook for now.

2. Per-service matrix. platform-integration is now a per-service matrix.include (the same 6 services as platform), one boot check per service. Latest CI shows the six platform-integration (<service>) legs.

3. No artifacts / parallel. Dropped the upload/download-artifact hand-off and needs: platform; the two jobs run in parallel now. Rather than the shared build job, I made platform-integration fully independent — it builds each service's dist on its own classpath and boots it — mirroring how amber-integration compiles its own tests independently of amber. That keeps it artifact-free (a shared build job would reintroduce an upload/download hand-off) and matches the existing amber/amber-integration shape. The trade-off is a second compile per service (as amber and amber-integration each compile today), in exchange for zero artifacts and full parallelism.

All six platform-integration legs are green on b197f29: file-service boots against real MinIO + LakeFS and reaches LISTEN on :9092, and because the jobs are independent, a platform-integration leg finishes before the same service's platform leg (they run in parallel).