test(integration): run the Alpine distro build with a musl-capable envd - #3444
Conversation
PR SummaryMedium Risk Overview Reviewed by Cursor Bugbot for commit 7201839. Bugbot is set up for automated code reviews on this repo. Configure here. |
❌ 5 Tests Failed:
View the top 2 failed test(s) by shortest run time
View the full list of 8 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
There was a problem hiding this comment.
LGTM — this is a well-scoped test/build-tooling fix with a clear, verified root cause.
What was reviewed: the build-debug Makefile target and the new DEBUG_LDFLAGS static-link/tag combination, and the distro test's removal of the Alpine skip. Confirmed build (the production envd binary) is untouched — only the debug binary used by the integration test host changes. Also considered the CI failures reported by codecov on this run, but they're in unrelated template-tag tests, not the modified distro_build_test.go, and match tests with pre-existing high flake rates on main (10-25%), so they don't appear caused by this change.
Extended reasoning...
Overview
This PR touches two files: packages/envd/Makefile (adds a DEBUG_LDFLAGS var with -linkmode external -extldflags=-static and switches build-debug to use it plus -tags osusergo,netgo) and tests/integration/internal/tests/api/templates/distro_build_test.go (removes the skip for the Alpine test case, since the underlying blocker is fixed). The production build target (CGO_ENABLED=0, PROD_LDFLAGS) is unchanged — this only affects the race-instrumented debug binary that the integration test host injects into guest VMs.
Security risks
None. This is a test/dev tooling change to how the debug envd binary is linked; it does not touch auth, crypto, permissions, or any production code path.
Level of scrutiny
Low-to-medium is appropriate. The change is confined to test infrastructure (the integration host's debug envd) rather than the shipped production binary, so the blast radius is limited to CI/dev workflows. The PR description shows solid manual verification: all four distro builds passing with the static binary on a KVM dev stack, confirmation the race detector still has its ~297 __tsan symbols and still catches a deliberately racy program on musl, and confirmation guest console output still reaches orchestrator.log for the workflow's data-race grep.
Other factors
The bug-hunting pass found only a cosmetic nit (a leftover unused LDFLAGS variable), which doesn't warrant blocking. I also checked the codecov failures posted on this PR — they're in template_tags_test.go and build_template_test.go, unrelated to the modified distro_build_test.go, and several are explicitly flagged with pre-existing flake rates in main (10-25%), so they read as ambient CI flakiness rather than a regression from this change. Cursor's bot flagged this as "Medium Risk" because it changes how every debug envd binary is built, but since build-debug is test/dev-only and the production build path is untouched, I don't think that risk rating changes the calculus here.
| @@ -7,6 +7,7 @@ ENV := $(shell cat ../../.last_used_env || echo "not-set") | |||
| BUILD := $(shell git rev-parse HEAD | cut -c1-7) | |||
| LDFLAGS=-ldflags "-X=main.commitSHA=$(BUILD)" | |||
There was a problem hiding this comment.
🟡 The LDFLAGS variable defined at packages/envd/Makefile:8 is now unused — this PR switches build-debug (its only consumer) over to the new DEBUG_LDFLAGS. It's dead code left over from the refactor and can be safely deleted.
Extended reasoning...
What happened: This PR adds a new DEBUG_LDFLAGS variable (line 10) with the extra -linkmode external -extldflags=-static flags needed to statically link the race-instrumented debug binary so it can run on musl guests. It correctly updates the build-debug target (line 80) to use ${DEBUG_LDFLAGS} instead of ${LDFLAGS}.
However, the original LDFLAGS=-ldflags \"-X=main.commitSHA=$(BUILD)\" definition on line 8 was left in place. Before this PR, build-debug was the sole consumer of LDFLAGS (the build target has always used the separate PROD_LDFLAGS). After this PR, nothing references LDFLAGS anywhere in the file.
Proof:
- Before the PR:
grep LDFLAGS packages/envd/MakefileshowsLDFLAGSdefined on line 8 and consumed via${LDFLAGS}on line 80 (build-debug). - After the PR: line 80 now reads
CGO_ENABLED=1 go build -race -tags osusergo,netgo -gcflags=all=\"-N -l\" -o bin/debug/envd ${DEBUG_LDFLAGS}— the only other ldflags usage isPROD_LDFLAGSin the unrelatedbuildtarget (line 73). - A boundary-aware grep for the bare
LDFLAGStoken (excludingPROD_LDFLAGS/DEBUG_LDFLAGS) matches only the definition on line 8, confirming it has zero remaining references.
Impact: None functionally — Make does not warn on unused variables, and there's no implicit rule here that would pick it up accidentally, so the build behaves identically whether or not the line is removed. It's purely a leftover from the refactor that muddies the file (a future reader might wonder if it's still used somewhere, or reintroduce a stale reference).
Fix: Delete line 8 (LDFLAGS=-ldflags \"-X=main.commitSHA=$(BUILD)\") since PROD_LDFLAGS and DEBUG_LDFLAGS now cover both build targets.
) ## What Rebase + extension of #3235 (cc @dobrac), squashed onto current main after #3462 landed. Opened as a **draft for a measuring run** — the shard timings from this PR's CI decide the final shard balance. Two things changed since #3235 was written, and this branch reconciles both: - #3462 already landed the coverage philosophy (uncompressed runs the whole suite once; zstd1/lz4 only re-run the allow-list). This rebase **keeps that decision** and applies #3235's sharding on top — PRs now shard the *uncompressed* config, not zstd1. - The distro-family template tests (#3437/#3444) landed into what is now templates shard 2, so the name-prefix split needs rebalancing from this run's junits. ## PR-path matrix (8 jobs; push to main unchanged: 3 unsharded configs) | Job | Shard | Runs | |---|---|---| | uncompressed-templates-1/2 | `^TestTemplateBuild` split by name prefix | 11 + 9 real-build tests | | uncompressed-sandboxes | sandboxes + metrics + volumes + proxies | 121 tests | | uncompressed-rest | everything else, `-skip '^TestTemplateBuild'` | 86 tests | | zstd1-templates / lz4-templates | templates package ∩ allow-list | 6 snapshot-build entries | | zstd1-other / lz4-other | package complement ∩ allow-list | 33 entries | Static partition verified: 227 top-level tests land exactly once across the uncompressed shards; templates/no-templates is an exact package complement. `TESTS_ONLY` composes only with package-pure shards (the name-split shards' own `-run` would collide — guarded, fails loudly). Unknown shard names fail loudly instead of silently running the whole suite. ## Carried from #3235 (unchanged in spirit) - DB containers start right after checkout and overlap the Go builds (`start-databases`); `start-services` waits with bounded health loops. Docker Hub pre-pull retries (#3410) folded in, now covering the pinned tags + otel. - ClickHouse migrations via goose on the host (`migrate-host`) — no migrator image build. - Salted optimized Go cache; `DEBUG_GCFLAGS` opt-out (race stays on, `-N -l` off in CI). envd keeps the musl-static link from #3444, gcflags parametrized. - 7 metadata/authz template tests use `RequestTemplateWithoutBuild`; envd process tests poll instead of fixed sleeps. ## Expected (to be confirmed by this run) Baseline on main (run 30626752076): integration jobs 8.4–9.9 min, PR wall ~10–12 min. Expected here: **~5–5.5 min wall**, long pole likely uncompressed-templates-2 (it inherited the distro suite). Follow-ups after measurement: rebalance `TEMPLATE_BUILDS_SHARD1_RE`, cache the built sandbox template keyed on inputs. Relationship to #3235: this supersedes the mechanical parts; adopt into the original PR or take this one over — @dobrac's call. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…vd (#3444) `make build-debug` builds envd with `-race`, which needs cgo, so the binary the integration host injects into every guest is dynamically linked against glibc — and a musl guest has no loader for it. envd never answers, the base layer times out and the build dies with a masked internal error, which is why the Alpine case was skipped. Linking it statically fixes that; `osusergo,netgo` keep user and DNS lookups pure Go as in the `CGO_ENABLED=0` binary we ship, so only the race instrumentation differs. The detector is intact (297 `__tsan` symbols; a deliberate racy program built with the same flags still reports on musl), and guest console output reaches `orchestrator.log`, so the workflow's data-race grep keeps covering envd. Checked on a KVM dev stack, cold cache each time: with the old binary ubuntu/fedora/arch pass and alpine fails on `wait for envd: syncing took too long`; with the static one all four pass (alpine 30s). `envd -version` also runs under `alpine:3.24`, where the old binary gives `exec: no such file or directory`.
) ## What Rebase + extension of #3235 (cc @dobrac), squashed onto current main after #3462 landed. Opened as a **draft for a measuring run** — the shard timings from this PR's CI decide the final shard balance. Two things changed since #3235 was written, and this branch reconciles both: - #3462 already landed the coverage philosophy (uncompressed runs the whole suite once; zstd1/lz4 only re-run the allow-list). This rebase **keeps that decision** and applies #3235's sharding on top — PRs now shard the *uncompressed* config, not zstd1. - The distro-family template tests (#3437/#3444) landed into what is now templates shard 2, so the name-prefix split needs rebalancing from this run's junits. ## PR-path matrix (8 jobs; push to main unchanged: 3 unsharded configs) | Job | Shard | Runs | |---|---|---| | uncompressed-templates-1/2 | `^TestTemplateBuild` split by name prefix | 11 + 9 real-build tests | | uncompressed-sandboxes | sandboxes + metrics + volumes + proxies | 121 tests | | uncompressed-rest | everything else, `-skip '^TestTemplateBuild'` | 86 tests | | zstd1-templates / lz4-templates | templates package ∩ allow-list | 6 snapshot-build entries | | zstd1-other / lz4-other | package complement ∩ allow-list | 33 entries | Static partition verified: 227 top-level tests land exactly once across the uncompressed shards; templates/no-templates is an exact package complement. `TESTS_ONLY` composes only with package-pure shards (the name-split shards' own `-run` would collide — guarded, fails loudly). Unknown shard names fail loudly instead of silently running the whole suite. ## Carried from #3235 (unchanged in spirit) - DB containers start right after checkout and overlap the Go builds (`start-databases`); `start-services` waits with bounded health loops. Docker Hub pre-pull retries (#3410) folded in, now covering the pinned tags + otel. - ClickHouse migrations via goose on the host (`migrate-host`) — no migrator image build. - Salted optimized Go cache; `DEBUG_GCFLAGS` opt-out (race stays on, `-N -l` off in CI). envd keeps the musl-static link from #3444, gcflags parametrized. - 7 metadata/authz template tests use `RequestTemplateWithoutBuild`; envd process tests poll instead of fixed sleeps. ## Expected (to be confirmed by this run) Baseline on main (run 30626752076): integration jobs 8.4–9.9 min, PR wall ~10–12 min. Expected here: **~5–5.5 min wall**, long pole likely uncompressed-templates-2 (it inherited the distro suite). Follow-ups after measurement: rebalance `TEMPLATE_BUILDS_SHARD1_RE`, cache the built sandbox template keyed on inputs. Relationship to #3235: this supersedes the mechanical parts; adopt into the original PR or take this one over — @dobrac's call. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
make build-debugbuilds envd with-race, which needs cgo, so the binary the integration host injects into every guest is dynamically linked against glibc — and a musl guest has no loader for it. envd never answers, the base layer times out and the build dies with a masked internal error, which is why the Alpine case was skipped. Linking it statically fixes that;osusergo,netgokeep user and DNS lookups pure Go as in theCGO_ENABLED=0binary we ship, so only the race instrumentation differs. The detector is intact (297__tsansymbols; a deliberate racy program built with the same flags still reports on musl), and guest console output reachesorchestrator.log, so the workflow's data-race grep keeps covering envd.Checked on a KVM dev stack, cold cache each time: with the old binary ubuntu/fedora/arch pass and alpine fails on
wait for envd: syncing took too long; with the static one all four pass (alpine 30s).envd -versionalso runs underalpine:3.24, where the old binary givesexec: no such file or directory.