fix(docker): retry go mod download and cache Go modules - #703
Merged
Conversation
Why: The v1.168.5 release container job failed at `RUN go mod download` with a transient HTTP/2 stream reset from proxy.golang.org. The Docker build fetched all 198 modules uncached with no retry, so a single reset broke the release and left no image published for the tag. What: - Retry `go mod download` up to 5 times with linear backoff (5/10/15/20s). - Mount BuildKit caches for the module cache and the Go build cache. Notes: GOPROXY's `direct` fallback does not cover this error class (only 404/410), so a retry is the only way to absorb it.
Astach
marked this pull request as ready for review
August 31, 2026 12:59
benjaminch
approved these changes
Aug 31, 2026
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 1 file
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Why: The retry loop retried every failure, including ones decidable from go.mod and go.sum alone. A go.sum checksum mismatch can never succeed on retry, so the build burned 5 download passes and ~50s of backoff before failing, and buried Go's SECURITY ERROR banner in retry noise. What: Capture stderr and short-circuit when it matches a non-transient error class (checksum mismatch, missing go.sum entry, go.mod parse or version errors). Everything else stays retried. Notes: Exit codes cannot discriminate here: network failures, checksum mismatches and unknown revisions all exit 1, and -json changes neither the exit code nor the stderr routing, so stderr matching is the only available signal. The match list is a denylist rather than an allowlist of retryable errors on purpose: an unmatched permanent error costs ~50s of CI, while an unmatched transient one would break a release.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Why: The fail-fast list missed deterministic errors, so a bad module path or an unresolvable version still burned 5 download passes and ~50s of backoff. The list was drawn on the wrong axis: local-vs-remote rather than whether the proxy actually answered. `unknown revision` is a definitive negative answer, not a failed round trip, so retrying it cannot help. What: Retry only transport failures. Add `unknown revision`, `malformed module path` and `module lookup disabled` to the non-transient set. Notes: Kept as a denylist rather than an allowlist of retryable errors. The failure that motivated this PR was `stream error: stream ID 171; INTERNAL_ERROR`, a string no hand-written transient allowlist would plausibly have contained; missing an entry there breaks a release, while missing one here costs ~50s. `no matching versions for query` is not reachable from `go mod download` with pinned versions, which report `errors parsing go.mod` or `unknown revision`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The
containerjob of the v1.168.5 release (run 33393047477) failed atDockerfile:12:A transient HTTP/2 stream reset on the way to the Go module proxy. The tag itself was fine —
BuildandRelease Lateston the same commit were green, andgo4.orgis a long-standing indirect dependency.What turned the blip into a failed release is the builder stage, unchanged in this respect since #167 (2023): it re-fetches all 198 modules on every build with no cache and no retry, and one failed request kills the layer, the job, and the image publish. The result was a half-released v1.168.5 — GitHub release, Homebrew, Scoop, AUR and R2 artifacts published, but no Docker image, with
lateston ECR/GHCR still pointing at v1.168.4.What
go mod downloadup to 5 times with linear backoff (5/10/15/20s, ~50s worst case before failing)./go/pkg/mod) and build cache (/root/.cache/go-build), on both the download and build steps.Notes
GOPROXY'sdirectfallback does not help here: it only applies to 404/410, not to a stream reset, so retrying is the only way to absorb this class of error.docker/build-push-actionwithcache-to: type=gha; happy to do that as a follow-up if wanted.public.ecr.aws/r3m4q3r9/pub-mirror-go:1.25.1: image builds andqovery versionreports the injectedAPP_VERSION; retry path exercised withGOPROXY=http://127.0.0.1:1, which logs 5 attempts and exits 1.FromAsCasingwarnings (asvsASon the twoFROMlines) that show up in every build log.Summary by cubic
Makes the Docker build resilient to transient proxy failures by retrying
go mod downloadand caching Go modules, so an intermittent HTTP/2 stream reset no longer fails the release and leaves no image published.Bug Fixes
go mod downloadup to 5 times with 5-second linear backoff on transport failures only.Written for commit f322bd4. Summary will update on new commits.