panic: send on closed channel in util/progress Printer during remote integration tests

[lohitkolluri]

Summary

Observed a panic: send on closed channel in github.com/docker/buildx/util/progress.(*Printer).Write while running integration tests with the remote worker. This appears to be a lifecycle race where Write() can run after Wait() has closed p.status.

Reproduction

From the buildx repo:

# Narrow reproduction (remote worker)
TESTFLAGS=\"--run /TestBuildPolicyRemotePolicyFiles/worker=remote/remote-policy-no-local-fallback -v\" make test-integration

(See .github/CONTRIBUTING.md for running integration tests with a specific worker.)

Environment / versions

• BuildKit: v0.28.1 (from CI job: test-integration (v0.28.1, remote, ./tests, experimental))
• Worker/driver: remote

Stack trace

panic: send on closed channel

goroutine 109 [running]:
github.com/docker/buildx/util/progress.(*Printer).Write(0x25a72f19810, 0x25a72da3260)
	github.com/docker/buildx/util/progress/printer.go:89 +0x4d
github.com/docker/buildx/util/progress.(*prefixed).Write(0x25a732337a0?, 0x25a72e6a3c0?)
	github.com/docker/buildx/util/progress/multiwriter.go:32 +0x82
github.com/docker/buildx/util/progress.NewChannel.func1()
	github.com/docker/buildx/util/progress/writer.go:73 +0x6a
created by github.com/docker/buildx/util/progress.NewChannel in goroutine 108
	github.com/docker/buildx/util/progress/writer.go:54 +0xd3

Notes

In the failing run, the test TestIntegration/TestBuildPolicyRemotePolicyFiles/worker=remote/remote-policy-no-local-fallback was expecting an error message about a missing policy file, but the panic occurred while progress was still emitting status updates.

Ask

• Add a targeted regression test that exercises the lifecycle where Write() can race with Wait() closing p.status.
• Fix the actual lifecycle race (ideally without making normal-operation progress writes lossy, except perhaps during shutdown).

[lohitkolluri]

Tracking fix/PR: #3873

[crazy-max]

• (from CI job: test-integration (v0.28.1, remote, ./tests, experimental))

Can you give a link to the workflow run please?

[lohitkolluri]

@crazy-max

Here’s the failing CI run with the panic in test-integration (v0.28.1, remote, ./tests, experimental):
https://github.com/docker/buildx/actions/runs/26557909549
(direct job link: https://github.com/docker/buildx/actions/runs/26557909549/job/78233681642)

[crazy-max]

@crazy-max

Here’s the failing CI run with the panic in test-integration (v0.28.1, remote, ./tests, experimental): https://github.com/docker/buildx/actions/runs/26557909549 (direct job link: https://github.com/docker/buildx/actions/runs/26557909549/job/78233681642)

It seems this runs against changes made from #3872 and not current master. Do you have a repro with master branch? If not this might linked to an issue with the implementation made in #3872

[lohitkolluri]

I tried reproducing this locally on current master but couldn’t trigger it and the test passed without the send on closed channel panic.

The only occurrence I still have is from the CI run in PR #3872:
https://github.com/docker/buildx/actions/runs/26557909549

If there’s a specific CI config difference I should mirror, or any suggestions to make it reproduce reliably on master, happy to try.

[lohitkolluri]

I spent some time trying to reproduce this directly on current master (f5fb6314e) using the same narrow integration filter from the issue:

TESTFLAGS='--run /TestBuildPolicyRemotePolicyFiles/worker=remote/remote-policy-no-local-fallback -v' make test-integration

I also retried with -count=10 and TEST_BUILDKIT_TAG=v0.28.1, but locally the subtest consistently passed and returned the expected error (policy file does-not-exist.rego not found) without triggering the panic.

That said, while the exact integration repro appears timing-sensitive, the underlying lifecycle race still exists on master in util/progress. Printer.Wait() closes p.status, while goroutines created via progress.NewChannel can still forward status updates through Write(). I was able to reproduce the same send on closed channel panic reliably with focused stress/unit tests around the NewChannel + Wait() shutdown path, including under go test -race, with stacks matching the CI failure (prefixed.Write → NewChannel.func1 → printer.go:89).

Also, #3872 itself only changes commands/build.go and related validate/repro wiring — it doesn’t modify util/progress or the remote policy loader directly. So my current assumption is that the CI run exposed a pre-existing timing issue rather than introducing a new one in the progress package.

I’m happy to keep investigating for a deterministic integration repro on master, but I think the safest path is still to add a targeted regression test around the printer shutdown lifecycle and fix the Write()/shutdown coordination there.

[crazy-max]

Let's close this one for now as not reproducible but thanks for the analysis

[lohitkolluri]

@crazy-max hey was someone able to reproduce this issue?