Skip to content

fix(dist): close piped stdin before waiting on the child - #2806

Open
oguzhanmeteozturk wants to merge 1 commit into
mozilla:mainfrom
oguzhanmeteozturk:fix/dist-close-piped-stdin
Open

fix(dist): close piped stdin before waiting on the child#2806
oguzhanmeteozturk wants to merge 1 commit into
mozilla:mainfrom
oguzhanmeteozturk:fix/dist-close-piped-stdin

Conversation

@oguzhanmeteozturk

Copy link
Copy Markdown

Fixes #2805

What

CommandExt::check_piped drops the ChildStdin it took from the Child before
calling wait_with_output(), so the child sees EOF on stdin.

     pipe(&mut stdin).context("Failed to pipe input to process")?;
+    // `process.stdin` was moved out by the `.take()` above, so
+    // `wait_with_output()` has no stdin left to close. Without this the write
+    // end stays open for the rest of the function and a child that reads to EOF
+    // never returns.
+    drop(stdin);
     let output = process
         .wait_with_output()
         .context("Failed to wait for process to return")?;

Why

check_piped takes stdin out of the Child:

let mut stdin = process.stdin.take().expect("Requested piped stdin but not present");

stdin is owned and lives until the end of the function, so it is still open
while wait_with_output() blocks. Child::wait_with_output closes only the
stdin still held by the Child (drop(self.stdin.take())), which is None
here, making that a no-op.

The closure cannot close it either — its parameter is &mut ChildStdin, so the
callee cannot drop the value. Only check_piped can.

The two callers are DockerBuilder::make_image (build.rs:720) and
DockerBuilder::perform_build (build.rs:771), both of which pipe a tar into
docker cp - <container>:/. docker cp - reads until EOF, so with the write end
held open it never exits, and every job on a type = "docker" build server
blocks indefinitely with its container left in Created.

Both call sites already drop() the corresponding reader right after the call
(drop(toolchain_rdr), drop(inputs_rdr)); this makes the writer symmetric.

Reproducing the underlying behaviour

That docker cp - requires EOF rather than merely a complete archive can be
shown without sccache:

docker run --rm alpine sh -c 'mkdir -p /s && echo hi > /s/f && tar -cf - -C /s f' > /tmp/t.tar
docker create --name cptest alpine true

time ( cat /tmp/t.tar ) | docker cp - cptest:/            # docker cp:  0.019s
time ( cat /tmp/t.tar; sleep 20 ) | docker cp - cptest:/  # docker cp: 20.013s

The archive is complete and valid in both cases; only the writer's behaviour
differs. Measured on Docker 29.7.2, and reproduced on an earlier client, so this
is not specific to a particular release.

Testing

  • Before: a single sccache gcc -c hello.c against a type = "docker" build
    server made no progress; the container stayed in Created and the client fell
    back to local compilation.
  • After: the same compile completes and is reported as a distributed compile.
  • Larger workload: a Linux 6.12 defconfig build distributed 2,867 compilations
    across 11 build servers with no scheduler allocation failures.

Scope

One statement, in a helper used only by the Docker builder. The overlay and
pot builders do not call check_piped. No public API or configuration change.

`CommandExt::check_piped` moves the child's stdin out of the `Child` with
`.take()`, binding it to a local that lives until the end of the function.
`Child::wait_with_output` closes only the stdin still held by the `Child`
(`drop(self.stdin.take())`), which is `None` here, so that is a no-op and the
write end of the pipe stays open while the parent waits for the child to exit.

The closure cannot close it either: its parameter is `&mut ChildStdin`, so the
callee has no way to drop the value. Only `check_piped` can.

Both callers are in `DockerBuilder` and pipe a tar into
`docker cp - <container>:/`, which reads until EOF. With the write end held open
it never exits, so every job on a `type = "docker"` build server blocks
indefinitely with its container left in `Created` and the client falls back to
local compilation.

Both call sites already drop the corresponding reader immediately after the call
(`drop(toolchain_rdr)`, `drop(inputs_rdr)`); this makes the writer symmetric.
@sylvestre

Copy link
Copy Markdown
Collaborator

it will need a test to make sure it won't regress

please make comment #0 a bit shorter in the future

@codecov-commenter

codecov-commenter commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 73.12%. Comparing base (67683cc) to head (f1bc87b).

Files with missing lines Patch % Lines
src/bin/sccache-dist/build.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2806      +/-   ##
==========================================
- Coverage   73.14%   73.12%   -0.02%     
==========================================
  Files          72       72              
  Lines       37615    37592      -23     
==========================================
- Hits        27514    27490      -24     
- Misses      10101    10102       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@oguzhanmeteozturk

Copy link
Copy Markdown
Author

Writing the test exposed a gap: all nine test_dist_* tests go through DistSystem::add_server, which hardcodes BuilderType::Overlay. Nothing ever sets type = "docker", so the Docker builder had no coverage. sccache_server_cfg now takes a BuilderType, there’s a new add_docker_server() that mounts the daemon socket, however the harness image (Dockerfile.sccache-dist) will need the Docker client. Happy to push it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sccache-dist: Docker builder hangs indefinitely — docker cp - is never sent EOF

3 participants