Daemon doesn't cancel in-flight builds when CLI is force-killed (zombie build processes)

[zackees]

Problem

When the fbuild CLI is force-killed (Ctrl+C, SIGKILL from an AI agent, terminal close), the daemon continues running the in-flight build to completion. This leaves zombie compiler/linker subprocesses running, holds the per-project lock, and blocks subsequent builds for the same project.

Repro:

1. Start a long build via fbuild build ... or fbuild deploy ....
2. Kill the CLI process (taskkill /F /PID <cli> on Windows, kill -9 <cli> on Unix).
3. Observe: gcc / clang / linker subprocesses keep running under the daemon. A new CLI invocation for the same project blocks on the still-held project lock until the orphaned build finishes naturally.

Root cause (investigation)

There is no cancellation infrastructure anywhere on the build path.

1. CLI -> daemon: DaemonClient::build_streaming (crates/fbuild-cli/src/daemon_client.rs:281-370) POSTs to /api/build and consumes a long-lived application/x-ndjson stream. One HTTP request, no separate cancel channel, no job ID.

2. Daemon handler: handlers::operations::build (crates/fbuild-daemon/src/handlers/operations/build.rs:99-762) acquires ctx.project_lock, then tokio::spawns an inner build_task that runs the platform orchestrator (build.rs:386-389). The spawned JoinHandle is detached from the HTTP request lifetime — only the 60-minute hard deadline aborts it (build.rs:399).

3. Disconnect detection: The streaming body is built with axum::body::Body::from_stream(stream::unfold(async_rx, ...)) (build.rs:569-579). When hyper drops the body, async_rx is dropped, but the build worker's let _ = async_tx.send(...) calls silently swallow SendError and keep going. Nothing signals the orchestrator.

4. Subprocesses: All compiler/linker invocations go through fbuild_core::subprocess::run_command (crates/fbuild-core/src/subprocess.rs:151) which uses tokio::process::Command via tokio_spawn::spawn_contained. kill_on_drop(true) is never set. Containment (Windows Job Object / Linux PR_SET_PDEATHSIG) only kills children when the daemon itself dies — daemon stays alive, so children stay alive.

5. DaemonContext (crates/fbuild-daemon/src/context.rs:128-190) has no active_builds map, no CancellationToken storage, no /api/cancel route exists.

Workspace grep for CancellationToken, kill_on_drop, /api/cancel: zero hits on the build path.

Why CLI kill currently does NOT propagate

SIGKILL CLI -> TCP socket closes -> hyper drops body stream -> async_rx dropped -> next async_tx.send(...) in worker returns Err -> let _ = swallows it -> orchestrator keeps .awaiting wait_with_output() on every subprocess -> processes are contained-but-not-kill_on_drop -> they run to completion -> next build queues behind held project_lock.

Fix

Minimal correct fix, in priority order:

1. Add per-build tokio_util::sync::CancellationToken stored in DaemonContext as active_builds: DashMap<String, CancellationToken> keyed by request_id. Insert at the start of build() in build.rs; remove via Drop guard.

2. Detect client disconnect via body drop: Replace the stream::unfold body (build.rs:569-579) with a wrapper whose Drop impl calls token.cancel(). When hyper drops the response body, the wrapper drops, cancellation fires. Axum has no built-in disconnect callback; body Drop is the canonical signal.

3. tokio::select! the build task against the token so when cancellation fires we build_task.abort() immediately instead of waiting for the orchestrator's next await point.

4. Plumb the token into BuildParams (crates/fbuild-build/src/lib.rs:155-211) so parallel::compile_sources_parallel (crates/fbuild-build/src/parallel.rs:53) can tokio::select! cancellation against child.wait() inside subprocess::wait_and_capture (subprocess.rs:390).

5. Set kill_on_drop(true) on every tokio::process::Command constructed in subprocess::build_command (subprocess.rs:455-508). Combined with (4), this guarantees that when cancellation ends an in-flight .await, dropping the child handle terminates the OS process.

6. (Optional) /api/cancel/{request_id} route + CLI Ctrl+C handler enhancement (fbuild-cli/src/cli/dispatch.rs:50-54) so explicit Ctrl+C can fire a best-effort cancel POST before exiting, as a fallback for platforms where body-drop propagation is slow.

Steps 1-3 alone cover the AI-agent SIGKILL case (the orchestrator's JoinHandle gets abort()ed and all currently-awaited subprocesses get dropped — with (5) they get killed; without (5) they'd only get killed when the daemon dies). Steps 4-5 make subprocess termination immediate rather than depending on Drop.

Acceptance

• Kill the CLI mid-build. Within ~500ms, all child compiler processes for that build are gone (Get-Process gcc,clang,ld | Where-Object {...} empty on Windows; pgrep -f empty on Unix).
• The per-project lock is released so the next fbuild build for the same project starts immediately.
• Test: spawn fbuild build, kill CLI, immediately spawn a second fbuild build for same project; the second should start within 1 second.