audit: go fully async — whole-app tokio runtime sharing for tokio-console (meta)

[zackees]

Meta tracking issue. Sub-agents will post per-crate audit reports
as sub-issues. Goal: convert every blocking sync code path in
fbuild that CAN be async to async, so the whole app runs on one
tokio runtime — the same one the embedded `ZccacheService`
(#789 Phase 4 stage 2 / #800) and the axum HTTP
server use.

Motivation

Tokio-console attaches to a single runtime and surfaces every task
spawned on it. Today only:

• axum HTTP/WS handlers
• The embedded `ZccacheService` background tasks (cache writer,
  audit writer, in-flight counter)

…run on the daemon's tokio runtime. Everything else — rayon
workers, `std::thread::spawn` background work, every
`std::process::Command::output()`, every `std::fs::read_to_string` on
a hot path — is invisible to tokio-console AND can't be cancelled
via the daemon's `tokio::sync::watch` shutdown signal.

After this migration:

• one runtime
• one tokio-console attach pane shows everything
• a single ctrl-C collapses every in-flight operation in the
  daemon (compiles, fingerprints, downloads, serial reads, deploy
  child processes — everything)
• tokio's structured cancellation propagates naturally, so per-
  request deadlines (the partner audit at audit: blocking operations with no timeout (meta) #802)
  compose with the existing shutdown notify

What to look for (sub-agents)

• `std::thread::spawn(...)` → `tokio::spawn(...)` (if the body can
  be made async) or `tokio::task::spawn_blocking(...)` (if it's
  CPU-bound). The current state mixes both without a clear contract.
• `rayon::scope` / `rayon::par_iter` / `ThreadPoolBuilder` →
  `futures::stream::FuturesUnordered` + `tokio::task::spawn`,
  or keep rayon for genuinely CPU-bound parallelism but ensure it
  doesn't hold tokio workers (`spawn_blocking` wrapper).
• `std::process::Command::output()` / `.wait()` →
  `tokio::process::Command::output().await`. Every subprocess
  spawn in fbuild's hot paths (compiler, linker, esptool,
  avrdude, picotool, pyocd) is a candidate.
• `std::sync::Mutex` / `std::sync::RwLock` held across `.await` →
  switch to `tokio::sync::Mutex` / `RwLock`. Held across nothing
  → leave alone.
• `std::sync::mpsc` cross-task channel → `tokio::sync::mpsc` (or
  `flume` if a sync `Receiver` is needed on the rayon side).
• `std::fs::read` / `std::fs::write` on hot paths → `tokio::fs::*`
  (compile output reads, fingerprint cache writes, daemon status
  file). One-shot config reads at startup are fine to keep sync.
• `reqwest::blocking::` → `reqwest::` (async client). This is
  the dominant pattern in `fbuild-packages` and is on the
  CRITICAL list in the partner audit audit: blocking operations with no timeout in fbuild-packages (sub-issue of #802) #805.
• `tokio::runtime::Handle::block_on(...)` calls from async
  contexts — these are pure anti-pattern; should be `.await`.
• `std::sync::OnceLock` for state that gets lazy-initialised from
  inside an async task → `tokio::sync::OnceCell` (the embedded
  `ZccacheService` handle in `compile_backend::GLOBAL` is an
  example to study).

What is NOT in scope

• CPU-bound work that legitimately needs a thread pool (hashing,
  compression, native cargo compilation work). Keep those on
  rayon / `spawn_blocking`, but document the contract.
• Anything in `crates/*/tests/` or `bench/` (those don't need to
  run on the prod runtime).
• `Drop` impls.
• Code that's only invoked from CLI binaries with their own
  short-lived runtime (`fbuild-cli`'s diagnostic subcommands).
• CI scripts and Python tooling.

Sub-issue checklist

Each sub-agent files one of these. The checklist auto-updates as
sub-issues land.

• `fbuild-build` (per-platform orchestrators, compile_many, build_fingerprint, pipeline) — biggest crate, highest risk
• `fbuild-daemon` (HTTP/WS handlers, broker, device manager — verify async completeness)
• `fbuild-serial` (SharedSerialManager, USB-CDC retry, deploy preemption — partner with audit: blocking operations with no timeout in fbuild-serial (sub-issue of #802) #803)
• `fbuild-deploy` (esptool / avrdude / picotool / pyocd invocation, ESP32 native espflash)
• `fbuild-packages` (downloads, install pipeline, library manager — partner with audit: blocking operations with no timeout in fbuild-packages (sub-issue of #802) #805)
• `fbuild-cli` + `fbuild-python` (HTTP client, PyO3 bindings — surface async boundaries to Python via `pyo3-async-runtimes`)
• `fbuild-core` + foundational crates (`subprocess.rs` is the choke point — converting it cascades into every caller)

Per-sub-issue report format

Each sub-issue should include:

• a brief intro saying which crate was audited
• a table of findings: `Severity` | `File:line` | `Current pattern` | `Proposed async replacement` | `Notes (cascades / blocking gotchas)`
• severity per finding (CRITICAL = blocks tokio worker on the daemon's hot path; HIGH = blocks user-visible build; MEDIUM = blocks setup/discovery; LOW = invisible improvement)
• a quick "what was searched / how" so a follow-up auditor can re-check

This issue auto-closes once every sub-issue is closed.

Relationship to #802

#802 audits "blocking operations with no timeout". This issue
audits "operations that could be async on the daemon runtime".
There's overlap: a blocking `reqwest::blocking::get` in #805 is
both untimely AND not on the runtime. The fixes compose — adding
a timeout AND making it `async` is the right end-state. Each
audit is filed separately so the diffs can be reviewed
independently.

[zackees]

8 sub-agents finished — 8 sub-issues filed covering every Rust crate plus the cross-cutting architecture concerns:

• audit: sync code that could be async in fbuild-serial (sub-issue of #813) #814 — fbuild-serial (sync serialport reads on a tokio worker + the serialport vs tokio_serial migration question)
• audit: sync code that could be async in fbuild-daemon (sub-issue of #813) #815 — fbuild-daemon (the runtime owner; spawn_blocking inventory, any sync I/O leaking into axum handlers)
• audit: sync code that could be async in fbuild-core + foundational crates (sub-issue of #813) #816 — fbuild-core + foundational crates (subprocess.rs::run_command is the choke point — the highest-leverage single conversion)
• audit: sync code that could be async in fbuild-cli + fbuild-python (sub-issue of #813) #817 — fbuild-cli + fbuild-python (PyO3-async-runtimes adoption — let FastLED await fbuild.SerialMonitor.read_lines())
• audit: cross-cutting async/sync architecture (sub-issue of #813) #818 — cross-cutting architecture (the async-trait vs native-async-fn decision, trait migration order, spawn_blocking / block_on inventory)
• audit: sync code that could be async in fbuild-deploy (sub-issue of #813) #819 — fbuild-deploy (esptool / avrdude / picotool / pyocd subprocess spawns, Deployer trait async question)
• audit: sync code that could be async in fbuild-build (sub-issue of #813) #820 — fbuild-build (per-platform orchestrators, compile_many, pipeline; BuildOrchestrator trait async question — the biggest cascade)
• audit: sync code that could be async in fbuild-packages (sub-issue of #813) #821 — fbuild-packages (3 CRITICAL + structural block_on sandwich at the Package trait boundary; shared async reqwest::Client proposal complements audit: blocking operations with no timeout in fbuild-packages (sub-issue of #802) #805)

This issue auto-closes once all 8 sub-issues close. Per-finding triage now belongs to each sub-issue's owner.

Notable cross-references:

• audit: sync code that could be async in fbuild-packages (sub-issue of #813) #821 explicitly notes its findings COMPOSE with audit: blocking operations with no timeout in fbuild-packages (sub-issue of #802) #805 (timeout audit) — both fixes go on the same Client::builder() chain.
• audit: sync code that could be async in fbuild-serial (sub-issue of #813) #814 explicitly notes its findings COMPOSE with audit: blocking operations with no timeout in fbuild-serial (sub-issue of #802) #803 (timeout audit on the same crate).
• audit: sync code that could be async in fbuild-deploy (sub-issue of #813) #819 explicitly notes its findings COMPOSE with audit: blocking operations with no timeout in fbuild-deploy (sub-issue of #802) #804.
• audit: cross-cutting async/sync architecture (sub-issue of #813) #818 is the architectural sequencing doc — read it BEFORE picking up any of the per-crate sub-issues to understand the right migration order.