audit: blocking operations with no timeout in CI scripts + workflows (sub-issue of #802)

[zackees]

Intro

Sub-issue of #802 — CI infrastructure audit slice.

Recently the Linux ARM64 (musl) cross-toolchain fetch from musl.cc hung for 135+ seconds in a release-matrix lane before failing. That single uncaught network fetch is the textbook prototype for what this audit found across the CI surface: nearly every workflow runs with GitHub Actions' default 6-hour per-job timeout, and the network-fetching steps inside them (apt, pip, cargo, curl, rustup, soldr-prepare, zigbuild's Zig + sysroot fetches) almost never set their own caps. A single stuck upstream — PyPI mirror, ESP-IDF/musl.cc artifact CDN, NuGet feed, Apple SDK source — can burn the full 6 h before GHA reaps the job.

The fix style is asymmetric:

• Workflows — add timeout-minutes: at the job level (single line) and on individual network-heavy steps (5–20 min cap). For curl/wget add --max-time + --connect-timeout. ~95% of the wins live here.
• Python scripts — add timeout=N kwarg to subprocess.run calls. Lower-impact since most ci/ Python is hook code that only runs locally on dev machines, but the ones that do run in CI (build_dylint_driver.py via the dylint workflow) are easy wins.

Findings — GitHub Workflows (CRITICAL surface)

The single biggest gap: 86 of 89 workflow files have no timeout-minutes: anywhere. Only bench-205.yml (30 min), acceptance-205.yml (45 min), and hw-ci.yml (5 min on one step) cap anything. Every other job, including the autonomous-release matrix, inherits GHA's 360-minute default.

Severity	File:job/step	Operation	Suggested fix
CRITICAL	release-auto.yml (entire workflow — prepare, build, publish, build-pypi, publish-pypi jobs)	No timeout-minutes: on any job. Daily run on every main push, plus manual dispatch.	Add timeout-minutes: 60 (or 90) at each job level.
CRITICAL	release-auto.yml line 86 — prepare step	curl -fsSL https://pypi.org/pypi/fbuild/json — no --max-time	Add --max-time 30 --connect-timeout 10.
CRITICAL	release-auto.yml line 403 — Verify all wheels visible on PyPI step	curl -fsSL https://pypi.org/pypi/fbuild/${PYPI_VERSION}/json in a 300s deadline loop — inner curl has no --max-time, so one stuck request can blow the whole budget on a single iteration	Add --max-time 10 so the outer deadline loop actually polls.
CRITICAL	release-auto.yml lines 354–355 — Smoke test Linux x86_64 wheel step	python -m pip install --upgrade pip + python -m pip install dist/wheels/*.whl — no timeout, no PyPI mirror fallback	Wrap in timeout 300 ... or add --timeout 60 to pip + step-level timeout-minutes: 5.
CRITICAL	template_native_build.yml line 167 — Install Linux dependencies step	sudo apt-get update && sudo apt-get install -y musl-tools pkg-config — used by every non-Windows release matrix lane	Add step-level timeout-minutes: 5 and apt-get -o Acquire::http::Timeout=30 -o Acquire::Retries=3.
CRITICAL	template_native_build.yml line 185 — Install xwin system dependencies	sudo apt-get update && sudo apt-get install -y --no-install-recommends clang lld llvm zip	Step-level timeout-minutes: 10.
CRITICAL	template_native_build.yml line 210 — Install cross-compilation tools step	pip install cargo-zigbuild — Python install that also downloads the Zig toolchain on first use. This is the closest analog to the musl.cc hang.	Step-level timeout-minutes: 10; pass --timeout 60 to pip.
CRITICAL	template_native_build.yml line 223 — Fetch Windows Python lib for PyO3 cross-compile step	curl -fsSL -o python.zip "https://www.nuget.org/api/v2/package/${PKG}/${PYTHON_VERSION}" — NuGet feed, no --max-time, no retries	Add --max-time 120 --connect-timeout 15 --retry 3 --retry-delay 5 + step-level timeout-minutes: 5.
CRITICAL	template_native_build.yml line 242 — Prepare Apple SDK (Linux → mac cross) step	soldr prepare --target ${{ inputs.target }} — downloads MacOSX.sdk.tar.xz from a remote mirror, no enforced cap	Step-level timeout-minutes: 15.
CRITICAL	template_native_build.yml lines 244–277, 291–338 — Build release binaries + Build Python extension steps	cargo zigbuild/soldr cargo build — Zig fetches sysroots, ring + cc-rs fetch crates; on the linux_cross aarch64-unknown-linux-musl lane this is where the recent musl.cc hang lived	Step-level timeout-minutes: 45 on the build, timeout-minutes: 30 on the python extension step.
HIGH	template_build.yml (88 callers — every build-<board>.yml) entire job	No timeout-minutes, default 6h. Includes apt-get install libudev-dev pkg-config, pip install esptool, soldr cargo build, fbuild build.	Job-level timeout-minutes: 30.
HIGH	template_build.yml lines 92–93 — Install Linux dependencies step	sudo apt-get update && sudo apt-get install -y libudev-dev pkg-config	Step-level timeout-minutes: 5.
HIGH	template_build.yml line 97 — Install esptool step	pip install esptool — only runs on ESP32 boards, but no timeout	Step-level timeout-minutes: 3; --timeout 60 to pip.
HIGH	template_build.yml lines 74–88, 133–143 — actions/cache/restore + actions/cache/save	No timeout-minutes on cache restore/save steps. The save can stall on GH cache backend.	Step-level timeout-minutes: 10 on each.
HIGH	check-ubuntu.yml, check-macos.yml, check-windows.yml — check job	No timeout-minutes. Runs on every PR.	Job-level timeout-minutes: 30.
HIGH	dylint.yml entire job	No timeout-minutes. Runs rustup toolchain install nightly-2026-03-26 --component llvm-tools-preview --component rust-src --component rustc-dev, cargo install cargo-dylint dylint-link --version 5.0.0 --locked, cargo dylint in a 3-attempt retry loop. All network-bound. Runs on every PR.	Job-level timeout-minutes: 30.
HIGH	dylint.yml lines 44, 46, 52 — install steps	rustup toolchain install ..., cargo install cargo-dylint dylint-link ..., uv run python ci/build_dylint_driver.py (which does git clone https://github.com/trailofbits/dylint + cargo build)	Step-level timeout-minutes: 10 on each install step.
HIGH	validate-boards.yml lines 28, 31–37 — Install PlatformIO, Install PlatformIO platforms	pip install platformio==..., 7x pio pkg install -g -p ... — each a full platform fetch from PlatformIO's package registry, no caps	Job-level timeout-minutes: 30. Step-level timeout-minutes: 15 on the multi-platform install.
HIGH	msrv.yml job	No timeout-minutes. PR-blocking.	Job-level timeout-minutes: 20.
HIGH	fmt.yml, docs.yml, crate-gate.yml, lint-subprocess.yml, loc-gate.yml jobs	No timeout-minutes. PR-blocking. Mostly short, but the soldr setup + cargo doc network paths can hang.	Job-level timeout-minutes: 15.
HIGH	template_native_build.yml line 92 — Setup soldr step	uses: zackees/setup-soldr@v0.9.63 with: cache: true, build-cache: true, target-cache: true — downloads soldr binary + restores cache from GitHub Actions cache backend. No timeout. Cache restore is a known stall point.	Step-level timeout-minutes: 10.
MEDIUM	update-data.yml job	No timeout-minutes. Daily scheduled. Inner curls already have --max-time 90 (good!) but git fetch, pio pkg install-equivalents via the dump_pio step, soldr setup, and cargo build --release --example dump_usb_ids have no cap.	Job-level timeout-minutes: 30.
MEDIUM	update-data.yml line 97 — Fetch + worktree the online-data branch step	git fetch origin "${ONLINE_BRANCH}:${ONLINE_BRANCH}" — can hang on stuck remote	Step-level timeout-minutes: 5 + set GIT_HTTP_LOW_SPEED_TIME=30 GIT_HTTP_LOW_SPEED_LIMIT=1000 in env.
MEDIUM	acceptance-205.yml job	timeout-minutes: 45 set (good), but the cargo test pulls Teensyduino + arm-gcc + STM32duino which are network-bound. 45 min is OK for the upper bound, but no inner step caps.	No-op or step-level cap on the cargo test step (e.g. 30 min).
MEDIUM	build.yml (workflow_dispatch)	No timeout-minutes. Calls template_native_build.yml which inherits.	Job-level timeout-minutes: 60.
LOW	add-to-project.yml job	No timeout-minutes. Issue/PR-triggered, GitHub-API-only.	Job-level timeout-minutes: 5.
LOW	hw-ci.yml bringup job (overall)	Only the Run bring-up tests step has timeout-minutes: 5. Other steps (cargo run, gh issue create) can hang. Self-hosted only, so blast radius is the runner host, not GHA minutes.	Job-level timeout-minutes: 30.

Findings — Python scripts in ci/

Lower priority overall — most are dev-machine scripts, not run in CI. The exceptions are build_dylint_driver.py (run inside the dylint workflow on every PR), lint.py/test.py/hooks/*.py (local hook surface, but still good hygiene).

Severity	File:line	Operation	Suggested fix
HIGH	ci/build_dylint_driver.py:34	subprocess.check_output(["rustup", "run", TOOLCHAIN_CHANNEL, "rustc", "-vV"]) — no timeout. Runs in CI dylint.yml.	Add timeout=60.
HIGH	ci/build_dylint_driver.py:45	subprocess.check_output(["rustup", "which", "--toolchain", full_toolchain, "rustc"]) — no timeout.	Add timeout=30.
HIGH	ci/build_dylint_driver.py:26 (called from lines 122, 123, 159)	run(args, check=True, ...) — wraps git clone --filter=blob:none https://github.com/trailofbits/dylint, git checkout, rustup run ... cargo build. None get a timeout.	Add timeout=600 default in the run() helper (overrideable per call).
MEDIUM	ci/hooks/check-on-stop.py:54	subprocess.run(cmd, capture_output=True, ...) — runs soldr cargo clippy --workspace --all-targets -- -D warnings and soldr cargo test --workspace. No timeout. Local-only (Stop hook) but a cold workspace test can wedge.	Add timeout=1800 (30 min).
MEDIUM	ci/hooks/lint.py:66	subprocess.run([sys.executable, lint_script, file_path]) — runs the project's ./lint wrapper which calls soldr cargo clippy. No timeout.	Add timeout=600 (10 min).
MEDIUM	ci/hooks/code-review-on-stop.py:29	subprocess.run(["git", ...]) — git status / diff. Local-only, but stuck filesystem can wedge git on Windows.	Add timeout=60.
MEDIUM	ci/hooks/check-on-start.py:27	subprocess.run(["git", "status", "--porcelain"]) — same as above.	Add timeout=60.
LOW	ci/lint.py:23	subprocess.run(cmd, ...) — runs soldr cargo clippy / soldr rustfmt. Dev-only uv run lint entry point.	Add timeout=1800.
LOW	ci/test.py:47	subprocess.run(cmd, ...) — runs soldr cargo test. Dev-only uv run test entry point.	Add timeout=3600 (1h).
LOW	ci/bench_uv_run.py:44, 67, 73	subprocess.run(cmd, ...) — runs uv run / uv sync / uv sync --reinstall-package fbuild. Bench tool, dev-only.	Add timeout=600.
LOW	ci/build_dist.py:54	run(cmd, check=True) helper used for gh workflow run / gh run view / gh run download. Dev-only script. (wait_for_run correctly uses its own deadline loop with time.sleep(15).)	Add timeout=300 default in the run() helper.
LOW	ci/trampoline.py:48, 63	subprocess.run(cmd) / subprocess.run([sys.executable, str(script), ...]) — uv run fbuild / uv run publish shims. Dev-only entry points; blocking is the expected behavior.	Optional: leave as-is, or add timeout=86400 purely as a watchdog.

What was searched

Files covered

• .github/workflows/*.yml — all 89 workflow files (88 build-*.yml calling template_build.yml, plus release-auto.yml, template_native_build.yml, template_build.yml, check-{ubuntu,macos,windows}.yml, bench-205.yml, acceptance-205.yml, update-data.yml, validate-boards.yml, dylint.yml, crate-gate.yml, loc-gate.yml, lint-subprocess.yml, fmt.yml, docs.yml, msrv.yml, hw-ci.yml, add-to-project.yml, build.yml).
• ci/**/*.py — top-level + ci/hooks/, plus the build_dist.py, build_dylint_driver.py, check_flash_offsets.py, enrich_extra_flags.py, extract_pio_build_flags.py, board_sources.py, validate_boards.py, bench_uv_run.py, publish.py, trampoline.py, lint.py, test.py, find_direct_subprocess.py, check_workspace_crates.py, measure_baseline_205.py.
• ci/hooks/*.py — check-on-start.py, check-on-stop.py, code-review-on-stop.py, lint.py, board_context.py, crate_guard.py, forbidden_commands.py, tool_guard.py, readme_guard.py, _output.py.

Patterns searched

• subprocess.(run|Popen|check_output|check_call|call) cross-referenced with timeout= (Python).
• urllib, requests., socket. (Python network).
• time.sleep + while True: / while : / deadline (busy-wait detection).
• curl, wget, git clone, git fetch, pip install, cargo install, npm install, rustup ... install (workflow shell).
• timeout-minutes: to inventory existing caps (only 3 workflows had any: bench-205, acceptance-205, hw-ci).
• actions/cache / actions/upload-artifact / actions/download-artifact (network-bound action steps).

Out-of-scope notes

• Tests inside crates (crates/*/tests/) — out of scope for this sub-issue; covered by sibling sub-issue(s) of audit: blocking operations with no timeout (meta) #802 if any auditor was assigned the Rust test surface.
• soldr / setup-soldr internals — zackees/setup-soldr is an external action. Its internal timeouts (cache restore, soldr binary fetch) are not in this repo's surface. Mitigation here is the step-level timeout-minutes: cap on the Setup soldr step itself.
• fbuild runtime subprocess spawns — covered by the existing find_direct_subprocess.py / ban_raw_subprocess dylint gate (Migrate all subprocess spawns to running-process; eliminate direct std::process / tokio::process usage #141) which is about routing through fbuild-core::subprocess. Timeout semantics on those wrappers are a separate concern from this audit.
• extract_pio_build_flags.py:217, measure_baseline_205.py, board_sources.py, check_flash_offsets.py, enrich_extra_flags.py — already have explicit timeouts on all network/subprocess calls. No findings here.
• update-data.yml inline curls — already use --max-time 90 --retry 5 --retry-delay 10 (good!). Only finding is the missing job-level cap and git fetch step.
• publish.py — pure stdlib wheel-assembly library, no subprocess/network. No findings.

Prioritized fix order (one-line summary)

1. Add timeout-minutes: to every job in release-auto.yml and template_native_build.yml (the daily-run release matrix — biggest blast radius).
2. Add --max-time + --connect-timeout to every curl in release-auto.yml (line 86, 403) and template_native_build.yml (line 223).
3. Wrap pip install, apt-get install, rustup toolchain install, cargo install, soldr prepare with timeout-minutes: step caps (5–15 min depending on payload).
4. Add timeout-minutes: 30 to template_build.yml (88 PR-triggered callers).
5. Add timeout-minutes: 15–30 to check-{ubuntu,macos,windows}.yml, dylint.yml, validate-boards.yml, msrv.yml, fmt.yml, docs.yml, crate-gate.yml, lint-subprocess.yml, loc-gate.yml.
6. Add timeout= to the four subprocess calls in ci/build_dylint_driver.py (only ci/ script that runs in CI without timeouts).
7. (Optional) Add timeout= to the Stop/PostToolUse hooks for dev-machine hygiene.

[zackees]

Closed by PR #843 (workspace-wide timeout sweep).