meta: fbuild build-performance burndown — round 2 (residuals from #942)

[zackees]

Context

Rollup of the sub-issues that remained open when the Phase-1 profiling meta #942 was closed. #942 delivered the harness + the major wins (no-op rebuild 108 s → 2.1 s, cold toolchain∥framework overlap, fw-libs workspace-relative, plus correctness/hygiene fixes — 15 PRs). These residuals are genuinely cross-repo, CI-infra, or P3, so they're tracked here rather than blocking the (now-closed) #942.

The same reproducible harness applies: ci/docker-profile/ (#950), NightDriverStrip demo, cold vs hot, fbuild cache not persisted between runs. Compile settings stay frozen — value comes from caching, concurrency, and pipeline overlap only.

Burndown

High value — the real warm-cache win (cross-repo)

• [~] perf(zccache): firmware compiles don't hit zccache cross-project even when args are workspace-relative #966 — sketch+core cross-project caching FIXED (PR fix(build): ESP32 sketch+core compiles hit zccache cross-project (#966) #985: object filename + library-order determinism); fw-libs remainder split to perf(build): fw-lib (framework library) object compiles still miss zccache cross-project #986.
• perf(build): fw-lib (framework library) object compiles still miss zccache cross-project #986 — firmware compiles don't hit zccache cross-project even with workspace-relative args. Root cause: the embedded ZccacheService derives its key_root from the git root / cwd, not the per-compile request. Fix: add a per-request worktree_root to ZccacheService::compile in the zackees/zccache crate, then have fbuild pass each build's workspace through it (both the in-process service path and the zccache wrap fw-libs path). Unlocks sub-second warm sketch/core/fw-libs. Supersedes the remainder of perf(build): framework libs (~150s) recompiled per project; core cache never hydrates #952.

CI-infra — makes the guardrails actually enforce

• ci(dylint): Dylint job is red on main — alias-loop workaround fails, so NO custom lints are enforced #964 — Dylint job red on main. cargo-dylint 5.0.0 builds lib<name>.so then looks for lib<name>@<toolchain>.so; the post-hoc alias loop can't converge because cargo-dylint's check runs at build time. Real fix: wire dylint-link as the link-time linker for the lint-crate build (so the @<toolchain> name is produced at link time), or upgrade the cargo-dylint/dylint_linting/driver triple together. Until fixed, ban_raw_subprocess + ban_raw_path_prefix_compare (feat(dylint): ban raw Path::{starts_with, strip_prefix} so the #952 cache-key mistake can't recur #963) are authored but not enforced.

Done

• fix(build): rebuild_signature omits build_unflags stripping → persistent recompiles for build_unflags projects #970 — build_unflags signature asymmetry centralized (PR fix(build): centralize build_unflags stripping in build_rebuild_signature (#970) #975).

Correctness — rebuild-signature family (continues #951/#956/#957)

• fix(build): rebuild_signature omits build_unflags stripping → persistent recompiles for build_unflags projects #970 — Esp32Compiler::rebuild_signature (and the default/ch32v/renesas overrides) hash raw flags, but the write side strips build_unflags via apply_compile_unflags before hashing → persistent recompiles for any project that sets build_unflags (latent for NightDriverStrip, which sets none). Fix: apply the same unflag stripping inside the signature computation (prefer centralizing). Needs a build_unflags fixture to validate.

Cold-pipeline overlap — further (continues #953/#967)

• perf(packages): overlap download/extract/install in cold pioarduino-resolve (~185-237s serialized) #953 — beyond the toolchain∥framework overlap already merged (perf(esp32): download GCC toolchain concurrently with framework + SDK libs (#953) #967/fix(esp32): surface both errors if toolchain AND framework resolve fail (#967 follow-up) #969): kick off user lib_deps downloads in the background to overlap the toolchain/SDK fetch, and start the core/fw-libs compile as soon as the toolchain + framework core-sources exist while the SDK libs finish extracting.

P3

• feat(deploy): manage esptool as a provisioned package instead of requiring pip install #954 — provision esptool as a managed package instead of the harness workaround.
• build(esp32): NightDriverStrip partitions CSV rejected by esp32-core-3.3.9 gen_esp32part.py (nvs 0x2000 < 0x3000) #960 — NightDriverStrip partitions CSV rejected by gen_esp32part.py (framework-parity follow-up to fix(build): project-relative board_build.partitions CSV not found (resolved against framework tools dir) #955).

Acceptance criteria

• perf(zccache): firmware compiles don't hit zccache cross-project even when args are workspace-relative #966 lands (cross-repo zccache change + fbuild follow-up) and a clean cold→warm harness run shows sketch/core/fw-libs hitting zccache cross-project.
• ci(dylint): Dylint job is red on main — alias-loop workaround fails, so NO custom lints are enforced #964 green — Dylint enforces on main.
• fix(build): rebuild_signature omits build_unflags stripping → persistent recompiles for build_unflags projects #970 fixed with a build_unflags regression test.
• perf(packages): overlap download/extract/install in cold pioarduino-resolve (~185-237s serialized) #953 further-overlap PR(s) merged with before/after harness numbers.
• feat(deploy): manage esptool as a provisioned package instead of requiring pip install #954, build(esp32): NightDriverStrip partitions CSV rejected by esp32-core-3.3.9 gen_esp32part.py (nvs 0x2000 < 0x3000) #960 resolved.
• All linked sub-issues closed; this issue closed.

Decisions

• Priority: P2 — DX/perf, nothing shipping is blocked.
• Compile settings frozen — same as meta: profiling-driven build performance burndown (Docker Linux, NightDriverStrip, cold vs hot cache) #942.
• Follow-up to meta: profiling-driven build performance burndown (Docker Linux, NightDriverStrip, cold vs hot cache) #942 (closed with the Phase-4 report).

[zackees]

#966 FULLY RESOLVED — all ESP32 firmware compile phases now hit zccache cross-project:

• fix(build): ESP32 sketch+core compiles hit zccache cross-project (#966) #985 — sketch+core (object-filename + library-order determinism): 94.5 s → 0.8 s, 12.9 s → 0.5 s.
• fix(build): route framework-lib compiles through embedded zccache — fw-libs cross-project hit (#986) #987 (perf(build): fw-lib (framework library) object compiles still miss zccache cross-project #986) — fw-libs routed through the embedded zccache service (was bypassing it entirely, direct gcc): 152 s → 2.2 s.

No zccache-side change was needed; every leak was fbuild-side. Also filed soldr#1387 (clippy-driver bypasses zccache).

Remaining in #974: #964 (dylint-link CI), #953 (cold-pipeline overlap), #954/#960 (P3).

[zackees]

#954 DONE (PR #988): esptool provisioned as a managed standalone binary (tasmota PyInstaller build, deps bundled). Clean-machine ESP32 builds now produce firmware.bin with no user pip install esptool — validated with esptool removed from PATH.

#964 in progress: root-caused — only 1 of 25 dylint crates (ban_std_pathbuf) had .cargo/config.toml wiring linker=dylint-link; the other 24 emit bare <name>.so so cargo-dylint can't find <name>@<toolchain>.so and the alias-loop workaround never converges. Fix (adding the config to all 24) under local validation.

#974 status: #970 ✓, #966 ✓ (incl #986 ✓), #954 ✓ — remaining: #964 (validating), #953, #960.

[zackees]

#960 resolved (investigation): fbuild intentionally uses pioarduino platform-espressif32@stable (arduino-esp32 3.3.9) and does not honor the espressif32@^6.12.0 pin — a deliberate design choice, not a resolution bug. NightDriverStrip's nvs 0x2000 CSV is stale for 3.x's ≥0x3000 rule; fix is upstream/harness-local, out of fbuild-core scope. Closed.

#974 status: #970 ✓, #966 ✓ (incl #986 ✓), #954 ✓, #960 ✓. Remaining: #953 (cold-pipeline overlap — implementing+profiling), #964 (dylint-link config fix landed + all 25 lints now enforce; fixing the real violations enforcement surfaced).