perf(serial): per-line WebSocket messages drop lines under sustained TX bursts

[zackees]

Context

Filed from FastLED #2700 — fbuild needs to own ESP toolchain/package resolution end-to-end so ESP32-S3/ESP32-P4/etc. builds never require hardcoded local PlatformIO package paths or a PlatformIO fallback.

Failure modes observed

1. fbuild surface error is opaque. While building examples/AutoResearch/AutoResearch.ino with bash compile esp32s3 --examples AutoResearch (FastLED side), fbuild's streaming build path failed with:

   error: daemon error: stream error: error decoding response body
   

   The user-facing error gives no clue what actually failed (crates/fbuild-cli/src/daemon_client.rs:516). This needs to surface the underlying daemon-side error (the daemon must serialize a usable final event before the stream closes — even on a crash path).

2. PlatformIO-fallback path also stalls. A temporary PlatformIO fallback ran into pioarduino platform-espressif32@54.03.20's platform.json requiring toolchain-riscv32-esp@14.2.0+20241119 even for an ESP32-S3 build. PlatformIO then stalled at Downloading 0%. The machine already had a usable local RISC-V toolchain at <HOME>/.platformio/packages/toolchain-riscv32-esp with package metadata for 14.2.0+20251107 and a .piopm URI, but PlatformIO refused to reuse it. (FastLED policy is that fbuild should be the exclusive board-build path; this point just illustrates why the PlatformIO escape hatch isn't viable.)

3. fbuild currently only resolves the MCU-primary toolchain. crates/fbuild-build/src/esp32/orchestrator/packages.rs::resolve_pioarduino_packages calls get_toolchain_metadata_url(is_riscv) which returns exactly one of toolchain-riscv32-esp or toolchain-xtensa-esp-elf. The pioarduino platform.json lists multiple toolchain packages in its packages section (Xtensa + RISC-V for ULP helper, etc.); fbuild skips the secondary ones. If a real build ever needs the secondary toolchain, fbuild won't provision it and won't even diagnose its absence.

Acceptance criteria

Mirrors FastLED #2700:

• bash compile esp32s3 --examples AutoResearch (and equivalent direct fbuild invocations) succeeds through fbuild without --platformio, --no-filter, or generated platform_packages path overrides.
• bash autoresearch esp32s3 --lcd … builds/deploys through fbuild only.
• No generated config needs user-specific paths such as C:/Users/<user>/.platformio/packages/toolchain-riscv32-esp.
• fbuild reports a clear actionable error when a required ESP toolchain is missing or incompatible, instead of failing as stream error: error decoding response body.
• Regression coverage covers ESP32-S3 with a cached pioarduino/platform-espressif32 and the RISC-V helper toolchain requirement.

Proposed scope

• Better diagnostics on the streaming build path. Make build_streaming decode a final OperationResponse (or equivalent error envelope) even on the failure path, so users get an actionable message instead of stream error: error decoding response body. Daemon side: ensure a terminal event is always sent (and flushed) before the stream is closed.
• Audit pioarduino platform.json package consumption. Enumerate the packages section and provision every package referenced by the orchestrator's build (toolchains, frameworks, helper toolchains), not just the MCU-primary toolchain.
• Version-compatibility policy. Decide whether fbuild should:
  • (a) accept a newer cached toolchain (e.g. 14.2.0+20251107 when platform.json asks for 14.2.0+20241119), or
  • (b) always provision the exact manifest version into its own cache.
    Document and test the choice.
• Tests.
  • Unit test: feeding a real pioarduino platform.json (54.03.20) into Esp32Platform::get_package_url returns the RISC-V helper toolchain URL for ESP32-S3.
  • Integration: a tests/platform/esp32s3 build that uses platform = https://github.com/pioarduino/platform-espressif32/releases/download/54.03.20/platform-espressif32.zip (not the stock platform = espressif32) succeeds end-to-end.

Related

• FastLED #2700 (this issue's parent)
• FastLED #2507 (prior ESP32-P4 autoresearch fbuild failure)
• fbuild#378 (ESP32-C2 skeleton install — same orchestrator/packages area)
• fbuild#380 (route ESP32-S3 writes through esptool — landed in 2.2.17)