Harden SSE stall watchdog: guard peekByte + receiveHead, make stalls retryable (reconnect)

[justrach]

Summary

graff's streaming reader (postStream) could hang forever at "thinking..." when a connection went half-open mid-stream (the HttpConnectionClosing / WriteFailed / truncated-TLS case): the retry re-entered postStream, the spinner restarted, then the resumed read wedged - "reconnect doesn't reconnect."

Root cause (confirmed against the code)

The 2026-06-17 idle-stall watchdog (racing streamStallTask against the line read via Io.Select) guarded the SSE line read but left sibling blocking reads unguarded, and surfaced stalls as a non-retryable error:

1. reader.peekByte() (the post-line "is there more?" probe, ~main.zig:8638) ran after the watchdog was cancelled. On a half-open socket (server gone, no FIN/RST) it blocks forever. (Primary hang.)
2. Watchdog stalls returned error.Interrupted - which request()'s retry loop deliberately does NOT retry (Esc semantics) - while the non-streaming postWatched correctly returns retryable error.HungRequest. So even a detected stall ended the turn instead of reconnecting.

Done (release/0.0.16)

• Guarded peekByte with the same stall watchdog (new streamPeekTask select-arm).
• Watchdog deadline stalls now return error.HungRequest (retryable -> request() reconnects + redials a fresh connection); real Esc still maps to error.Interrupted.
• zig build + zig build test green.

Net: a dead/half-open stream now trips the watchdog (<= stream_stall_ms = 120s), prints [network error: HungRequest - retrying], and redials (up to 3 attempts) instead of hanging at "thinking".

Remaining - the broader watchdog work

1. receiveHead: guard the HTTP response-header read. Before the line/peek loop, postStream reads the response head. A server that accepts the TCP connection but never sends headers (or sends them then dies) hangs there, before the first byte - same "stuck at thinking", not yet covered. Race receiveHead (and/or the request send) against the stall/deadline watchdog so a pre-stream hang also becomes a retryable HungRequest.
2. Fail-safe the concurrency-starved fallbacks. The Io.Select arms fall back to bare blocking reads when the pool has no spare slot (8606 / 8615, and the new peek equivalent). Under subagent fan-out this can still wedge. Poison the connection + return HungRequest instead of blocking.
3. Consider a shorter stall threshold for the between-lines peek (bytes were flowing, then stopped) vs. the before-first-token case (a legit long reasoning pause).

Refs

• postStream read loop + the now-guarded peek: src/main.zig ~8597-8672
• watchdog infra: streamStallTask / WatchdogFired / stream_stall_ms; retryable mapping in postWatched
• retry loop: request() (error.Interrupted not retried; error.HungRequest retried)

[justrach]

✅ receiveHead guard landed (release/0.0.16). Wrapped req.receiveHead() in the same Io.Select stall watchdog as the line/peek reads (new streamHeadTask select-arm): a server that accepts the connection but never sends headers now trips the watchdog (≤ stream_stall_ms) → retryable error.HungRequest → request() redials a fresh connection, instead of blocking for minutes on the OS timeout before the first byte. zig build + zig build test green.

Remaining (bounded):

1. The send/connect path — the literal WriteFailed (writing to a half-open socket) still relies on the OS write timeout. A connect/write deadline would cap that too.
2. Fix-B: fail-safe the concurrency-starved fallbacks (bare blocking reads under pool exhaustion).

So mid-stream (peekByte) and pre-stream header (receiveHead) hangs are now both guarded; the send-side timeout is the last piece.

[justrach]

Related fix landed in 6ad68a6 — postStream now races receiveHead against the stall watchdog so a hung response-head wait is treated as a retryable stall instead of hanging forever.

Included in release/0.0.161 via the main/release/0.0.16 baseline.

[justrach]

Status: the two headline guards + reconnect are in main. receiveHead+send are raced against a 30s head-stall watchdog → retryable error.HungRequest that the request() loop redials (src/agent_stream.zig:259-298, src/http.zig:229-275, src/agent_request.zig:104-170), and the peekByte forever-hang is structurally gone (the blocking read is now the guarded streamLineTask/streamDelimiterEnding, which leaves the \n buffered except at real EOF). Keeping this open for the remaining Fix-B: the concurrency-starved fallback paths (src/agent_stream.zig:264-271 & 344-350, src/http.zig:303-304) are still bare blocking reads that 'accept the hang risk' under subagent pool exhaustion, plus the shorter between-lines peek threshold. That's a real correctness gap, not a nice-to-have, so it stays tracked here rather than being closed.

[yxlyx]

Improvement follow-up from multi-user simulations: please include the user-visible retry case in this issue’s transport recovery contract. Distinguish retryable SSE stall/drop failures from cancellation; define automatic versus explicit retry; ensure a retry creates a new attempt without duplicating accepted work; preserve the original failure cause and attempt relationship; and add stall/drop regression coverage. Broader reconnect/reattach semantics belong to #60.

[justrach]

The retry/reconnect part of this is now on main — PR #230 (merge `2a2fa7f`).

A mid-stream stall or drop (never a user Esc) now reconnects on a fresh stream and re-sends the full request (codex `stream_max_retries` style), bounded by a 2-retry budget; on exhaustion the turn ends as before, recorded as a stall/drop and never an Esc (the #134 invariant is preserved). This resolves the #229 recurrence — no more manual `continue`.

Verified: live recovery on Codex WS transport (`GRAFF_FORCE_STALL_ONCE` now reconnects + completes instead of ending early), plus the offline exhaustion path via new `GRAFF_FORCE_STALL_ALWAYS` / `GRAFF_FORCE_DROP_ALWAYS` seams + `test-stall-drop.py`.

Still open here: the guard peekByte + receiveHead watchdog-hardening item — separate from the retry logic — so leaving this issue open for that part.

[justrach]

Fixed in v0.0.226 (PR #328).

The last remaining item. A single 120s threshold governed every SSE line read; between-lines silence now trips sooner than a pre-first-token reasoning pause, which is what the issue asked for.

Tests went 449 -> 575 on this release; every Zig fix carries a pinning test that was falsified independently (revert the production change, confirm the test goes red naming itself, restore).