[bug] A 4-second retry budget kills long runs — Heartfull died at iteration 2 of 50 on a runner blip the runner itself recovers from

[serge-ivo]

What happened

Heartfull Coder (f8ddc272), 2026-08-06. Objective: work through all 30 open issues in
HeartFull-online/platform, P1 first, maxIterations: 50.

02:34:29  run f3991cac starts
02:34:36  Loop → engine (step 1)   ← runner connected, engine driven
02:35:25  Loop → engine (step 2)   ← still fine
02:39:08  Loop stopped (failed) — "run error: No runner connected — run `pags up`"

279 seconds, iteration 2 of 50. The runner was demonstrably attached — two steps reached the
engine — and then dropped. A four-and-a-half-minute setup for a multi-hour job, gone.

Note the run did not stall or wait: a missing runner is mapped straight to a terminal outcome
(workflows/coding-session.ts:300):

const noRunner: CodingResult = { outcome: "failed", detail: "No coding runner connected. Start it with: pags up", steps: 0 };

The retry budget is shorter than the runner's own reconnect

coding-session.ts:311:

const retry = { retries: { limit: 2, delay: "2 seconds", backoff: "constant" }, timeout: "3 minutes" };

Two retries, two seconds apart — a ~4 second tolerance for the runner being absent.

Meanwhile the runner's relay client reconnects with exponential backoff capped at 30 seconds
(packages/cli/src/commands/runner/relay.ts). So the workflow gives up while the runner is still
in its own retry loop, for a disconnect the runner is about to recover from by itself. A laptop
lid, a wifi handover, or a pags up restart all exceed 4 seconds comfortably.

The platform already knows how to do this

• isTransientInfraError exists and is applied to DO resets — "a durable Workflow should
  retry/resume through it (that's the point of Workflows surviving deploys)". A runner disconnect
  is the same category and is not covered by it.
• The delivery outbox ([bug] Delivery outcome writes are unguarded — a slow attempt can resurrect a delivered event, bury a dead one, and stall the attempt counter #239) retries at 1m · 5m · 15m · 1h · 3h, with the reasoning written
  down: "Wide on purpose: these retries wait out someone else's outage, which resolves in
  minutes-to-hours." The coding loop waits four seconds for the same class of event.

And the message names the wrong remedy

"run pags up" — but pags up was running; it dropped and would have come back. Same pattern as
#271 (a 409 appending "and run pags up" while the runner was up) and #321 ("no github access" for
an unknown owner). An error that prescribes a fix the user has already applied trains them to
distrust the message.

Proposal

Treat a runner disconnect as a pause, not a failure:

• Wait for the relay to return, bounded — 10–15 minutes is the precedent already used for a human
  handoff (STALE_DRIVER_MS is 15 min) and comfortably outlasts the 30s reconnect cap.
• Report the pause in the thread ("runner went away, waiting for it to come back") so the state is
  visible rather than looking hung — the [bug] Navigating back loses 'working on it' — the run is still going and the API already reports it #252 lesson.
• Resume from the same iteration when it reconnects. The session survives now ([bug] Delegation is single-use — the Pilot ends the session its own driver requires, and the error blames the runner #271), so the engine
  is usually still there.
• Only after the window expires, fail — and say "the runner did not come back", not "run
  pags up".

This matters most for exactly the runs where it hurts: maxIterations: 50 over 30 issues is hours
of work, and any single blip in that window currently ends it.

Verification

• A run survives a 30-second runner disconnect and resumes at the same iteration.
• The thread shows the pause and the recovery.
• A runner that never returns fails after the window with an accurate message.
• A run that fails for a real reason still fails immediately — this must not slow down genuine errors.

[serge-ivo]

Shipped in b347f68.

Two mistakes, and raising a timeout fixes neither:

1. The wait was shorter than the recovery it waited for. A step.do budget of 2 × 2s gave the runner ~4 seconds to be absent. The runner's relay client reconnects with exponential backoff capped at 30 seconds, so the workflow gave up while the runner was still inside the loop it is designed to win. Any budget under ~30s loses that race by construction.
2. Losing the runner ENDED the run. A run with 48 iterations left has time to wait — the delivery outbox ([bug] Delivery outcome writes are unguarded — a slow attempt can resurrect a delivered event, bury a dead one, and stall the attempt counter #239) already waits 1m/5m/15m/1h/3h for exactly this class. So the fix is a different terminal condition: a disconnect is a PAUSE, and only a runner that never returns fails.

Follows 99bf6d8 (#339/#321): the failure carries retryable as a field, raised by callRunner when the RelayDO reports no socket or a mid-command drop — protocol facts, not a guess inferred from message wording.

Salvaged and verified after the authoring agent hit a usage limit mid-run.