[bug] The dominant live cause of dead runs: the Engine's own subscription usage limit is handled as a 15-minute human takeover, so a run dies 41 minutes before the CLI comes back

[serge-ivo]

The population, first

I pulled GET /v1/instances/:id/loop for all 32 instances on the owner's account (2026-08-13).
80 autonomous runs, 30 failed (37.5%). Full class distribution:

n	class	live?
8	account token circuit breaker (250M/24h)	dead — BUDGET_ENFORCE off since d951b5f, 2026-08-10T08:16Z
5	WorkflowInternalError	live
4	stuck not resolved in time	live
3	$50 daily ceiling	dead — same commit
3	CF subrequest ceiling (#523)	live
2	provider stall mid-reply	live
2	no runner connected	live
1 each	relay 504, session not running, [object Object]	live

d951b5f ("demote AI circuit breakers to observe-only") landed 23 minutes after the last budget
death (last budget kill 2026-08-10T07:53:19Z; commit 2026-08-10T08:16:32Z). So 11 of the 30
historical failures — 37% — cannot recur. Filtering to the live era leaves 11 failures, and
stuck not resolved in time is the largest class at 4 (36%). All four are in the last 35 hours.
It is what is killing his runs now.

What it actually is: the Engine's own subscription limit, handled as a human takeover

Three of the four are the same thing, and it is not "the agent got stuck on a widget". Quoting the
handoff notifications the platform itself sent (/v1/notifications, verbatim):

2026-08-12 11:34:16Z — heartfull/platform: The Claude CLI session has hit its usage limit and
resets at 10:30pm Australia/Sydney time. I cannot proceed until the session limit resets.

2026-08-12 12:06:16Z — dev/aipa: The Claude CLI session has hit its usage limit and will reset
at 10:30pm Australia/Melbourne time.

2026-08-12 12:08:56Z — dev/aipa: The Claude CLI has hit its session limit and cannot process
any more requests until it resets at 10:30pm (Australia/Melbourne time).

(The fourth, 2026-08-12T22:41:08Z, is different and genuinely human-resolvable: Codex refusing with
"Not inside a trusted directory and --skip-git-repo-check was not specified".)

The platform knows the reset time. It is in the sentence it wrote. It does nothing with it.

The measurement that makes this concrete

Run 6550f673 (Heartfull), the clean case with no workflow retry muddying the clock:

event	UTC	local (AEST, UTC+10)
run starts	11:34:00	21:34
Pilot declares stuck, handoff opens	11:34:16 (+16s)	21:34
run declared failed	11:49:31 (+15m15s)	21:49
CLI usage limit resets	—	22:30

The run died 41 minutes before the resource it was explicitly waiting for came back, having
waited 15. Run ebc70ac1 is the same shape: handoff at +24s, death at +15m15s.

start → handoff was 16s, 23s, 24s and 24s across the four. These runs never did any work at all —
the very first decide saw a rate-limited CLI and escalated.

Mechanism — two individually-correct decisions

1. The Pilot's decision vocabulary has no way to say "wait".

workers/api/src/lib/coding-loop.ts:444-461, toDecision — the complete tool surface the brain can
return:

case "send_message":      return { action: { kind: "message", text: str(a.text) } };
case "finish":            return { finish: { status: ... } };
case "request_human":     return { stuck: { why: str(a.why) || "needs a human" } };
case "request_user_info": return { needsInput: { field: ..., why: ... } };

There is no wait_until / retry_after. CodingOutcome (coding-loop.ts:75) is
"done" | "stuck" | "needs_input" | "failed" | "max_steps" | "cancelled" — no paused/deferred state
either. Faced with "the CLI is rate-limited until 22:30", request_human is the only honest move
the brain has. It behaved correctly.

2. stuck is bound to a 15-minute human deadline.

workers/api/src/workflows/coding-session.ts:54:

const HANDOFF_WAIT_POLLS = 180; // 180 × 5s = 15 min

and coding-session.ts:643-658:

for (let poll = 0; poll < HANDOFF_WAIT_POLLS && !resolved; poll++) { … }
if (!resolved) {
    result = { outcome: "failed", detail: `${reason} not resolved in time`, … };
    break;
}

15 minutes is a sensible bound for "a human needs to click something". It is the wrong bound for "a
subscription window reopens in 56 minutes", and the two are indistinguishable once both are stuck.

Neither decision is wrong on its own. The composition kills the run.

The absence, grepped

$ grep -rniE "usage limit|session limit|limit (will )?reset|resets at" workers/api/src packages/browser-runner/src
(no matches)

The only rate-limit handling in the coding path is coding-failure.ts:155, and that classifies the
Pilot's own BYOK provider error, not the Engine CLI's subscription window. The platform has no
concept of the Engine having a limit — which is odd, because on a subscription auth mode
(lib/coding-engines.ts) that is the expected steady-state failure, not an exception.

What to do — cheapest first

(a) One-line-ish: don't kill a stuck run at 15 minutes; report it as parked.
When the wait expires, end with an outcome that says "waiting on you" rather than failed. He is
re-typing "Retry" on runs that were never broken. This alone removes the "Retry" loop even without
(b) or (c).

(b) Give the brain a wait_until verb. Add case "wait_until": return { waitUntil: { at, why } }
to toDecision, a CodingOutcome of waiting, and have the workflow step.sleep to that instant
(Workflows sleep durably — this is exactly what step.sleep is for; the handoff loop already uses
it at coding-session.ts:644). Cap it — 6 hours is more than a Claude 5-hour rolling window — and
resume the same objective at the same step. The reset time is already in the model's hands: it read
it off the pane to write the sentence quoted above.

(c) Detect it below the brain, so it costs no decision at all. The runner sees the CLI's stdout;
packages/browser-runner/src/coding/headless.ts could surface usageLimitResetsAt on the snapshot,
and the loop could park without spending a BYOK decision. More work; do it only if (b) proves the
brain's extraction unreliable.

(d) Separately: the stuck handoff never reaches the chat thread. grep -n "postToChat" coding-session.ts → lines 218, 330, 527, 532, 537. A runner disconnect is announced in chat
(announce: postToChat, line 330); a human handoff is not — the handoff block (620-660) only
calls notifyUser (line 627). Push fired for all four (verified: pushed_at set on every one), so
this is not "no surface" — but the thread the owner started the run from says nothing while the run
waits on him, then reports it failed.

Alternatives considered and rejected

• Just raise HANDOFF_WAIT_POLLS. Rejected: this is the fix [bug] A 4-second retry budget kills long runs — Heartfull died at iteration 2 of 50 on a runner blip the runner itself recovers from #341 explicitly argued against for
  the runner case — "a bigger number with the same terminal behaviour only moves the cliff"
  (coding-session.ts:318-321). A 5-hour rolling window can reset anywhere; and a genuine
  human-needed handoff should time out in minutes, not hours. Raising the number degrades the case
  it is right for to fix the case it is wrong for.
• Have the Pilot retry the instruction on a fixed backoff. Rejected: it burns BYOK decisions
  against maxSteps re-discovering the same wall, and The Pilot answers a CLI safety refusal by repeating the instruction — 8 escalating sends, 2 with a byte-identical command, then "Loop complete" on the subset the owner did not ask for #522 just landed machinery specifically to
  stop the Pilot repeating itself. A known reset instant deserves a sleep, not a poll.
• Pre-flight the engine's remaining quota before starting a run. Rejected as the primary fix: it
  does not help the run that exhausts the window mid-objective, which is the more expensive case.
  Reasonable as a later addition.

Acceptance criteria

1. A Pilot run whose Engine reports a usage limit with a reset time does not end failed within 15
   minutes. It either sleeps to the reset and continues, or ends in a non-failed "waiting" state.
2. A run parked on a genuine human handoff (the Codex trusted-directory case) still times out and is
   still distinguishable from (1) in check_instance_loop's stopReason.
3. The handoff reason appears in the instance chat thread, not only in the notification tray.
4. Replaying the three 2026-08-12 transcripts through runCodingLoop yields waiting, not stuck.

Regression risk

• A wait_until the brain sets wrongly parks a run for hours. Bound it (≤6h), require a reason
  string, make it cancellable (isCancelRequested is already read every capture,
  coding-session.ts:~390), and count the sleep against the run's wall-clock budget.
• sweepStaleRuns presumes a run dead after STALE_RUN_MS = 3h (lib/run-sweeper.ts:30). A
  sleeping Pilot must tick last_progress_at or the sweeper will close it — the handoff wait already
  has this hazard and handles it via tick (coding-session.ts:335-338); a wait_until needs the
  same. This is the one that will bite. Test: a run that sleeps 4h is still running afterwards.
• Changing the terminal outcome for expired handoffs touches closeDelegation's reason mapping
  (coding-session.ts:194-200) and shouldEndSessionAfterRun's error/ended choice
  (coding-session.ts:754) — a waiting outcome must not close the session as error.

Provenance

Runs and timings: /v1/instances/:id/loop + /v1/notifications, production, 2026-08-13 — all four
runs predate nothing, they are current. elapsedMs/phase for the two long ones came from
error_log rows written by ba67afc (#529), which only exists from 2026-08-12 ~10:00 UTC; the two
15m15s runs are timed from run rows alone and need no post-#529 data. Everything above is measured
except the claim that (b) would have saved these three runs, which is inferred from the reset times
the model itself reported.

[serge-ivo]

Shipped in c68b82a — CI green (incl. Playwright chromium + webkit), API Worker deployed, https://api.proagentstore.online/health 200.

What was built, and the design calls behind it

A verb, not a bigger number. wait_for_reset → CodingOutcome: "waiting" → the workflow parks durably (step.sleep) and resumes the same objective at the same step. Raising HANDOFF_WAIT_POLLS was rejected for the reason the issue gives and #341 already argued for the runner case: it would degrade the case 15 minutes is right for in order to fix the case it is wrong for.

The deadline is a hint, never an authority. This was the trap worth spending time on:

• 2026-08-12T22:30:00 is a valid ISO string that Date.parse resolves against the runtime zone — UTC in a Worker. Accepting it would turn "10:30pm Sydney" into a silent ten-hour error that looks like a working parse. So parseResetInstant requires an explicit offset or Z, and the model is handed the current instant and the owner's offset (accountTimeZone, resolved once per run, unset → rendered as UTC out loud) so it converts rather than guesses. The clock goes in the user message, not the system prompt — a run that parks for an hour must convert against the time it is now.
• No offset / unparseable → bounded backoff (15m · 30m · 60m), which is the answer the issue asks for when the deadline cannot be trusted.
• Reset already in the past → a one-minute beat, not a full backoff: the model's own reading says the window reopened, and that conversion is probably nearly right.
• Every branch clamped into [1min, 6h] and against the run's remaining budget, and the number of parks is bounded (3) independently of the parse. A wrong deadline costs one extra cycle, never an unbounded park.

What a park costs, stated rather than assumed. It holds the session's driver claim (STALE_DRIVER_MS, 15 min), the loop-run row the sweeper reads (STALE_RUN_MS, 3h) and the session itself (IDLE_SESSION_MS, 6h). So the park is a loop of 5-minute ticked chunks — a third of the tightest of the three — which also makes it cancellable (isCancelRequested is read on every tick). It also holds the repo, and that is accepted deliberately: ending-and-rescheduling needs a resumption scheduler that does not exist, and the run that ends loses its objective, its step log and the engine's warm context.

The stale-pane cycle. runCodingLoop restarts its step log each round, so a resumed brain would read the limit message the CLI printed once and park again. engineResumeNote tells it that text is history — attributed to the platform, never folded into userHint ("The user just told you"), which is what #505 stamps reports for.

Acceptance criteria

1. ✅ A usage-limit run parks and resumes instead of ending failed at 15 minutes.
2. ✅ A genuine human handoff still times out at 15 minutes and is distinguishable: stuck/needs_input → escalated, exhausted usage wait → new engine_limit. Both map to needs_human (relevant to [bug] The board showed three dead runs as "completed" and kept "Needs you" empty while all three waited on a human — a coding card has no needs-human status and follows the session, not the run #553), neither says failed.
3. ✅ The handoff reason now reaches the chat thread, not only the notification tray.
4. ✅ All three 2026-08-12 transcripts replayed through runCodingLoop yield waiting, not stuck.

Plus, from item (a): an expired handoff reads "Waiting on you: …" rather than "stuck not resolved in time" — the message that had runs which were never broken being re-tried by hand.

Also fixed, because the same 15 minutes produced them

• The handoff wait never heartbeated its own claims. 15 minutes of waiting against a 15-minute STALE_DRIVER_MS was one hiccup from a second Pilot taking the session. It ticks now, and is cancellable.
• runSucceeded replaces !== "failed" && !== "max_steps", which would have greeted a run that stopped waiting on the owner with "✅ Coder finished".

On the "early handoff is a signal" question

Not implemented, and I'd argue against it. start → handoff in 16–24s is a correlate of this class, not a cause, and a rule keyed on elapsed time would fire on the genuine fast handoffs too (the Codex trusted-directory case in this very issue is diagnosable in seconds). Now that waiting is its own outcome, the class is named directly and the timing proxy has nothing left to add.

File-size constraint

workflows/coding-session.ts was 799/800. The pause machinery is extracted, not added: lib/coding-wait.ts (pure — parsing, planning, clock, sentences) and lib/coding-pause.ts (the effectful wait, deps injected the way makeRunnerGuard injects its own). The workflow comes out at 799, no pin raised. 41 new tests.

What is NOT verified

Whether the brain reliably picks wait_for_reset over request_human on a live pane. The prompt rule is asserted and the tool→decision mapping is asserted, but the choice between them needs a real model against a real rate-limited CLI, and I did not drive any live agent to force one. This is the residual risk, and it is the reason to leave this open: the confirmation is d951b5f-style — the next usage-limit event on the account should produce a waiting/engine_limit run rather than a stuck not resolved in time one, checkable against the classified failure rows ba67afc (#529) writes.

Also unverified live: the durable step.sleep chunking under a real multi-hour park (unit-tested against injected deps only).

[serge-ivo]

The WITHIN-REACH case is delivered — re-verified with the arithmetic. No further code written.

The open question on this ticket was narrow and worth stating precisely. 67c41b7 (#580) made a
usage limit whose reset is beyond the wait ceiling end the run immediately, naming the instant —
verified live on run 70ea298e. That is the beyond-the-ceiling case. This ticket's headline is a
limit that resets WITHIN reach, 41 minutes out. Traced end to end today:

Constants (lib/coding-wait.ts): MIN_ENGINE_WAIT_MS 60s (:70) · MAX_ENGINE_WAIT_MS 6h
(:79) · ENGINE_WAIT_TOTAL_MS 6h (:82) · MAX_ENGINE_WAITS 3 (:91) · ENGINE_WAIT_TICK_MS
5min (:99) · backoff 15/30/60m (:108).

Run 6550f673's own numbers, the clean case in this ticket. Start 11:34:00Z, reset
22:30 Sydney = 12:30:00Z → delta 56 min. planEngineWait (:168-217) with {waits:0, spentMs:0}: the beyond-ceiling refusal at :193 asks 3,360,000 > min(6h, 6h) → no, so 56
minutes falls on the park side of that fork. Returns {wait:true, ms:56min, until:12:30:00Z, source:"deadline", capped:false}. waitForEngineReset (lib/coding-pause.ts:211-248) sleeps
ceil(3,360,000/300,000) = 12 chunks — eleven 5-minute sleeps plus a 60,000ms remainder (:231) =
exactly 3,360,000ms — then resumes the same goal object (workflows/coding-session.ts:653-677).

Old behaviour: dead at 11:49:15Z. New: resumes at 12:30:00Z. The 41 minutes are recovered.

reset distance	plan	outcome
41 min (headline)	park 2,460,000ms, capped:false	resumes at the reset
56 min (measured run)	park 3,360,000ms, capped:false	resumes at the reset
5 h	park 5h, capped:false	resumes, 1h budget left
6.5 h	> 21,600,000 → refusal (:193)	ends now, naming the instant — the 67c41b7 path

The park does not trip the three clocks it holds. pauseDeps.tick
(coding-session.ts:576-583) fires once per 5-min chunk and writes touchSessionDriver (vs
STALE_DRIVER_MS 15min), touchSessionActivity (vs IDLE_SESSION_MS 6h) and recordLiveness (vs
STALE_RUN_MS 3h). Post-fd1c323 this writes last_alive_at without touching
last_progress_at, and run-sweeper.ts:81 reads COALESCE(last_alive_at, last_progress_at, started_at) — so the run survives the sweeper and stops reading as work. Cancellable: tick
returns !isCancelRequested(...) (:582), read every chunk; worst-case Stop latency one 5-min
chunk.

AC1 ✅ (arithmetic above; resolvePause branches on waiting before the human path,
coding-pause.ts:131-133, so HANDOFF_WAIT_POLLS is never reached). AC2 ✅ (stopReasonFor,
:265-281, is a pure table: stuck/needs_input → escalated, waiting → engine_limit; both
needs_human, neither failed). AC3 ✅ (announce: postToChat, coding-session.ts:563).
AC4 ⚠️ asserted but weaker than its wording — coding-loop.test.ts:441 iterates the three
verbatim transcripts, but the harness injects the decision, so it proves the mapping
wait-decision → outcome:"waiting", not that a model reads those panes that way.

Tests: coding-wait (24), coding-pause (24), coding-loop (37) — 100 passed.

What remains unverifiable from code — and it is why I am not claiming more

Whether the brain picks wait_for_reset over request_human on a live rate-limited pane. Both
halves exist and are asserted — the tool (coding-loop.ts:451-466), the prompt rule (:490), the
clock in the user message (:530), and toDecision's mapping (:582-585) — but the choice is a
model behaviour. Confirming it needs a real usage-limit event producing a waiting/engine_limit
run row. The durable step.sleep chunking under a real multi-hour park is likewise exercised only
against injected deps.

Leaving open for that observation, exactly as the delivery comment asked. The code question this
ticket was reopened on — does a within-reach limit now survive — is answered: yes.

[serge-ivo]

Audit verdict: delivered — closing. c68b82a9 (the verb) plus 67c41b76 (#580's unreachable-deadline refusal). All four ACs are met and each has a test naming this issue.

AC	evidence
1. a run whose Engine reports a usage limit does not end failed in 15 minutes	lib/coding-loop.ts:136 adds "waiting" to CodingOutcome; :635 maps the wait_for_reset tool; lib/coding-pause.ts sleeps to the reset in 5-minute ticked chunks. coding-pause.test.ts:81 — "sleeps to the reported reset and resumes the SAME objective", asserting 56 minutes, the exact gap run 6550f673 died short of
2. a genuine human handoff still times out and is distinguishable in stopReason	coding-pause.test.ts:285 — stopReasonFor("stuck") === "escalated", stopReasonFor("waiting") === "engine_limit", and neither says failed. Both map to needs_human, so the column that means "someone should look" still catches both
3. the handoff reason reaches the instance chat thread	coding-pause.ts:157 calls deps.announce, wired to postToChat at coding-session.ts:592. coding-pause.test.ts:187 — "says so in the CHAT THREAD, not only the notification tray (#541 item d)"
4. replaying the three 2026-08-12 transcripts yields waiting, not stuck	coding-loop.test.ts:559-577 holds all three verbatim sentences and asserts outcome === "waiting", the detail, and the parsed waitUntil for each

The regression risks named on the ticket were all handled rather than hoped: the park heartbeats every chunk so it out-ticks the driver claim (15m), the run sweeper (3h) and the idle reaper (6h) — coding-pause.test.ts:99; it is cancellable mid-park — :113; and the resumed round is handed a platform note, never a userHint, so a platform action is not attributed to the human (:120, cross-referencing #505).

One thing the fix got right that the ticket did not ask for: a resume instant is refused when it has no explicit offset or Z, because Date.parse on a naked local timestamp resolves against the Worker's UTC and silently produces a ten-hour error (coding-wait.ts:33-37, coding-wait.test.ts:35).