A 2-hour run that closed ten issues and pushed 15 times is reported as "outcome: failed" with a Wrangler docs link — the Cloudflare subrequest ceiling is relayed raw as the objective failing

[serge-ivo]

What the owner saw

His Coder Lead delegated "Work through all open GitHub issues by priority order — bugs first, then enhancements". Two hours later the Assistant thread said, in full:

Loop stopped (failed)

The board card says more, and it is worse:

Failed · deleg-ae7fa3f2-53a1-43ba-b3ad-921dd8dba7b1
outcome: failed — run error: Too many API requests by single Worker invocation. To configure this limit, refer to https://developers.cloudflare.com/workers/wrangler/configuration/#limits | Acts: pushed directly to the trunk origin main; pushed directly to the trunk origin main; pushed directly to th…

Read from the owner's side: his objective failed, and the remedy is a Cloudflare Wrangler configuration page for a Worker he does not own. There is no sentence anywhere telling him that the run had already implemented and pushed most of the backlog before it died.

What actually happened

Session csess_e80b6a21-a154-4905-b689-fe5c5cd16289, instance bd43f4de-ef35-4051-bdec-43f8571414a1, node RLs-MacBook-Air.local, 2026-08-11 01:29:42 → 03:31:35, status: error.

Measured from GET /v1/instances/:id/coding/sessions/:sid/timeline and agent_trace:

• 26 Pilot steps driven into the engine over 2h 02m.
• Real work throughout: issues [connectors] Web-search connector (Google Custom Search / SerpAPI) — enrichment #99, Connector framework: registry + connectorClient (#86, step 2) #104, [pipelines] Pipeline definition format + durable runner (trigger → declarative pipeline) #97, Overseer agent: one cross-repo conversational AI (smart auto voice mode) with memory + self-awareness #9, Unify tool registry on a single ToolDef shape (#85, step 1) #102, Document platform architecture #12, [browser][connectors] Expose the browser as a registry tool — let config/data agents declare browser_* (not only the Tier-0 workflow path) #103, Defense-in-depth: encodeURIComponent repo segments in github-issues.ts #110, Trigger payload mapping UI and config validation #16, Capstone: lead-finder as a pure declarative pipeline (#94) #117 read, implemented, committed; issue Capstone: lead-finder as a pure declarative pipeline (#94) #117 opened by the run itself at 03:16:36.
• 15 act.consequential / push.trunk events, each irreversible: true, ok: true.

The run did not fail. It was cut off at 2 hours by a platform ceiling, and the only thing it left behind that a human reads says "failed".

Mechanism

Verified. workers/api/src/workflows/coding-session.ts:669-677 wraps the whole loop:

} catch (e) {
    const message = e instanceof Error ? e.message : String(e);
    result = { outcome: "failed", detail: isRunnerGone(e) ? message : `run error: ${message}`, ... };
}

There is exactly one classified escape — isRunnerGone, from #341. Everything else, including a Cloudflare runtime limit, becomes run error: <raw platform string> and is then the detail on the board card, the loop-run row and the chat message.

Verified. The run's dominant subrequest consumer is the idle poll. coding-session.ts:497-509:

// Poll capture until the CLI goes idle … Bounded so the loop can't outrun idleRetry's 10-minute step timeout.
waitIdle: () =>
    guard(runIdle, `s${n++}-waitidle`, async () => {
        await sleep(1500);
        let snap = await capture();
        for (let poll = 0; poll < 240 && snap.runState !== "idle" && snap.alive && !snap.cancelled; poll++) {
            await sleep(2000);
            snap = await capture();
        }
        return snap;
    })

sleep is a plain setTimeout promise (coding-session.ts:795) — not step.sleep. So up to 241 capture() calls happen inside a single step.do. Each capture() (coding-session.ts:383-431) issues, at minimum: one relay callRunner for /coding/capture, one D1 SELECT status FROM coding_sessions, and one isCancelRequested D1 read — plus recordEngineUsage, recordEngineActs, recordAuthorityViolations and touchSessionActivity, all of which reach D1. That is ≥3 and typically 5–7 subrequests per poll, so one waitIdle can spend 700–1700 subrequests. A message action calls waitIdle every step (coding-loop.ts, end of the action branch), 26 times in this run.

Inferred, and labelled as such: that this is where the invocation's budget went. I have not instrumented a live Workflow to attribute the exhausted counter to a specific step, and Cloudflare's step/invocation boundary semantics are not something I verified from the runtime. What is verified is that (a) the ceiling was hit, (b) the platform already knows this ceiling — workers/api/src/lib/repo-ingest-runner.ts:252 describes an incident where "ONE tick drained a 300-file queue (blowing past the 1,000-subrequest cap, which embed's catch swallows)" and a chunk budget was added specifically to stay under it — and (c) workers/api/wrangler.toml declares no [limits] block, so the API Worker runs at whatever the default is.

What to do

1. Say what happened, in the platform's own voice — the cheapest and most valuable step.
Follow the pattern #341 already established in this exact catch block: a failure that KNOWS what it is says so, instead of being inferred from a message at the catch site. Add a classifier beside isRunnerGone for a platform-ceiling error (Too many API requests by single Worker invocation, Too many subrequests, exceeded CPU) and emit something the owner can act on:

This run reached a Cloudflare per-invocation limit after 2h and 26 steps. It was not your objective failing — the work it had already committed and pushed is intact (15 pushes to main). Resume it, or narrow the objective.

Crucially it must not be reported as outcome: failed with no qualification: a delegating supervisor reads check_delegation and the board card, and both currently say the objective could not be done.

2. Stop spending the budget in one invocation. Break waitIdle into bounded chunks: poll ~15 times inside a step, then await step.sleep(...) and continue. step.sleep is already used in this very file for the handoff wait (coding-session.ts:650), for the same reason — a long wait must not be held open inside one invocation. This also makes a long engine turn survive a Workflow eviction instead of burning the retry budget.

3. Consider raising the ceiling explicitly, but not instead of (2). wrangler.toml supports [limits] subrequests (Cloudflare docs: default 10,000 on paid, maximum 10,000,000). Adding it is one line. It is deliberately listed third: raising the ceiling alone means the next run of this shape goes further unsupervised before it dies — a run that pushed to main 15 times without a human reading anything is not obviously improved by letting it push 30 times.

4. Checkpoint, so a cut-off run is resumable. The Pilot's progress lives entirely in the run's in-memory actionLog/transcript (coding-loop.ts). When the invocation dies, so does the plan. A run that has closed 8 of 12 issues should be resumable from issue 9, not from the objective.

Alternatives rejected

• Just raise [limits] subrequests and close this. Fixes the symptom for a run of this length and leaves the message, the outcome classification and the unresumability exactly as they are. It also does nothing for the CPU-time ceiling, which the same wrapper will report the same way.
• Cap maxSteps lower. A 26-step run that closes ten issues is the product working. The defect is that the platform cannot tell the owner what happened to it.
• Poll less often. 2s is what makes the terminal feel live and is what touchSessionActivity is throttled against. The problem is the poll's placement inside one invocation, not its frequency.

Regression risk

Chunking waitIdle across step.sleep boundaries adds durable steps to every engine turn — n (the step-name counter) grows, and a Workflow step name must stay unique within an instance, which this file already manages via s${n++}. The risk to watch is a long engine turn now spanning many steps and colliding with the Workflow's own step-count limits; test with a synthetic engine that stays non-idle for the full 8-minute window and assert the loop still terminates at the same boundary with the same snapshot.

Not reproduced end-to-end: I did not force a subrequest exhaustion in a live Workflow. Everything above is read off the production record of one run plus the code paths it took.

Related: #341 (the precedent — a connectivity failure classifies itself instead of being guessed from its message at the catch site).

[serge-ivo]

Frequency data from a population sweep (2026-08-13)

I pulled GET /v1/instances/:id/loop across all 32 instances on the owner's account: 80
autonomous runs, 30 failed. This issue's class is 3 of the 30 — and the interesting part is
that it is almost perfectly predictable.

It tracks accumulated work, and the practical boundary is ~2 hours

run	wall clock	Pilot steps	outcome
FWS platform	122m	37	subrequest ceiling
Chess coder 2	122m	26	subrequest ceiling
PAS Coder	120m	37	subrequest ceiling
FGS platform	99m	34	completed
Heartfull	85m	35	(died of a relay 504 instead)

Every run that reached ~2 hours of active stepping died of this. The nearest survivor stopped at 99
minutes. It is not a step-count threshold — Chess coder 2 died at 26 steps while FGS survived at 34
— and it is not wall clock alone either: the two longest runs in the whole population (204m and
189m, #541) had 1 step each and never came close. It tracks the journal, which is what a
replayed invocation has to walk.

All three were one delegation fan-out

They started within 59 seconds of each other (2026-08-11 01:28:43 / 01:29:40 / 01:29:42), from a
Coder Lead delegating to five subordinates, and died within 93 seconds of each other two hours later
(03:30:05 / 03:30:46 / 03:31:38). The other two runs in that fan-out died of WorkflowInternalError
(#546). Five of five subordinate runs in that batch died; none of the five deaths was the
objective failing. That is the single worst hour in the population, and the owner's read of it was
five "outcome: failed" cards.

Why this is now the top fixable platform-fault class

With BUDGET_ENFORCE off since d951b5f (2026-08-10T08:16Z), the 11 budget kills cannot recur. That
leaves 11 live-era failures, of which this is 3. Ranked by "can the platform fix it": stuck
mis-handling (4, #541), this (3), WorkflowInternalError (2 in the live era, 8 all-time, #546).

One addition to the fix

Whatever this issue lands, the acceptance should include a duration/journal guard that ends a run
cleanly before the ceiling rather than only re-wording the message after it. A 2-hour run that
closed ten issues and pushed fifteen times is the good case, and today the platform's answer to it
is to lose it. The ~2h figure above gives a concrete number to checkpoint or hand off at.

Cross-refs: #541 (full 80-run distribution), #546 (WorkflowInternalError, the other half of this
fan-out).

[serge-ivo]

Shipped in 7e23b60. CI green, API Worker deploy green (Current Version ID: c8122fe2-0f03-45c1-99a0-b2369c5efb14), /health 200.

Leaving this open — item 4 (checkpointing) is not done. See the bottom.

One documented fact changed the design, and it invalidates fix #2

Cloudflare applies this ceiling per Workflow instance, not per invocation:

Maximum number of subrequests per Workflow instance — 50/request (Free) | 10,000/request (default) / configurable up to 10 million (Paid)
— workflows/reference/limits, read 2026-08-13

So neither a step.do boundary nor a step.sleep resets the counter. Breaking waitIdle into chunks separated by step.sleep would have bought nothing at all, while adding a durable step per chunk. It is not implemented, and that is why. (The step budget was never the constraint either: 10,000 steps per Workflow on paid, configurable to 25,000.)

The arithmetic, which turned out to be the finding

capture() costs four subrequests, counted from source rather than estimated:

1	callRunner → one RELAY DO stub.fetch (runner-client.ts:267)
1	SELECT status FROM coding_sessions
1	isCancelRequested → SELECT cancel_requested FROM agent_loop_runs
1	touchSessionActivity — its 60s throttle is a WHERE clause, so the subrequest was spent on every poll to move a row 1 time in 30

(recordEngineUsage, recordEngineActs, recordAuthorityViolations all return before touching D1 on an empty report — the steady state between turns.)

At a flat 2s poll that is 2 subrequests per second of Engine work, so 10,000 buys 83 minutes of waiting. Modelled over Chess coder 2's own measured shape (26 Pilot steps across 122 minutes = 4.7 min of Engine per step): 26 × 142 polls × 4 = 14,768 against a 10,000 ceiling. That is the mechanism, and it is arithmetic rather than inference. It also explains the population data in the comment: 26 steps and 37 steps both died while the 99-minute survivor ran 34, because it never tracked step count.

What shipped

1. The poll backs off — 2s for the first 30s (where most turns finish), then 5s, then 10s past two minutes. Same run: 26 × 51 × 3 = 3,978, which fits under the ceiling that killed it with 60% to spare.

The issue rejected "poll less often" on two grounds and both are false of this poll:

• "2s is what makes the terminal feel live" — it is not. The live terminal is the console's own 3-second GET …/sessions/:sid/capture, on its own rate-limit bucket. This poll runs inside a durable Workflow with no browser attached; all it decides is when the Engine stopped working.
• "touchSessionActivity is throttled against it" — it is throttled against 60 seconds (ACTIVITY_TOUCH_MS), which any interval under a minute satisfies identically.

The price is up to 10s of extra latency at the end of a turn already measured in minutes: 3.5% on a 2-hour run, against a run that used to not finish.

2. The activity heartbeat is asked before it is paid for — shouldTouchActivity at the caller, the same 60 seconds, taking a poll from 4 subrequests to 3.

3. The ceiling is raised — third, and not instead. The Wrangler docs link was not inapplicable: wrangler 4.98's own config-schema.json carries limits.subrequests, and wrangler deploy --dry-run accepts the block with no warning. workers/api/wrangler.toml now declares [limits] subrequests = 100_000 — 1% of the documented 10,000,000 maximum, ~25 hours of Engine waiting at the new poll cost, still small enough to stop a genuine runaway. Raising it alone would only have let the next run of this shape go further unsupervised, which is this issue's own argument and it stands.

Item 1 was already done by c64d9f5 (#546) — platform_ceiling is classified and reported as an interruption with its own LoopStopReason. This change deliberately adds no second word for that idea.

The stated regression risk, tested

The old bound was a poll count (240 × 2s) sized against idleRetry's 10-minute step.do timeout. Carried onto a backed-off schedule it would have stretched the window to ~40 minutes and moved the failure from "the wait gave up" to "the durable step timed out", which reads as a crash. IDLE_WAIT_MAX_MS is elapsed sleeping time, asserted equal to 240 * 2_000, and a synthetic Engine that never goes idle is held for the full window and asserted to stop at the same boundary with the same snapshot.

Extracted to lib/coding-idle-poll.ts — "how long may a run wait, and what does waiting cost" is a rule, and a rule inside a Workflow can only be tested by running one. workflows/coding-session.ts is 796 after, one line above the 795 it started at and under its 800 ratchet; the pin is unchanged.

21 tests, stating the sizes measured (ADR 0002), each proven by watching it fail (G4): flat 2s poll → 7 red; 240-poll count carried onto the new schedule → 3 red; no activity throttle → 1 red; no settle sleep → 2 red; ignoring cancelled/alive → 2 red.

Verified, and not verified

Verified: all ten CI gates green in a clean worktree at the commit (typecheck 11 projects; 8,422 tests; biome 1,172 files; seven check-*.mjs; docs:drift; docs:build; console tsc + vite build), plus the same bar again after rebase alongside #516 (8,447 tests). Deploy green, /health 200.

Not verified live. [limits] subrequests is not observable through any endpoint I can reach, and the behavioural proof is a 2-hour Pilot run, which I did not start. The deploy accepting the config is the evidence I have. pnpm test:e2e skipped — the diff touches no store/, agents/coder/web or anything Playwright exercises.

Left open

Item 4 — checkpointing. The Pilot's progress lives entirely in the run's in-memory actionLog/transcript (coding-loop.ts), so a cut-off run still restarts from the objective rather than from issue 9. Making that durable is its own change.

The comment's "one addition" — a guard that ends a run cleanly before the ceiling. Not implemented, deliberately. With spend 3.7× lower and the ceiling 10× higher, the run this was filed for now needs ~25 hours of Engine waiting to reach either, so a pre-emptive stop would guard a case that should no longer arrive. It is worth having as a backstop; it was not worth folding in here on an unmeasured trigger point. If it is wanted, it should be sized against a real observation rather than against the old 2-hour number, which no longer means anything.

[serge-ivo]

No code change in this pass. Re-checked against production because the run-lifecycle work of the last two days moved the ground under this. Items 1–3 are delivered (c64d9f5, 7e23b60); item 4 remains, and it is a design decision — options below. Leaving open.

The ceiling has not recurred

Swept every one of the 124 rows in the error log (all sources, 2026-07-23 → 2026-08-15) for platform_ceiling and for each of the classifier's own CEILING_MARKERS — too many api requests by single worker invocation, too many subrequests, exceeded cpu, cpu time limit, exceeded memory, exceeded resource limits — matched against the full serialized row, so message, context, last_context and stack were all covered.

Zero hits. Post-fix window: 2026-08-13 04:46 (deploy of 7e23b60) → 2026-08-15 09:17, ~2.35 days, during which 4 coding:session failures were recorded — all other classes. Two of them were long runs that would have been candidates under the old poll cost (elapsed 4h25m and 5h23m); both died at notify-coding-wait:0-0, not at a ceiling.

Honest limit: the error log holds no coding:session rows before 2026-08-12, so I can show you "0 after" but not "0 after vs N before". On a low-traffic single-user account, 2.35 days quiet is suggestive, not conclusive.

Item 1 is now doubly covered

c64d9f5 classified platform_ceiling and put it in INTERRUPTED_CLASSES, so it reports as an interruption with stop reason interrupted. 4ff78f9 (#546) adds the other half: the durable error_log row for an interruption the platform resumes now says coding run interrupted … resumed rather than coding run failed. platform_ceiling is resume: false, so a ceiling still ends the run — correctly, since a replay would walk into the same ceiling — but it no longer reads as the objective failing anywhere.

Item 4 — checkpointing. Options, and why I did not just build one

The Pilot's progress still lives entirely in the run's in-memory actionLog/transcript (lib/coding-loop.ts), so a cut-off run restarts from the objective rather than from issue 9. That is unchanged and real. But the shape of the problem changed underneath it, twice:

• 03762fd (A platform deploy kills in-flight coding runs, and the Pilot writes retryable:true for a reader that does not exist #583) made a resume the normal path for infra_transient. A resumed run replays the Workflow journal, so every completed step.do returns its recorded result and the transcript is rebuilt for free. Checkpointing would be redundant for exactly the interruption class that now dominates the log (4 of 10 rows).
• The ceiling's own arithmetic moved 10×. With the backed-off poll and [limits] subrequests = 100_000, the run this ticket was filed for now needs ~25 hours of engine waiting to reach either limit.

So the case checkpointing still serves is narrower than when it was written: a run killed by a class that is not resumed (provider_stall, provider_credentials, platform_ceiling) after real work.

(a) Persist the action log per step. Truest to the ask; costs a durable write per Pilot step on every run to serve the minority that die. Wants a size bound — a 26-step transcript is not small.

(b) Checkpoint the OBJECTIVE, not the transcript. The run already writes what it did: agent_trace acts, coding_timeline, the board card. A resumed run could be handed "these issues are already closed, these commits are already pushed" as a resume note — the field goal.resumeNote already exists and is already cleared per round. Much cheaper, and it matches what a human would actually need. It does not restore the model's reasoning, only its results.

(c) Do nothing, and let #583's resume cover it. Defensible for infra_transient; wrong for the other three classes.

My read: (b). It reuses machinery that exists, it is bounded, and the thing a cut-off run actually needs is not its old plan but a truthful statement of what has already landed — which is the same complaint this ticket opens with. But that is a product judgement about what a resumed run should be told, and it should be your call rather than mine, so I have not built it.

On the comment's "one addition" — a guard that ends a run cleanly before the ceiling

Still not implemented, and I agree with the previous decision. With spend 3.7× lower and the ceiling 10× higher, a pre-emptive stop would guard a case that no longer arrives — and there is now 2.35 days of production evidence for that rather than only arithmetic. If it is wanted as a backstop it should be sized against a fresh observation; the original 2-hour number no longer means anything.