[bug] A CODING_SESSION run that dies logs nothing — 0 logError calls where every peer workflow has 3-5, so a failed run exists only as a chat bubble

[serge-ivo]

A CODING_SESSION run that dies leaves no durable record anywhere but a chat bubble

$ grep -c logError workers/api/src/workflows/coding-session.ts
0

Every peer workflow logs its own crash:

workflow	logError calls
job-apply.ts	4
browser-task.ts	4
pipeline-run.ts	5
agent-loop.ts	3
coding-session.ts	0

Measured on a real failure

Instance f8ddc272-0390-4826-8812-94989e3d2ebd, run died 2026-08-12 07:18:48Z:

surface	newest record
instance_messages	✅ "Loop stopped (failed) — run error: UserAiProviderError: The AI provider stopped sending mid-reply…"
list_errors	❌ newest row for this instance is 07:07:10
agent_trace (all levels, 200 rows)	❌ newest event is 07:14:34
instance_board	❌ no card

The board is empty by design on this path: cards are minted only for delegations (coding-session.ts:231, if (event.payload.boardTaskId)), and a chat-initiated start_work passes none. That is defensible on its own; combined with no error log it means the only record of a failed autonomous run is a chat message the owner has to scroll to.

Why this is more than tidiness

It made a real question unanswerable. 2bc9475 (#518) added an automatic retry for a failed chat turn, gated on the provider's own retryable verdict (lib/resumable-round.ts:163-167). The 07:18:48 failure was a Pilot run, not a chat turn, and the natural next question — did the retry path apply here, and if not should it — cannot be answered, because the workflow recorded nothing to inspect.

It also could not be established whether that failure was a genuine provider stall or the owner's Anthropic credit exhaustion (two Anthropic (400): Your credit balance is too low errors at 06:59:50 and 07:07:09) resurfacing on the Pilot's streaming path. Those are different problems with different fixes, and the run left nothing to tell them apart.

Adjacent, and distinct

#527 asks for an MCP reader over coding_timeline — that is about auditing a finished run. This is about a failed run producing no error record at all. Fixing #527 would not surface this failure, because the timeline is not where a crash is recorded on the peer workflows either.

Acceptance criteria

1. A CODING_SESSION run that throws writes a durable error record, the way job-apply, browser-task, pipeline-run and agent-loop already do — readable via list_errors / agent_trace without scrolling chat.
2. The record distinguishes the failure class where the platform knows it: a provider stall, a credentials/credit failure, and a platform ceiling (see A 2-hour run that closed ten issues and pushed 15 times is reported as "outcome: failed" with a Wrangler docs link — the Cloudflare subrequest ceiling is relayed raw as the objective failing #523) are three different things and currently read identically.
3. It carries enough to correlate — run id, session id, trace id — so a failed run can be joined to the turn that started it.
4. A test asserting the write on the throw path, not merely on the happy path.
5. Consider whether a failed chat-initiated start_work should mint a board card too, or state why the chat message is the right and only owner-facing surface.

Verified vs inferred

• Verified: the grep counts, all four surface queries above with their timestamps, coding-session.ts:231's board-card condition, and resumable-round.ts:163-167.
• Inferred: nothing load-bearing. The claim is an absence, and the absence was grepped.

[serge-ivo]

Shipped in ba67afc, deployed (CI green, Deploy API Worker success, api.proagentstore.online/health ok). The next Pilot death on f8ddc272 will be the first one that can be read back.

What the record now says

A CODING_SESSION run that throws writes one error_log row through recordCodingFailure (workers/api/src/lib/coding-failure.ts), mirrored into agent_events:

source   coding:session          ← filterable in list_errors; the trace bridge splits at the colon,
                                   so the agent_trace row lands under `coding` beside the run's acts
message  coding run failed (provider_stall) at s7-decide after 4 steps: <the provider's own words>
context  { failureClass, retryable, upstreamStatus, phase, steps, instructionChars, paneChars,
           elapsedMs, sessionId, runId, taskId, traceId, node, repo, resumableRound, stack }

AC 2 — the class. classifyCodingFailure is pure and follows #339/#341's rule: the site that knows says so, instead of a catch site guessing from wording. Nine classes; the three the ticket names are provider_stall, provider_credentials (an invalid key or an exhausted balance — the 07:07 credit balance is too low would have been recorded as its own class, not as a stall) and platform_ceiling (#523's Cloudflare per-invocation limit — a run cut off, not an objective failing). Matching is textual as well as structural, because a Workflow serialises a thrown error across a step boundary and the receiving side gets a message, not a prototype — which is why the live bubble reads UserAiProviderError: The AI provider stopped sending mid-reply. The test feeds the classifier the real output of deadlineMessage() / connectionLostMessage(), so rewording a message in ai-deadlines.ts fails the test rather than silently degrading every future stall to unknown. Explained classes (runner offline, deploy reset, exhausted balance, throttled key) log warn per #424; a stall, a ceiling and an unclassified death stay error.

AC 3 — correlation. This needed a fix outside the workflow: logError's trace bridge read context.taskId and nothing else, so any caller whose correlation key has another name — a coding run's loop-run id, a pipeline's run id — mirrored into agent_events with a NULL trace_id. Present in the timeline, invisible to ?trace_id=<run>, which is the query an investigator makes. It now prefers an explicit context.traceId and still honours taskId. The coding record uses the loop-run id, falling back to the session id so a chat-initiated start_work is still findable.

Enough to diagnose the failure, not just its existence. CodingRunProbe carries the durable step name (s7-decide, s13-waitidle), the pilot step count, the instruction length and the terminal size. All set from code outside step.do, which a Workflow replay re-executes while replaying journalled results — so a run resumed after an eviction still reports where it actually died, rather than the zeroes a step-journalled counter would give.

AC 4 — the test. workers/api/src/lib/coding-failure.test.ts, 13 tests. The write is asserted on the throw path: the workflow class cannot be constructed in vitest (it imports cloudflare:workers, which is not resolved here), so the catch-block region of coding-session.ts is read from source and asserted to call recordCodingFailure with the probe, the step count and the run start — the same style as agent-workflows.test.ts. A test that only exercised the writer would prove the writer works while the crash path still called nobody, which is the exact state this ticket describes.

AC 5 — no board card, and why

The board is where pending human work lives: a delegation in flight, an approval gate, an uncommitted diff awaiting a decision (#276). A crashed run's only decision is "start it again", and that is done from the thread it was started in. Three owner-facing surfaces already exist for a failed run and all three are ungated by loopRunId: the chat message, the ⚠️ Coder stopped push from the notify-end step, and — as of this commit — the durable error row and trace event. A card would be a fourth notice with no new decision attached to it, competing with the cards that do carry one. What was missing was never a card; it was a queryable record.

The retry question — judgement, not a guess: leave it, and the reason is not idempotency

I looked at whether 2bc9475's treatment (#518) belongs on the Pilot. It does not, and the honest reason is that the Pilot already has a stronger version of it.

coding-session.ts wraps the decision in step.do(..., retry) where retry = { retries: { limit: 2, delay: "2 seconds", backoff: "constant" }, timeout: "3 minutes" }. A throwing decideCodingAction is therefore attempted three times before the run dies — where thinkWithAutoResume gives a chat turn exactly two. Adding an auto-resume on top would be a fourth attempt wearing the name of a mechanism the Pilot does not need:

• There is no round to preserve. [bug] #442 stores a resumable round the user cannot reach — the gate is byte-equal text, the owner speaks, and the console clears the composer before sending #518 exists because a failed chat turn had already executed tools, and autoResumableRoundOf replays those results so they are not re-run. decideCodingAction executes nothing — it is one stateless call over {goal, actionLog, snapshot} returning one action, which the workflow performs in a separate step. resumableRoundOf on a Pilot failure returns null, always. The new record logs resumableRound so that stays an observation rather than an assumption.
• The genuinely dangerous replay is the one that already happens. act and message carry the same retries: { limit: 2 }, so an instruction whose response is lost after the engine received it is re-sent today. That is a pre-existing at-least-once property of the drive path. An outer "retry the run from its stored round" would replay a sequence of engine instructions with no equivalent of the round's executed/mutations dedup — that is the "re-drives an engine on the owner's machine" hazard, and I am not shipping it.

So: no new issue for a Pilot retry. The manual "Retry" the owner has typed three times is not filling a gap in the retry policy — it is compensating for the fact that three runs with three attempts each died. Which brings the real finding:

What the three deaths now look like, read against the code

Nine provider calls (3 runs × 3 step attempts) failed on one instruction — "read the full contents of both files", 366 + 309 lines. A transport that fails nine times out of nine on one payload and works on every other instruction is not the intermittent stall its message describes; the stall deadline is 20s of complete silence after the stream has begun (AI_STALL_TIMEOUT_MS), so the reply started and then stopped, repeatedly, on the same request. That points at the request — the decide prompt is the system prompt plus the whole actionLog plus snapshot.pane.slice(-6000), and an instruction that makes the engine emit two full files makes the next pane the largest of the run.

I have not verified that; it is the hypothesis the new paneChars / instructionChars / phase fields exist to confirm or kill on the next occurrence, and it deserves its own issue written from measurement rather than from this paragraph. Filing it belongs to pags-ba once there is one logged death to read.

Closing this: the logging, classification, correlation and test are delivered and live.