[bug] WorkflowInternalError, traced: it is not the objective failing and not terminal — one run filed its own death twice, and all 8 occurrences classify as `unknown`

[serge-ivo]

Follow-up to #529, which asked for classified records and got them. Now that they exist, the first
thing they show is that the largest class is unknown, and every member of it is the same
Cloudflare failure — the one nobody has ever traced. This is that trace.

Population

Across all 32 instances (80 autonomous runs, 30 failed — see #541 for the full distribution),
WorkflowInternalError is the largest live failure class after stuck:

• 5 loop-run rows: run error: WorkflowInternalError: Attempt failed due to internal workflows error
  (2026-08-05 07:38Z FWS, 2026-08-07 03:37Z + 03:52Z Heartfull, 2026-08-11 01:28Z FAS, 01:29Z Heartfull)
• 3 error_log rows, post-[bug] A CODING_SESSION run that dies logs nothing — 0 logError calls where every peer workflow has 3-5, so a failed run exists only as a chat bubble #529: internal error; reference = <id>, all at phase s6-decide,
  all on instance a1d3522f (2026-08-12 15:05:34Z, 15:13:35Z, 15:21:19Z)

The two message shapes are the same event — CF's internal-error wording differs by surface. That is
8 occurrences, and 3 of them are in the last 24 hours. Of the 4 coding:session rows that exist
since #529 shipped, 3 are this and all 3 classify as unknown.

It is not terminal — one run recorded its own death twice

The decisive evidence. Two error_log rows, eight minutes apart:

	row A	row B
created_at	2026-08-12 15:05:34	2026-08-12 15:13:35
runId / traceId	82739cb6-…	82739cb6-… (same)
sessionId	csess_2dd3124c-…	csess_2dd3124c-… (same)
elapsedMs	10,619,147	11,100,612
node	Sergeys-Mac-mini.local	RLs-MacBook-Air
instructionChars / paneChars	45 / 216	0 / 0
phase	s6-decide	s6-decide

elapsedMs is Date.now() - runStartedAt (coding-failure.ts:274) and the two differ by 481,465ms
against an observed wall-clock gap of 481,000ms — the same runStartedAt, i.e. one run, not two.
So the workflow's catch block (coding-session.ts:667-679) executed twice for a single run.

And it was not the end even then: the loop-run row for 82739cb6 was finally closed at 15:17:25Z
with a third, different reason — stuck not resolved in time. Neither failure record was terminal.

Why the record duplicates

recordCodingFailure is called directly in the catch, not inside step.do
(coding-session.ts:674-677). Anything outside step.do re-executes on every workflow attempt, so
each attempt writes a fresh row while the journalled step.do results are replayed.

Row B's 0 / 0 is the confirming signature. probe.saw(pane.pane) is at coding-session.ts:397,
inside capture, which is invoked through guard(runRetry, \s${n++}-snapshot`, capture)at line 447 — i.e. **inside**step.do. A replay gets the journalled result and never re-runs the callback, so paneCharsstays 0.probe.at(name)(line 281) *is* outside, which is whyphasestill readss6-decide`.

That directly falsifies the docblock #529 shipped, coding-failure.ts:186-192:

"Everything here is set from code OUTSIDE step.do, which a Workflow replay re-executes … So a
run resumed after an eviction still reports the phase and payload sizes it actually died at,
rather than the zeroes a step-journalled counter would give."

Production row B is exactly "the zeroes". The phase claim holds; the payload-size claim does not.

Three consequences

1. An infra event is reported to the owner as the objective failing.
coding-session.ts:671 wraps anything that is not isRunnerGone into `run error: ${message}`,
and closeDelegation turns that into outcome: failed. Two of the five loop-run rows carry
| Acts: pushed directly to the trunk origin main; pushed directly to the trunk origin main; … —
the run had already done real work and pushed it and was still reported as failed. Identical in
shape to #523, and identical in cost: he cannot tell a dead run from a finished one, so he retries.

2. It classifies as unknown, so it is indistinguishable from a genuine unexplained crash.
Walked through classifyCodingFailure (coding-failure.ts:129-162) with
"WorkflowInternalError: Attempt failed due to internal workflows error": not isRunnerGone, not
isRunnerUnreachable, no CEILING_MARKERS hit, isTransientInfraError returns false (it matches
only durable object reset / code was updated / object has been reset / reset because —
transient-error.ts:12-15), no credential marker, upstreamStatus null, no rate-limit / overrun /
stall marker, does not start with anthropic → falls through to at("unknown") at line 162.
Result: level: "error" (unknown is not in EXPLAINED, line 173-180) and retryable: null.

3. Nothing collapses, so any frequency count off list_errors over-reports.
collapseRepeat matches on exact message (error-log.ts:116-121). Every CF internal error
carries a distinct reference = <random id>, so no two ever collapse — all three rows show
repeat_count: 1. Combined with (1), row count ≠ run count: 3 rows here are 2 runs.

What to do — cheapest first

(a) Give it a class. Add workflow_internal to CodingFailureClass and match
attempt failed due to internal workflows error + internal error; reference =. Put it in
EXPLAINED so it is warn, and set retryable: true. One-line-ish, and it immediately makes
"is unknown shrinking?" a real question again — today unknown is 75% of the coding rows and
means nothing.

(b) Do not report it as the objective failing. Same argument #523 makes for the subrequest
ceiling: the run was cut off, its committed work is intact. It should read like the isRunnerGone
branch already does at coding-session.ts:671 — reported as itself, without the run error: prefix
that "reads as a crash". A run that pushed to main and then hit an attempt error is not failed.

(c) Strip the volatile reference out of the collapse key. Either normalise the message before
collapseRepeat or add a signature column. Otherwise every future CF internal error is a new row
and #522/#538's repeat machinery is inert for this class — which is the class that most needs a
count.

(d) Fix the coding-failure.ts:186-192 docblock, and either move probe.saw/probe.drove
outside the journalled callbacks or state plainly that payload sizes are absent on a replayed
attempt. As written it invites the next reader to treat paneChars: 0 as "the run read an empty
pane" rather than "this row is a replay" — which is a wrong diagnosis, not just a missing one.

Open question for the owner

Should the catch re-throw instead? If WorkflowInternalError were rethrown, CF's own retry
machinery would handle it and the run might genuinely resume rather than be declared dead. I would
not do this yet: recordCodingFailure, closeDelegation and the finally block all have
side effects that would then repeat per attempt, and #529's own comment
(coding-session.ts:667-668) exists because a run that vanished with no record was the worse
failure. (a)+(b) are safe and independent; the rethrow is a design change that wants its own ticket.

Acceptance criteria

1. classifyCodingFailure("WorkflowInternalError: Attempt failed due to internal workflows error")
   and …("internal error; reference = abc123") both return class: "workflow_internal",
   retryable: true, level: "warn".
2. A loop run killed this way does not carry the run error: prefix, and its stopReason is
   distinguishable from a run whose objective genuinely failed.
3. Two occurrences with different reference ids collapse into one error_log row with
   repeat_count: 2.
4. A unit test feeds the exact production strings from the 8 occurrences above (as
   coding-failure.test.ts already does for deadlineMessage()), so a reworded CF message fails the
   test rather than silently degrading to unknown again.

Regression risk

• Moving workflow_internal into EXPLAINED/warn hides it from an error-level triage view. That
  is the intent, but it means (c) must land with it — otherwise the class becomes both quiet and
  uncountable, which is worse than today.
• Changing the terminal detail string touches closeDelegation's reason mapping
  (coding-session.ts:194-200) and the board-card ok/fail choice (line 232). seed-drift-style
  string assertions elsewhere may pin the old wording — grep -rn "run error:" before changing it.
• (c) risks over-collapsing if the normaliser is too aggressive: two genuinely different errors
  merging into one row loses a real signal. Normalise only the reference = <id> token, nothing else.

Provenance

The 5 loop-run occurrences are reconstructed from agent_loop_runs.detail via
GET /v1/instances/:id/loop and predate #529 — they have no error_log rows and no phase or
payload data; that era is only readable from the run row and chat. The 3 error_log rows are
post-#529 (2026-08-12) and are where every structural claim above comes from. The duplicate-record
finding is measured. That Cloudflare retried the attempt (as opposed to some other double-entry
path) is inferred — it is the natural reading of "Attempt failed" plus the fresh-probe
signature, but I did not reproduce it, and CF's public Workflows API doc 404s on the
retry-semantics page.

[serge-ivo]

Fixed in c64d9f5 — CI green, Deploy API Worker succeeded, api.proagentstore.online/health serving.

The judgement you asked for: classify it, report it honestly, do not retry the run

retryable: true, and the branch comment now says which retry it means, because the field is ambiguous between two very different acts:

• The ATTEMPT retry is already happening, unasked, and it is safe. Row B is that retry — same runId, same runStartedAt, journalled step.do results replaying (which is exactly why paneChars was 0). Cloudflare does it by itself and the Engine is not re-driven for anything already journalled.
• A RUN-level retry is not safe, and I did not add one. Two of the five rows carry | Acts: pushed directly to the trunk origin main. A fresh run has no journal, so every act repeats against a repo that already has the commit. The Pilot answers a CLI safety refusal by repeating the instruction — 8 escalating sends, 2 with a byte-identical command, then "Loop complete" on the subset the owner did not ask for #522's dedup is per-run and would not help.

Grepped the consumers before deciding: CodingFailure.retryable is written to the error_log context at coding-failure.ts and read by nobody else — no machine acts on it. So the risk was never a bad retry today, it was somebody later building an auto-retry on a field that reads as a licence. It now says what it licenses.

(a) Given a class

workflow_internal, matching all three markers (attempt failed due to internal workflows error, workflowinternalerror, internal error; reference =), in EXPLAINED → warn. A test feeds it the literal production strings from both surfaces.

(b) Not reported as the objective failing

New pure module lib/coding-run-report.ts. codingCrashReport drops the run error: prefix for the three classes where the platform cut the invocation off — workflow_internal, platform_ceiling (#523's argument, which you invited) and infra_transient — and says the committed work is intact. runner_gone is deliberately excluded: #341 already wrote it a better sentence, and it differs in the way that matters (the machine is not coming back on its own).

stopReason is now distinguishable: a new LoopStopReason, interrupted. statusFor puts it in needs_human beside escalated and engine_limit — a run that may have pushed should send its owner to the repo, not to Retry. A test asserts no CodingOutcome can produce interrupted, so the reason means only one thing. Console label: "Cut off by the platform".

(c) Collapsible — and I fixed it at the caller, not in the collapse

error-log.ts's premise is "exact text carries no extra information by definition, which is what makes collapsing it lossless." That is true of every message that does not embed a unique id, and this row broke the premise by putting one in prose. So no migration and no signature column: splitCfReference moves the reference to context.cfReference, where both ends survive — context keeps the first occurrence's, last_context the latest (#538). Every other message is byte-identical to before, so nothing over-collapses.

You asked me to re-check the interaction with 6becf3a (#538) and 0125 (#539). It still holds, and neither was in the way: #538 changed the UPDATE's payload, not the identity; #539 added build to the identity, and build is NULL on server-side rows (the API Worker stamps no build), so build IS ?8 matched and was never what stopped the collapse. The reference in the message column was the only blocker.

(d) The docblock — I fixed the CODE

coding-failure.ts:186-192 promised phase and payload sizes survive a replay. probe.at did; probe.saw sat inside capture (the snapshot step's body) and probe.drove inside the event step's. Weakening the docblock would have left paneChars: 0 standing as "the run read an empty pane" — a wrong diagnosis, worse than a missing one. Both setters now run on the step's result, outside the step, so the value crosses the journal boundary. Proven by replaying a journalled step with a callback that asserts it was never invoked, plus a source guard that fails if either setter moves back inside.

Acceptance criteria

1. ✅ Both production strings → class: workflow_internal, retryable: true, level: warn.
2. ✅ No run error: prefix; stopReason: interrupted, unreachable from any outcome.
3. ✅ Two occurrences differing only in reference produce one row (verified through splitCfReference + the recordCodingFailure row it writes).
4. ✅ 30 tests fed the literal strings from all 8 occurrences.

The ratchet

workflows/coding-session.ts was 799/800 and has been at its pin for three lanes, so this is extraction: coding-run-report.ts (pure) and coding-decide-budget.ts (the #159 spend gate, which never needed the Workflow). 788 after; the pin is unchanged. delegation-budget.test.ts's account-ceiling guard named the workflow by hand and went red on the move — rewritten to follow the predicate (every caller of markExhausted, found rather than listed, walk size and caller list both asserted, ADR 0002 G1).

Every new assertion was proven by watching it fail (G4): deleting the classifier branch → 3 red; putting probe.saw back inside capture → 2 red; dropping splitCfReference → 1 red; breaking the ceiling predicate → 1 red.

Not done, deliberately

• The rethrow stays an open question, exactly as you framed it. recordCodingFailure, closeDelegation and the finally all have side effects that would repeat per attempt, and [bug] A CODING_SESSION run that dies logs nothing — 0 logError calls where every peer workflow has 3-5, so a failed run exists only as a chat bubble #529's comment exists because a run vanishing with no record was the worse failure. It wants its own ticket.
• Not verified in production. The change is only observable through a new occurrence of a Cloudflare internal error, which I cannot trigger without driving one of your live agents — and this environment has no PAGS API token to read /v1/errors with (~/dev/ops SOPS holds no pags.CLOUDFLARE_API_TOKEN either; the live credential is the CI repo secret). The next occurrence will file as workflow_internal/warn with a cfReference and a non-zero paneChars; that is the check to make.
• LoopRunsSection's REASON_LABEL is still missing engine_limit ([bug] The dominant live cause of dead runs: the Engine's own subscription usage limit is handled as a 15-minute human takeover, so a run dies 41 minutes before the CLI comes back #541, landed hours before this) — it renders the raw reason. Left for its own fix rather than folded in silently.

[serge-ivo]

Fixed in 4ff78f9 — CI's only red step is foreign (see the bottom), Deploy API Worker succeeded (Current Version ID: f2b57198-bff7-4e3b-873e-6a75f4b5b042), /health 200.

You asked me to verify the classification first and said the double-filing was the part most likely still live. Both were worth checking: the classification half is delivered, the double-filing is real but is not the mechanism the issue describes, and a third thing — AC3 — shipped and had never once worked in production.

The classification half is done, and measured

c64d9f5 landed workflow_internal. Over every coding:session row that exists (10 rows, 2026-08-12 09:04:55 → 2026-08-15 09:17:33 — that is the whole log, not a window I chose):

class	rows
infra_transient	4
unknown	3
workflow_internal	1
provider_stall	1
provider_error	1

unknown is 3/10, down from the 3/4 this issue was filed on — and all three are pre-c64d9f5 rows from 2026-08-12. Since the deploy on 2026-08-13 there have been 5 rows and none is unknown. AC1 and AC2 hold.

The double-filing: two runs, two different mechanisms, and only one is what the issue says

b9d9c051 is not one death recorded twice. The two rows are different measurements:

	00:25:19	00:28:27
class	infra_transient	provider_stall
phase	s9-waitidle	s21-decide
steps	2	4
elapsedMs	184,896	372,771
paneChars	4,081	33,137
level	warn	error

Phase advanced s9→s21, steps doubled, and the run kept working for 188 seconds after the isolate reset before dying of something else. That is an interruption followed by a real death — which is the correct number of records. The defect is that both rows said coding run failed, so from list_errors it read as a run that died twice.

And 03762fd (#583) made this routine rather than occasional: an infra_transient death now RESUMES on purpose. So the driver wrote "coding run failed" to the durable log and then, three lines later in the same catch block, posted "⏸ Interrupted by a platform update … nothing is needed from you" into your chat. Two durable accounts of one event, disagreeing, with the log the wrong one.

Fixed by asking the resume plan BEFORE writing the record and passing the answer in as disposition: "resumed" | "ended". The row now reads coding run interrupted (infra_transient) at <phase> after N steps, resumed: …, is pinned to warn while the platform is recovering, and carries disposition in context so a reader counting deaths gets one per run.

The reorder fixed a second thing I did not expect. The plan was computed from recordCodingFailure's return value, with .catch(() => null) at the call site — so a failed error_log INSERT landed on the !failure branch and ended a run the platform could have replayed. A durable control-flow decision was downstream of best-effort observability. agent-loop.ts:230, the other consumer of the same decision, already classified at the call site; 1 of 2 drivers was the outlier. workflows/driver-failure.test.ts now states the property over both.

AC3 shipped and has never fired — the reference is not hex

This is the finding I did not expect to make. splitCfReference matches [0-9a-f-]{6,}, and its test fed it a fabricated hex id — so the test confirmed the regex's own pattern instead of measuring anything. Cloudflare's handles are base36:

ta78s8dpekde3apmplf351m0   hknsjlipbemc1fi7lsn13vak   v5t1f9uth3ba0so067pi9qq5

Every one has a letter past f. The regex matched none of the three production rows that carry a reference. A sweep of all 124 rows in the error log finds cfReference in the context of exactly zero of them, and all three still read repeat_count: 1. The remedy shipped, the docblock described it in the present tense, and it had never once run. Fixed, and the test is now fed those three literal strings plus an assertion that the fixtures actually exercise the non-hex case.

(The collapse machinery itself works — one coding:session row carries repeat_count: 3 with both context and last_context. It was only ever this class that could not reach it.)

What is NOT fixed, and needs its own ticket

82739cb6 is a different mechanism and this change does not address it: the same step, retried by Cloudflare, failing identically twice — both s6-decide, both steps: 1, both terminal, elapsedMs differing by exactly the 8-minute wall gap. Both rows are disposition: "ended", correctly, because both attempts ended. Making the record exactly-once per run means journalling it, and the catch is deliberately outside step.do (#529) so that a replayed attempt still records at all. That trade is a design change, not a bug fix. Filed separately.

Also unaddressed and worth knowing: collapseRepeat's identity excludes context, so one row was observed collapsing occurrences from two different runs on two different instances (repeat_count: 3, context.runId ≠ last_context.runId). Row count ≠ run count in the other direction too.

Tests

Nine new assertions, each proven by watching it fail (G4):

• old hex regex restored → 3 red
• old unconditional "failed" verb + class-only level → 3 red
• pre-fix record-then-decide order restored in the workflow → 2 red

workflows/coding-session.ts is 871 lines, byte-for-byte the same count as before, so the 872 pin does not move and the closeDelegation seam is left for whoever needs it.

Verified, and not verified

Gates run from platform/: pnpm typecheck (workers/api clean), workers/api tsc -p tsconfig.test.json, pnpm vitest run (9,079 tests), biome over 1,239 files, all eight check-*.mjs, pnpm docs:drift. pnpm test:e2e skipped — the diff touches no store/, agents/coder/web or anything Playwright exercises.

Foreign failures, attributed rather than fixed: CI's File-size ratchet step went red on scripts/check-file-size.mjs grew to 1292, pinned at 1285 — introduced by 9bd2ed6 (#609), which grew that file 1284→1291 without raising its own self-pin. coding-session.ts was never flagged. That lane has since fixed it forward (self-pin now 1297). Locally, workers/mcp/** tests and store/console typecheck were also red; I reverted my four files and re-ran, and all of them reproduce without my change — they are other agents' uncommitted work in the shared tree.

Not verified in production. The change is only observable through a new platform interruption, which I will not manufacture by driving your agents. The next infra_transient on a Pilot run should file as coding run interrupted (infra_transient) … resumed at warn with context.disposition: "resumed", and the next Cloudflare internal error should carry a context.cfReference and be able to collapse. Both are cheap to check via list_errors.

[serge-ivo]

Audit verdict: delivered — closing. c64d9f5b (classification + the honest report) and 4ff78f90 (a run about to be replayed stops being filed as dead).

AC	evidence
1. both wordings classify workflow_internal, retryable, warn	lib/coding-failure.ts:222 matches WORKFLOW_INTERNAL_MARKERS; coding-failure.test.ts:285-300 feeds both production strings and asserts the class
2. no run error: prefix, and the stop reason is distinguishable from a genuine objective failure	coding-failure.test.ts:412-440 — "says INTERRUPTED, not failed, when the run was resumed past it", asserting "coding run interrupted (infra_transient)" and explicitly not.toContain("coding run failed")
3. two occurrences with different reference ids collapse into one row with repeat_count: 2	splitCfReference at coding-failure.ts:179 normalises only the reference = <id> token — the narrow normalisation the issue's own regression-risk section asked for — and lifts the id into context. Tests at :337, :348
4. a unit test feeds the exact production strings	coding-failure.test.ts:292-293 holds both verbatim, so a reworded CF message fails rather than degrading to unknown

The regression risk the issue flagged — "the class becomes both quiet and uncountable, which is worse than today" — is what splitCfReference prevents, and it is the reason the reference is preserved in context instead of dropped.

The issue's own labelled inference ("that Cloudflare retried the attempt is inferred") turned out to matter more than expected: 03762fd8 (#583) then made replay routine rather than occasional, which is why 4ff78f90 was needed on top — a run interrupted and later dying wrote two rows that read identically. Production run b9d9c051 is that shape (infra_transient 00:25:20, provider_stall 00:28:27, one runId), and the fix distinguishes them by what the platform did, which is the only party that knows.

workflow_internal is also declared non-resumable with the reason recorded at coding-failure.ts:334: "Cloudflare already retried this attempt; re-dispatching the run would repeat acts it has already committed."