A run whose engine died at step 1 reported 'running' for 4.35 hours with a fresh lastProgressAt, and wrote nothing to the trace

[serge-ivo]

A dead run reported "running" for 4.35 hours, and the only record of the truth was the pane

Measured live 2026-08-15, instance e4d2d031-833b-4ccf-8b39-8af08337075b (PAS Coder), run
70ea298e-bd0c-487e-a210-0efc2f019191, session csess_88a4de4b-045b-4814-b978-b55626f62375.

What each surface said, at the same moment

Surface	Answer
check_instance_loop	status:"running", stopReason:null, finishedAt:null, iteration:1/30
check_instance_loop.lastProgressAt	3.5 minutes ago
coding_session_capture.runState	"idle"
coding_session_capture.pane	[14:20:57] Please read GitHub issue #57… [14:20:58] You've hit your weekly limit · resets Aug 17 at 4pm [error] You've hit your weekly limit…
agent_trace source:"coding"	no event for this run; newest coding event is 2026-08-13 02:53:55

startedAt 1786767644486 → observed 1786783320834 = 4.35 hours, on iteration 1 of 30.
The engine died one second after receiving the objective.

Three separate defects

1. lastProgressAt measures liveness, not progress. It advanced to within 3.5 minutes of the
observation on a run that has not left iteration 1 since 14:20. Whatever touches it is not gated
on the run advancing, so the single field a stall detector would read is precisely the field that
conceals a stall. #579 §1 proposes distinguishing a stalled run from a clean failure; on this
evidence a stall is currently undetectable from the run record.

2. Two tools disagree about one session. check_instance_loop → running;
coding_session_capture → runState:"idle". Both read server state for the same session id.
The capture is correct.

3. The run wrote nothing to the unified trace. workflows/coding-session.ts contains
zero logEvent calls and zero logError calls:

$ grep -c logEvent workflows/coding-session.ts   → 0   (browser-task 6, job-apply 6, pipeline-run 7, agent-loop 3)
$ grep -c logError workflows/coding-session.ts   → 0   (browser-task 4, job-apply 4, pipeline-run 5, agent-loop 3)

Coding events DO reach the trace, but only via lib/coding-failure.ts:341, on a classified
failure. A run that simply stops classifies nothing and therefore records nothing. The only
durable account of this run's death is the pane, retrievable only by someone who already knows the
session id.

The same shape, three more times

Instance a1d3522f-6b91-4cd9-bf7b-380e470192b7 (AIPA coder), objective git pull — a
two-second command — three runs:

run	elapsed	iteration	outcome
82739cb6	3.15 h	1/10	failed — stuck not resolved in time
1dbb3e54	3.41 h	1/10	failed — stuck not resolved in time
ebc70ac1	0.26 h	3/10	failed — stuck not resolved in time

So a stuck-run timeout exists, but at a scale of hours, and it produces a message that names no
cause. Two of the three never left iteration 1.

Relationship to existing issues

• [bug] The dominant live cause of dead runs: the Engine's own subscription usage limit is handled as a 15-minute human takeover, so a run dies 41 minutes before the CLI comes back #541 is the same root trigger — the Engine's own subscription usage limit — handled as a
  15-minute human takeover. That handling is doubly wrong here: this limit resets on Aug 17,
  so no takeover window of any length can help, and the run should fail immediately with the cause
  named rather than wait.
• Notify proactively when an agent run finishes (webhook / event feed) - investigation #579 asked where a run terminates and what to notify on. This is the case that shows the
  question is upstream of notification: the platform does not currently know this run is over.
  Notification cannot fix a run record that says running.
• [bug] WorkflowInternalError, traced: it is not the objective failing and not terminal — one run filed its own death twice, and all 8 occurrences classify as unknown #546 (WorkflowInternalError classified unknown) and A Codex coding session is completely invisible — unmetered like a tmux drive, but unlike a tmux drive the absence is never recorded, and classifyEngineMetering has zero production callers #556 (invisible Codex session)
  are the same family: the coding driver's failure paths are the least instrumented.

Acceptance criteria

1. A run whose engine has emitted a terminal error does not remain status:"running". State where
   the transition is made and why that point is correct.
2. lastProgressAt reflects progress, or is renamed and joined by a field that does. Whichever
   is chosen, a stall must be detectable from the run record alone.
3. check_instance_loop and coding_session_capture cannot disagree about whether a session is
   working. A test drives both off one state.
4. coding-session.ts writes the trace/error events its four peer workflows write — enough to
   answer "what happened" without the pane. A guard asserts the coding driver is not an exception,
   stated over the whole workflow set per ADR 0002, not a hand-listed subset.
5. A provider limit that resets days away fails the run immediately, naming the reset time — it is
   not a takeover-shaped wait ([bug] The dominant live cause of dead runs: the Engine's own subscription usage limit is handled as a 15-minute human takeover, so a run dies 41 minutes before the CLI comes back #541).
6. Tests reproduce this exact case: engine emits a terminal provider error at step 1; assert the
   run record goes terminal, the trace carries the cause, and no surface reports running.

Verified vs inferred

• Verified: every value in the tables, read live 2026-08-15 from check_instance_loop,
  coding_session_capture and agent_trace; the elapsed arithmetic; both grep -c counts and
  their peer-file comparisons; coding-failure.ts:341 as the coding trace writer; the three AIPA
  run records.
• Inferred: what advances lastProgressAt — the field is observably fresh on a run that has
  not advanced, but the writer was not traced. AC2 should start by finding it; if it turns out
  to be a legitimate progress signal on a sub-iteration unit, this is a naming and surfacing
  problem rather than a correctness one, and should be re-rated.

[serge-ivo]

The owner's own agent recorded this 15 hours earlier, and the two records disagree in a useful way

Feedback b7b6c751-8ecd-43dd-86d0-d653ec9e8cbd (2026-08-15 02:18, instance
bd43f4de-ef35-4051-bdec-43f8571414a1, author:"agent", status:"open"):

The retry, run 5b4b813f-a67a-46be-984d-81246e5dfc7f, has sat at iteration 1 with
lastProgressAt frozen 12s after start for several minutes while still reporting status
'running'.

The run measured in this issue (70ea298e, 2026-08-15 ~17:00) showed the opposite:
lastProgressAt fresh — 3.5 minutes before observation — on a run that had not left iteration 1
for 4.35 hours.

So status:"running" is unreliable in two distinct ways, and they need different fixes:

	lastProgressAt	iteration	What is broken
Case A — feedback b7b6c751	frozen at +12s	stuck at 1	The stall signal is correct and unconsumed. Nothing reads it and transitions the run.
Case B — this issue, run 70ea298e	fresh, +4h20m	stuck at 1	The stall signal is itself wrong. Something advances it without the run advancing.

This changes AC2. Making lastProgressAt mean progress (Case B) fixes nothing on its own,
because Case A proves that even a correct frozen timestamp is read by no one. Both are needed: a
truthful field and a consumer that acts on it. An implementation that only does the first will
close this issue and leave the reported symptom intact.

Case A also gives a free reproduction that needs no provider outage — a run whose
lastProgressAt is artificially frozen must become stalled without any engine involvement.

Same feedback row, two further claims not covered here

Both are from the owner's agent and are recorded, not verified by me:

1. "the failed run's detail recorded 'Acts: deleted files recursively' even though the objective
   was investigation-only with an explicit do-not-write instruction" — a read-only objective
   producing destructive acts. Filed separately; it is a safety question, not an observability one,
   and must not be folded into this issue.
2. "nearly every MCP tool call from the external client fails once with 'has not been loaded yet'
   and succeeds on an identical retry." Also separate.

Why the feedback row matters procedurally

list_feedback had this at 02:19. The independent investigation that produced this issue ran
at ~17:00 and re-derived it from scratch. The rows carry trace_id and are explicitly designed
so that agent_trace(trace_id=…) is the next call — the triage path exists and was not walked.
Rows 5ac8b2e4, b7b6c751 and fd60c895 are all still status:"open", while every row older
than 2026-08-12 carries status:"filed" and an issue_url. The loop worked, then stopped being
run.

[serge-ivo]

The inferred claim is now traced, and the mechanism inverts this issue

The issue said "what advances lastProgressAt was not traced" and flagged AC2 as depending on
it. Traced now.

It is a deliberate heartbeat that exists to defeat the stall detector

workflows/coding-session.ts:563-571, the tick callback:

// Outside `step.do`, like the runner guard's own tick: idempotent writes whose only job is to be
// recent. `recordIteration` writes `last_progress_at`, without which `sweepStaleRuns` closes a
// parked run as dead at 3h.
tick: async () => {
  …
  await recordIteration(env, loopRunId, pilotSteps).catch(() => undefined);

recordIteration (lib/agent-loop-store.ts:171):

UPDATE agent_loop_runs SET iteration = ?2, last_progress_at = ?3 WHERE run_id = ?1

So tick refreshes last_progress_at on a timer, passing whatever pilotSteps currently is.
With pilotSteps stuck at 1, the timestamp advances and the iteration does not — exactly the
observation in this issue.

Its stated purpose is to stop the sweeper closing the run. run-sweeper.ts:31 sets
STALE_RUN_MS = 3h, and :62 closes any status='running' row whose
COALESCE(last_progress_at, started_at) is older than that. For a legitimately parked run —
waiting on a human — the heartbeat is correct and necessary. For a run whose engine is dead,
the same heartbeat makes it unsweepable in principle: run 70ea298e reached 4.35 h, well past
the 3 h cutoff, because tick kept it fresh.

That also explains the contrast with the three AIPA git pull runs in this issue, which did
terminate at 3.15 h and 3.41 h with stuck not resolved in time: those hit STALE_RUN_MS
normally. The sweeper works. The heartbeat is what suppresses it.

The platform already knows this and wrote it down — on a different surface

lib/work-report.ts:84-88:

What it deliberately does NOT claim: that the ENGINE is working. lastProgressAt is the
orchestrator's last recorded iteration, not the engine's last output — the live runState lives
behind /capture … Asserting "engine: working" from this column would replace a false stall with
a false all-clear, which is the same defect facing the other way.

That is precisely correct, and it is why isStalled is careful. None of that caveat reaches the
MCP surface. check_instance_loop returns status:"running" and a fresh lastProgressAt with no
indication that neither field is a statement about the engine, and the only field that is one —
runState from /capture — lives on a different tool that needs a session id.

What this changes

AC2 as written is wrong. "Make lastProgressAt reflect progress" would break the parked-run
case the heartbeat was built for, and re-arm the 3 h sweeper against runs legitimately waiting on a
human. The two signals are genuinely different and both are needed:

• liveness — the orchestrator is still ticking (today's last_progress_at, correct as-is)
• progress — the run advanced an iteration (today: conflated into the same column by
  recordIteration writing both fields in one statement)
• engine state — runState from /capture, the only one that answers "is the CLI alive"

Revised acceptance criteria, replacing AC2:

1. recordIteration stops conflating the two. A tick that does not advance the iteration updates
   liveness only; progress carries its own timestamp. The parked-run case must keep working — a
   test asserts a parked run is still not swept at 3 h.
2. A run is classified stalled when liveness is fresh but progress is stale — the case this
   issue reports, which no current field can express.
3. check_instance_loop carries the engine's runState, or states in its own description that it
   does not speak for the engine and names the tool that does. work-report.ts:84-88's reasoning
   is the reference; the defect is that it is confined to one surface.
4. Keep the original AC3 (the two tools must not disagree) — this is the mechanism behind that
   disagreement, not a separate bug.

Verified vs inferred, updated

• Verified now: the tick call site and its comment; recordIteration's single SQL statement
  writing both columns; STALE_RUN_MS = 3h and the sweeper's COALESCE predicate;
  work-report.ts:84-88 verbatim.
• No longer inferred: what advances lastProgressAt.
• Still inferred: that pilotSteps was genuinely 1 rather than the engine having advanced
  without the counter following. iteration:1 is what the row says, but the counter's own writer
  is tick, so a counter bug would look identical. Worth one check during implementation, not a
  blocker — either way the liveness/progress conflation is real and is the fix.

[serge-ivo]

Landed, and the run this issue is about was killed by the fix ~4 minutes after the deploy

Commits fd1c323, 67c41b7, 03762fd (CI green incl. test:e2e; migration 0127 applied to
production D1 at 09:24:40 UTC).

Verified in production, on run 70ea298e itself

The run was still alive when the deploy landed. Read live from agent_trace afterwards:

09:29:45 UTC  coding.run.end  trace_id=70ea298e-…
  "waiting — The coding CLI is still reporting its own usage limit and the limit does not reset
   until 2026-08-17 16:00 +10:00 (Australia/Melbourne), which is beyond the 6 hours a run may
   wait — no amount of waiting inside this run can clear it."
  context: {outcome:"waiting", stopReason:"engine_limit", steps:1, repo:"pas/platform"}

Three of the ACs, observable in one row: the run left running (AC1), the reset instant is NAMED
in the owner's zone rather than waited out (AC5), and coding-session.ts wrote a lifecycle event to
the unified trace for the first time (AC4 — that file's logEvent count was 0).

And the conflation, gone, on the same row:

lastProgressAt  09:24:47   ← last real advance, before the deploy
lastAliveAt     09:29:47   ← +299.6s, written by the terminal recordIteration
iteration       1

Under the old statement that same write would have dragged lastProgressAt to 09:29:47 and the run
would have looked busy again. Older rows carry lastAliveAt: null and fall through the COALESCE
chain unchanged — compatibility observed, not just asserted.

AC2 was implemented against its revision, with one deliberate departure

Migration 0127 splits the two columns and adds a third fact. recordIteration moves
last_progress_at only in a CASE WHEN ?2 > iteration — the rule is in the STATEMENT, not in the
callers, because splitting the callers would be enough for today and worth nothing the next time
somebody adds a heartbeat. The sweeper reads COALESCE(last_alive_at, last_progress_at, started_at),
so the parked-run protection survives; a test puts a real row four hours past the 3h cutoff with a
beating heartbeat and asserts it is not swept.

I did not implement "stalled = liveness fresh + progress stale", and this is the one thing to
push back on if you disagree. That signature is also produced by the two healthiest things a
coding run does — one long engine turn, and a deliberate park. Shipping it would re-arm #459 (the
Lead that read "step 3/50 after 9 minutes" as a stuck progress bar and told you there was "nothing I
can do" while the engine was mid-edit). So the verdict reads LIVENESS only, and progress-staleness
is printed as a fact ("this instruction has been running 4h") rather than as a diagnosis. It also
fixed a false stall that already existed: with only a per-instruction column, a healthy 20-minute
step crossed the 15-minute line and read as dead.

runHealth is three-valued — working / waiting / stalled — because the honest description of
this run was never one of the two words the record had.

The mechanism was sharper than the comment

The heartbeat was not concealing a stall. It was concealing a park: the engine reported a reset
two days out, and planEngineWait clamped that to the 6h ceiling and slept on it. That park could
never have helped — ENGINE_WAIT_TOTAL_MS is the same 6h, so on waking the run would have been
refused for having spent its budget. The outcome was fixed the moment the deadline was parsed; the
six hours bought a zombie run holding that repo's one active session. Hence 67c41b7.

Left open

• AC3 (check_instance_loop carries runState) — workers/mcp/** is another agent's lane; not
  touched. The API side now serves lastAliveAt / waitingUntil / waitingReason /
  interruptions on GET /v1/instances/:id/loop[/:runId], and runHealth + waitClause are
  exported from work-report.ts for whoever wires that surface.
• AC6, partially. Covered: the park refusal, the liveness/progress split, the sweeper, the
  parked-run regression, the trace. NOT covered by a test: an engine emitting a terminal provider
  error at step 1 that is not a usage limit — that path runs through runCodingLoop's decide,
  which needs a Workflow to drive. Named rather than claimed.
• The park columns have not yet carried a value in production, because no run has parked since
  the deploy. Unit-tested; not yet observed live.

[serge-ivo]

AC3 delivered — da4bcec (descriptions) and f01d5b4 (the verdict on the wire)

Scoped to AC3 only; AC1/AC2/AC4/AC5/AC6 were the API lane's (fd1c323, 67c41b7, 03762fd).

What AC3 turned out to need, after fd1c323

fd1c323 split the fact into three fields that can no longer contradict each other. What it did
not do is send the reading of them anywhere but the agent-facing work report — so
check_instance_loop still answered with status:"running", two timestamps and an iteration
counter, and left the caller to judge.

That derivation is the defect. work-report.ts:136-141 records a model reading "step 3/50 after
9 minutes" as a stall and telling the owner there was "nothing I can do" while the engine was
mid-edit — and the intervention that invites (restart the session, kill the engine) destroys work
that was progressing normally.

So GET /v1/instances/:id/loop and …/loop/:runId now carry health
(working | waiting | stalled) and waitNote, computed by the exported
runHealth/waitClause. Imported, not re-implemented — two surfaces answering "is this
healthy" independently is what this issue documents, so the verdict is computed once by the module
that defines it. Both routes, because a verdict on the detail read alone would leave the LIST — what
a caller with no run id reads — still shipping bare counters, and the original AC3 is that two
surfaces must not disagree about one run. The raw columns stay alongside: the verdict is the answer,
the fields are the evidence.

The inference I did not re-introduce

Your guidance was explicit and it is pinned by a test: health reads liveness only. A fresh
heartbeat beside a stale lastProgressAt is equally a long engine turn, a legitimate park and a
genuine stall, so "liveness fresh + progress stale ⇒ stalled" is an inference this surface must not
make. A run with a 20-second heartbeat and a 50-minute-old advance reports working, and its
lastProgressAt is reported plainly beside it — a fact, never a diagnosis.

Every expectation in those tests is computed by calling the shared runHealth/waitClause
rather than written by hand, so the assertion is that the route quotes the platform's verdict rather
than growing a second one that can drift. The clock is frozen, or waitClause's "expected to resume
in …" would be evaluated at two different instants. Watched failing before being trusted: all five
go red with the decorator removed.

The engine half — the disjunct AC3 allows

check_instance_loop and coding_loop_status now state that none of these fields speaks for the
ENGINE, and name the tools that do (coding_timeline, coding_session_capture).
work-report.ts:146 was the reference wording.

Description rather than a joined runState field, deliberately: both tools are generic and
serve apply, pipeline and chat agents that have no engine at all, so fetching one would make every
caller pay a runner round trip for a field null for most of them. coding_timeline (#581, shipped)
carries the engine's real runState for the callers who have one.

Both tools were changed, not just the one named in AC3 — they call the same endpoint, so a
caveat on only one is a caveat a caller misses by picking the other name.

Not mine, left open

AC1, AC2, AC4, AC5, AC6 belong to the API lane's commits; I did not verify them and make no claim
about them here. Leaving this issue open for whoever owns that confirmation.