fix(watchdog): stop false-orphaning executions parked before the agent spawns them (abilityai/trinity#2433)

[webmixgamer]

Summary

• The cleanup watchdog false-failed executions that were admitted but parked — in the backend's global agent-call queue (agent_call_limiter), in the agent's CPU-sized default thread pool, behind the agent /api/chat lock, or in the post-exit drain before unregister() — because its proof-of-life (GET agent/api/executions/running) could not see any of those. It wrote failed ("completed on agent but status not reported"), released the slot, and the parked call then ran anyway: billed, overbooked, its late 200 silently overwriting the row (bug: Execution shows Failed then Success after cleanup (race condition) #378).
• Orphan now = the agent does not know it AND no live backend dispatcher owns it. Agent side: /api/executions/running gains pending_ids (accepted at /api/task, /api/chat and the refactor: fire-and-forget dispatch — a hung turn holds zero backend resource #1083 async spawn but not yet spawned) and recently_completed_ids covers exited-but-registered handles. Backend side: every outbound agent call is registered for its whole lifetime (track_inflight_dispatch, in-process registry + a cross-worker Redis liveness marker execution:inflight:{id}, 60s TTL / 15s refresh, ONE refresher task per process); the watchdog reads a tri-state verdict (alive / absent / unknown) and withholds recovery accordingly (CleanupReport.dispatch_inflight_skipped).
• A park no longer spends the run's budget: at grant, a park ≥ 5s re-stamps started_at (admission kept in queued_at, the drained-backlog shape) and renews the slot lease, so the registry-blind Phase-1 sweep, the slot TTL, canary E-01 and duration_ms all measure the run, not the wait.
• Parked rows are cancellable — and agent-scoped: terminate consults the in-process registry, then the cross-worker marker; a parked phase is finalized CANCELLED and the grant raises BackendAgentCallCancelled, where the dispatcher writes CANCELLED itself (never FAILED). Cancel-while-pending on the agent is consumed by register() (SIGKILL at spawn), closing the check-then-Popen window.
• Gemini runtime now registers its subprocess (it never did — every Gemini run longer than the grace was false-orphaned); headless runs use a dedicated 32-thread pool pinned to MAX_PARALLEL_TASKS_CEILING_MAX.
• Packaging: BACKEND_AGENT_CALL_LIMIT / BACKEND_AGENT_CALL_QUEUE_TIMEOUT_S forwarded in prod + hosted compose and documented in .env.example (they lived only in docker-compose.yml, the #1039 class); the >5s queue-wait warning fires on both acquire branches.
• Security (from /cso --diff, fixed here): terminate_execution wrote CANCELLED keyed on the caller-supplied task_execution_id while only execution_id was scoped by the agent's 404 — a caller authorised on agent A could flip agent B's running row. One agent-scope gate at the function entry now covers all three arms (uniform 404, fail-closed 503). Report: docs/security-reports/cso-diff-2026-08-28-2433-watchdog-parked-executions.md.

Changes

• docker/base-image/agent_server/: services/process_registry.py (pending registry, cancel-consumed-at-register, exited-but-registered ids), routers/chat.py (/api/task, /api/chat, /api/executions/running), services/result_callback.py, services/headless_executor.py (_HEADLESS_EXECUTOR, pre-spawn 409), services/gemini_runtime.py (registers)
• src/backend/services/: agent_call_limiter.py (in-flight registry + marker refresher + cancel + on_granted), task_execution_service.py (whole-call tracking, re-stamp + slot renew at grant, CANCELLED terminal), chat_execution_service.py (entry agent-scope gate, parked-cancel arm, 409/503 split), cleanup_service.py (tri-state skip, honest orphan string, dispatch_inflight_skipped), slot_service.py (renew_slot), agent_runtime_state.py (exempt-by-construction note)
• src/backend/db/schedules/executions.py + database.py: restamp_execution_dispatch (CAS on RUNNING + NULL lease)
• docker-compose.prod.yml, docker-compose.hosted.yml, .env.example
• Docs: architecture.md (new "In-Flight Dispatch Proof-of-Life (bug: watchdog fails admitted-but-undispatched executions as "completed on agent but status not reported", releasing their slots and masking the real terminal #2433)" block + Cleanup/Redis rows), requirements/infrastructure.md, feature flows (cleanup-service, task-execution-service, capacity-management, parallel-headless-execution, chat-turn-cancellation), learnings.md (2 entries), security report
• Tests: 10 new tests/unit/test_2433_*.py files (registry, task handler, headless pool, Gemini, limiter, slot renew, DB restamp, dispatch wiring incl. the exploit replay, watchdog verdicts, packaging parity); 6 existing files adjusted (mock hygiene, _EXPECTED_UPDATE_SITES)

Test Plan

• Full unit suite under CI conditions (branch on a clean origin/dev worktree, no submodules): 12969 passed, 0 failed (baseline origin/dev: 12863 passed, 0 failed)
• python tests/lint_sys_modules.py — no new violations; secret scan on the diff clean
• Repro A (backend-queue parking: 10 async tasks over 3 agents, global cap 8): 10/10 success — 2 parked 485s, withheld by the watchdog at both 5-min cycles, re-anchored at dispatch (started_at restamped, slot lease renewed), duration_ms ≈ the 486s run; 0 orphan recoveries, 0 lost CAS, 0 active slots after
• Repro B (8 tasks on one agent, 2-CPU pin to recreate the old 6-thread default-pool bottleneck): 8/8 success — 5 parked (485s and 972s waves), every cycle withheld, every park re-anchored; 0 orphan recoveries, 0 lost CAS
• Live pending_ids probe: two concurrent /api/chat turns on one agent — the second reported as pending while waiting on the chat lock, then running, with the first in recently_completed_ids
• Rebuilt base image required for the agent-side half; the backend half alone already covers old images through the whole-call marker

Fixes #2433

Generated with Claude Code

[dolho]

Review — #2435 (fix #2433)

Read the whole diff, built a worktree off cfc2cfef, and ran the suites locally.

Verification I did

• pytest tests/unit/test_2433_*.py → 106 passed
• pytest tests/unit -k "cleanup or watchdog or slot or limiter or terminate or 679 or 921 or 1332 or 1094 or 2127 or schedule_status or 226 or 1804" → 534 passed, 2 skipped
• Verified the duration_ms claim end-to-end: it is computed in update_execution_status from the DB started_at (db/schedules/executions.py:439), not from the in-coroutine start_time (task_execution_service.py:1164, which is stamped before capacity acquire and still spans the park). So the restamp genuinely fixes duration_ms — the PR body is right, and the reason is worth keeping in the docstring because the local execution_time_ms still measures the park.
• Redis ACL: both ~*, so the new execution:* keyspace is not blocked. agent_runtime_state exemption note is correct — the parity test only greps agent:*.

Overall: the diagnosis is right and the two-sided design (pending_ids on the agent + a live-dispatcher marker on the backend) is the correct shape — proof-of-life had exactly one half. The tri-state alive/absent/unknown and the "a read that could not be asked ≠ a read that said no" rule are applied consistently, and the terminate_execution agent-scope gate is a real fix on a real hole. Three things below I'd want addressed before merge; the first is a new race, the second turns a pre-existing leak into a permanent one.

---

1. Cross-worker cancel reads a marker phase that is up to 15s stale → cancel-without-terminate on a running turn

acquire_agent_call_slot flips entry.phase = "calling" in memory at grant, but nothing writes the marker at that transition — the payload is only rewritten by _refresher_loop on its next INFLIGHT_TICK_SECONDS (15s) tick. So for up to 15s after a park→dispatch, execution:inflight:{id} still says "phase": "parked".

_cancel_inflight_if_parked trusts that value:

phase = agent_call_limiter.cancel_inflight(eid, agent_name=name)     # in-process: accurate
if phase is None:
    phase = await agent_call_limiter.request_cross_worker_cancel(...)  # marker: up to 15s stale
if phase != "parked":
    return None

With --workers 2 (prod), roughly half of cancels land on the worker that does not own the coroutine, so they take the marker path. If the cancel arrives in that 15s window:

• terminate_execution returns cancelled_while_parked and never reaches the proxy arm — the agent is never asked to terminate;
• the row is written CANCELLED and the capacity slot is released via release_if_matches;
• the other worker is already inside client.post(...) — its grant-time cancel check has already passed — so the agent runs the full turn, billed, with its slot given away (overbooking);
• when the 200 comes back, the SUCCESS write loses the CAS to the standing CANCELLED, so the work is silently discarded.

That is the #378 symptom this PR exists to remove, reproduced in a narrower window. The in-process arm is fine — cancel_inflight reads the live entry.phase; it is only the marker arm that can lie.

Cheapest fix: make the parked→calling flip write the marker synchronously (one SET in a to_thread at grant, alongside the existing _cancel_requested_cross_worker_sync round-trip you already pay for on a ≥5s park), so the marker can only ever be stale in the safe direction. Belt-and-braces alternative: on the cross-worker path, set the cancel key and then fall through to the proxy arm regardless of reported phase — a genuinely parked execution 404s on the agent, so falling through costs nothing and the grant-time check still catches the true park.

2. list_recently_completed_ids now reports exited-but-registered ids with no age bound

The new union is right in principle — an exited process whose owner hasn't reached finally: unregister() is semantically "completed, not yet reported". But _recently_completed has RECENTLY_COMPLETED_TTL_SECONDS = 300, and this new set has no clock at all:

for eid, entry in self._processes.items():
    if entry["process"].poll() is not None:
        ids.add(eid)

PoC against this branch:

running ids: []
recently_completed (t=0): ['leaked-eid']
recently_completed after clearing the 300s TTL buffer: ['leaked-eid']

A registry entry that leaks — an exception between register() and the finally: unregister() — is now reported to the backend as agent-known forever, so the orphan watchdog never recovers that row. Before this change the same leak was self-healing: list_running() filters on poll() is None, so the row was still orphan-eligible. The registry-blind Phase-1 stale sweep is still the backstop, so this is not unbounded — but it means a leak now costs the full 120-min sweep with a fabricated duration_ms instead of one watchdog cycle.

And the PR makes that leak newly reachable rather than theoretical. In both gemini_runtime.execute() and execute_headless() the new register() sits before process.stdin.write(prompt), while the finally: unregister() only wraps the run_in_executor further down:

get_process_registry().register(_registered_id, process, metadata={...})

process.stdin.write(prompt)     # <-- exception here leaks the entry permanently
process.stdin.close()

The new kill-at-spawn path makes an exception there likely: register() SIGKILLs the group, control returns, and the very next statement writes to a pipe whose reader is gone → BrokenPipeError. claude_code.py has the same register-before-stdin-write shape (pre-existing, its try/finally starts after the write), so it inherits the same newly-permanent consequence.

Two small changes cover it: bound the exited-but-registered set by entry["started_at"] against RECENTLY_COMPLETED_TTL_SECONDS (the entry already carries the timestamp), and move Gemini's register() inside the block the finally: unregister() guards.

3. restamp_execution_dispatch runs sync sqlite on the event loop

In _on_dispatch_granted:

restamped = db.restamp_execution_dispatch(execution_id)          # sync DB, on the loop
...
renewed = await asyncio.to_thread(get_slot_service().renew_slot, ...)   # correctly off-loop

agent_call_limiter's own module docstring exists because sync sqlite inside an async coroutine stalls the loop, and this write happens while both semaphores are held, at the moment the queue is by definition congested. The line immediately below already does the right thing. Wrap it the same way.

---

Smaller things

• /api/chat pending entries expire well before a chat can (chat.py) — execute_task passes timeout_seconds=request.timeout_seconds or 900, but the chat handler passes none, so the entry gets the 900s default → a 960s deadline. ChatRequest carries no timeout_seconds field at all (models.py:19-24), and a chat waiting on the execution lock behind a long turn can legitimately wait up to the agent's execution_timeout_seconds (7200s ceiling). The entry is then evicted mid-wait and pending_ids stops covering it. The backend in-flight marker covers this case today, so it is not a live regression — but the defence-in-depth layer silently stops defending, which is worth a comment at minimum and a plumbed timeout ideally.
• /api/chat finally: discard_pending is inside the lock, register_pending is outside it — a request cancelled while waiting on get_execution_lock() (client disconnect) never discards. Backstopped by the deadline; still, moving register_pending under the same try would make the pairing structural.
• _process_stale_slot_reclaims does one _inflight_verdict_map([execution_id]) per row inside the loop — i.e. one Redis MGET round-trip per candidate — while _reconcile_orphaned_executions deliberately batches into one. Under a backlog those are the same cycle. Worth batching for symmetry.
• register_pending logs at INFO unconditionally, so every /api/task and /api/chat now emits an extra line into Vector. DEBUG seems right for the happy path; the eviction WARNING is the line that matters.
• renew_slot moves the ZSET score before re-EXPIREing the metadata hash. If the hash has already expired, zadd XX still succeeds and the function returns True while expire is a no-op — leaving exactly the ZSET-without-hash state canary S-03 reports as missing. Not introduced here (the hash can expire first today), but renew_slot can now perpetuate it and reports success. A hset-if-missing, or returning False when the hash is gone, would keep the return value honest.
• prod/hosted run maxmemory-policy allkeys-lru, so an in-flight marker can be evicted before its TTL under memory pressure → absent → false orphan. Same exposure the slot ZSETs already have, so not a blocker; worth one line in the architecture block so the next reader knows the marker is not eviction-proof.

Things I checked and am happy with

• terminate_execution's entry gate is correctly placed above all three arms, fails closed on an unreadable row (503), uniform-404s a foreign row, and covers getattr(..., None) != name so a row object missing the attribute also refuses. All three routers (chat.py, public.py, client_portal/service.py) funnel through it, so the fix is complete, and test_proxy_arm_cannot_flip_a_foreign_row_via_task_execution_id replays the actual exploit.
• restamp_execution_dispatch's func.coalesce(queued_at, started_at) in the SET clause reads the pre-update row on both dialects — correct, and the CAS on RUNNING + lease_expires_at IS NULL correctly leaves pull-mode rows to the lease reaper.
• _inflight_verdict_map's "anything that is not a dict of known verdicts collapses to absent" guard, and the eager module-level import with the stub-leak rationale, are the right lesson applied — a leaked sys.modules MagicMock here would have silently disabled the whole watchdog.
• BackendAgentCallCancelled as a subclass so every existing except BackendAgentCallBudgetExhausted keeps working, with the terminal branched to CANCELLED — good, and emit_task_terminal_event already maps every non-SUCCESS terminal to agent.task.failed with the precise status in the payload, so no event-vocabulary change is needed.
• register_inflight doing _PENDING_DELETES.discard(execution_id) — that is what keeps the #678/SUB-003 same-execution-id retries from having their fresh marker deleted by the previous attempt's queued delete. Easy to miss, correctly handled.
• _get_client(use_negative_cache=False) on the watchdog read only, with the reason stated — right call; the negative cache exists for the 15s refresher, not for a 5-min sweep.
• Packaging parity (prod + hosted + .env.example) with a test pinning it — the #1039 class closed properly rather than only in the file that was noticed.

Nice work on the write-up and the repro evidence; the reasoning in the docstrings is genuinely the useful kind. The stale-marker race (1) is the one I'd insist on before merge.

[webmixgamer]

Thanks — the verification you did (worktree off cfc2cfef, the duration_ms trace to db/schedules/executions.py:439, the Redis ACL check) made every finding actionable without re-deriving it. All three blocking items and all six smaller ones are addressed in da1c0b07.

1. Stale marker phase → cancel-without-terminate — fixed, and closed rather than narrowed

Confirmed exactly as you traced it. I took the cheap fix you suggested but ordered both sides, because the synchronous write alone still leaves a window: if the remote reads the marker before the owner's write and then writes the cancel key after the owner's read, the owner dispatches and the remote finalizes CANCELLED anyway. It is microseconds instead of 15s, but it is the same bug.

The fix is store-then-load on both sides:

owner : W(marker=calling) -> R(cancel)    _publish_calling_and_check_cancel_sync
remote: W(cancel)         -> R(marker)    _set_cancel_then_reread_phase_sync

If the remote observes parked, then R_remote(marker) < W_owner(marker), and since W_remote(cancel) < R_remote(marker) and W_owner(marker) < R_owner(cancel), the chain gives W_remote(cancel) < R_owner(cancel) — the grant is guaranteed to see the key and raise before any POST. Both are single pipelines, so neither side pays a round-trip it wasn't already paying (the owner's replaces the bare cancel read).

Two details worth flagging:

• The publish gates on the ENTRY's age, not this attempt's park. track_inflight_dispatch wraps the whole retry loop, so a Async chat_with_agent: long execution silently fails with null response (reader-thread) #678/SUB-003 retry can grant in ~0s under a marker a tick left saying parked — gating on waited_s would have left that path exposed. entry_age >= INFLIGHT_MARKER_GRACE_SECONDS is _tick's own filter, so it publishes exactly when a marker can exist, and the fast-acquire hot path still never touches Redis (pinned by a test).
• The scope check stays on the remote's FIRST read. Setting the key before verifying the agent would have let a caller authorised on agent A poison agent B's cancel key — reintroducing the class the entry gate closed. So the remote reads (scope), writes, re-reads (phase).

I did not take the fall-through-to-proxy alternative: for a genuine park the agent 404s, so the user gets Execution not found in agent for a cancel that will in fact succeed at grant. The ordering fix keeps cancelled_while_parked honest.

2. Unbounded exited-but-registered set — fixed, plus the source of the leak

Bounded by the same RECENTLY_COMPLETED_TTL_SECONDS, but measured from when the exit was first observed, not started_at: a started_at bound would drop a legitimately long turn the moment it entered its drain, which is the hole the widening exists to close. list_recently_completed_ids is the only reader, so stamping there is sufficient, and a first observation that lags the true exit only extends the window conservatively.

You were right that this PR made the leak reachable rather than theoretical, so I also closed it at source: stdin.write is now paired with unregister() on failure in both Gemini sites and in claude_code.py (same shape, same newly-permanent consequence). There's an AST guard so a fourth runtime can't ship the unpaired shape. codex_runtime.py writes no prompt to stdin, so it isn't exposed.

3. Sync sqlite on the event loop — fixed

asyncio.to_thread, like the renew_slot line below it.

Smaller things

• /api/chat pending window — plumbed rather than commented: PENDING_CHAT_TIMEOUT_SECONDS = 7200 (the agent-timeout ceiling), since ChatRequest genuinely has no timeout field to thread through.
• register_pending / discard_pending pairing — made structural: the finally now wraps the lock acquisition, so a client disconnect during the wait can't leak an entry. Costs a re-indent of the handler body; git diff --ignore-all-space shows only the intended lines.
• Phase 3 batching — one _inflight_verdict_map for every reclaimed id before the loop, matching Phase 0.
• register_pending log level — DEBUG.
• renew_slot — reads the metadata hash first and refuses (score untouched) when it is gone, so it neither reports a renewal it didn't perform nor re-anchors the S-03 missing state. The default-TTL fallback still applies to a hash that is present with an unreadable timeout_seconds.
• allkeys-lru — documented in the architecture block: the marker is not eviction-proof, blast radius is one row, and the agent-side half is unaffected.

Verification

• tests/unit/test_2433_review_fixes.py — 15 tests; 11 fail against cfc2cfef (checked in a worktree), so they're real regression tests rather than restatements. The 4 that pass there are controls: scope-before-write, hot path stays Redis-free, fail-soft at grant, and the long-turn-in-drain case that must keep passing.
• Full unit suite under CI conditions (branch patch on a clean origin/dev worktree, no submodules): 12985 passed, 0 failed.
• origin/dev has moved 3 commits (ent#451) since the merge-base; test-merged clean, only architecture.md auto-merges, no overlapping code files.

Docs updated in the same commit: architecture.md plus the five affected flow docs, and two learnings.md entries — one for the cached-state/remote-decision class, one for "widening a liveness set without a clock un-heals a leak".

[dolho]

Re-review — da1c0b07

Built a worktree off the new head, and a second one off cfc2cfef to check the claims against the code they replace rather than against their own description.

All three blockers are closed, and the reply's claims hold where I checked them.

Claim	Verified
"11 of 15 fail against cfc2cfef"	Ran the new file in a cfc2cfef worktree: 11 failed, 4 passed — exact. Real regression tests, not restatements
Ordering closes the race, not narrows it	_marker_payload reads entry.phase, and the flip to "calling" precedes the pipeline, so the published value is genuinely calling
"gates on the ENTRY's age, _tick's own filter"	_tick: (now - e.registered_at) >= INFLIGHT_MARKER_GRACE_SECONDS; grant: entry_age >= INFLIGHT_MARKER_GRACE_SECONDS. Same constant, and both on time.monotonic() — the equivalence is real, not approximate
Exit bound measured from first observation	Stamped under self._lock, and register() builds a fresh dict so a re-registration cannot inherit a stale exited_seen_at
Codex not exposed	stdin=subprocess.DEVNULL, no write
"--ignore-all-space shows only the intended lines"	156 changed lines → 13 insertions, 5 deletions. Exactly the re-indent claimed
restamp off the loop	await asyncio.to_thread(db.restamp_execution_dispatch, …)

The store-then-load argument is right, and I like that the scope check stays on the first read — writing before verifying would have let a caller authorised on agent A poison agent B's cancel key, which is a fresh hole in the fix for a different one.

Two things left. Neither blocks; the first is a claim that is wider than the code.

---

1. The AST guard enumerates two files — it does not discover, so a fourth runtime can ship the unpaired shape

"There's an AST guard so a fourth runtime can't ship the unpaired shape."

_stdin_write_is_guarded is sound, but its caller is a hardcoded list:

@pytest.mark.parametrize("rel", [
    "docker/base-image/agent_server/services/gemini_runtime.py",
    "docker/base-image/agent_server/services/claude_code.py",
])

Proven — dropped a mistral_runtime.py into agent_server/services/ with register() followed by a bare process.stdin.write(prompt):

2 passed, 13 deselected

The guard did not notice. That is the class this repo keeps re-learning: #1677's caller-parity guard walks every call site for exactly this reason, and Invariant #5 records "a guard that walks only one of the two trees is not a guard". Here it walks two of N.

Fix is small — glob agent_server/services/*_runtime.py plus claude_code.py and assert over what it finds. One adjustment when you do: _stdin_write_is_guarded returns total > 0 and guarded == total, so a discovered file with no stdin write would read as a failure; that needs to become "vacuously true when total == 0".

The three real sites are correctly paired (except BaseException: unregister(); raise — and BaseException is the right width here).

2. The publish half fails soft in the unsafe direction, and the docstring only describes the other half

"Fails soft — a Redis error reads as 'no cancel', exactly as the bare read it replaces did."

True of the read. The write is new, and its failure has a consequence the bare read never had:

except Exception as e:
    _note_redis_failure(e)
    return False          # -> `cancelled` stays False -> the POST proceeds

On a transient pipeline error the marker is never republished, so it keeps saying parked — and if the remote worker's own pipeline succeeds where the owner's failed, the remote reads parked, finalizes CANCELLED, releases the slot, and the owner POSTs anyway. That is the original symptom, on the Redis-error path.

Narrow, and worth saying why: a hard unavailability is safe by construction — _get_client() is None in the owning process means its refresher never wrote a marker either, so the remote's first read returns None and routes through the agent. It needs a per-connection transient failure on one worker while the other is healthy.

Not asking for a mechanism — a retry cannot close it either. Asking for the docstring to say it, because the reply is otherwise scrupulous about exactly this shape ("true of the path it tested and false of the other one"), and a future reader will take "fails soft" as covering both halves.

---

Also verified

• renew_slot reads the metadata hash first and refuses when it is gone — so it neither reports a renewal it did not perform nor re-anchors S-03's missing state, and it deliberately does not rebuild the hash (its timeout_seconds is unknowable from there). Correct on both counts.
• PENDING_CHAT_TIMEOUT_SECONDS = 7200 is plumbed through to the register_pending call, not just defined.
• The finally now wraps the lock acquisition, so a client disconnect during the wait cannot leak a pending entry — structural, as claimed.
• da1c0b07 still test-merges clean against current dev (3 commits ahead; architecture.md the only overlap).
• tests/unit/test_2433_*.py → 122 passed on this head; the wider blast radius (cleanup / watchdog / slot / limiter / terminate / 679 / 921 / 1332 / 1804 / process_registry / schedule_status) → 554 passed, 2 skipped.

Recommendation

Approve once #1 is addressed — it is a five-line change to the guard, and the guard is the only thing standing between this fix and its own recurrence. #2 is a docstring sentence. Everything substantive in the three blockers is genuinely closed, and closed with tests that fail against the commit they fix.

[vybe]

Validated: root-caused watchdog false-orphaning fix (#2433) with extensive per-surface tests (14 new test files incl. cross-worker cancel race and slot-renew coverage); pg-migrations green; full pytest matrix green. Merged origin/dev to resolve the learnings.md append conflict (kept both sides).

[webmixgamer]

Heads-up: this merged (11:52 UTC) while I was still working through the re-review, so items 1 and 2 from your re-review did not land here. dev currently carries the enumerating guard — the one your mistral_runtime.py PoC walks straight past.

Both are in #2450 (issue #2448), plus the duration_ms docstring note from your first review's verification section.

Two things worth flagging from doing it, since one changes the shape of the fix you suggested:

A *_runtime.py glob would not have been enough. It misses claude_code.py and headless_executor.py — and headless_executor is a real fourth site the enumeration never counted. It is safe, but by a different shape: _run_headless_subprocess registers and writes stdin with no local guard, yet it is handed to run_in_executor inside execute_headless_task's try whose finally unregisters. So the guard walks every agent_server/**/*.py that drives the process registry and models both shapes (local, caller-paired), with a companion test pinning which shape covers each site — caller-pairing is name-based, so a name collision could otherwise mask a real offender.

Your vacuity note was load-bearing, exactly as you said: total > 0 and guarded == total would have failed codex_runtime.py the moment enumeration stopped. It returns an offender list now, and the discovery floor is pinned so a broken walk fails loudly rather than passing on an empty set.

On residual #2 — the docstring now says the fail-soft applies to the read half only, that the write half's failure leaves the marker saying parked (so a remote with a healthy connection can still finalize CANCELLED under a live POST), that it is bounded by the 30s negative cache plus one tick, that a hard outage is safe by construction, and why failing closed was rejected. Same qualification added to architecture.md and chat-turn-cancellation.md, which had both inherited the too-wide "closed" claim from my reply.

Thanks for catching the guard — "the guard is the only thing standing between this fix and its own recurrence" was the right call, and the fourth site is the proof.