Notify proactively when an agent run finishes (webhook / event feed) - investigation

[serge-ivo]

Problem

When an autonomous agent run (coding loop, browser task, delegated goal) finishes, there is no way for an external client to find out. The only way to learn the outcome is to poll coding_loop_status / check_instance_loop on demand. Nothing is pushed.

This is painful in practice. A run started from an MCP client (Claude, Codex, an IDE) may take many minutes. The human either sits there asking "is it done yet" or walks away and forgets. There is also no signal for the failure cases that matter most: a run that dies with a provider error, or one that stalls with a frozen lastProgressAt while still reporting running.

Related: we already have an observability gap on in-flight runs (no per-step trace queryable from outside). This ticket is about the terminal event - telling someone the run ended.

Investigation

This is an investigation ticket. Work out what it would take, do not build it yet.

1. Establish where a run's lifecycle actually terminates server-side - the point where status becomes done/failed/cancelled, or where a stall would be detected. Cite file paths and line numbers. Is there a single choke point, or several per driver (coding, browser, delegation)?
2. Determine what notification machinery already exists in the platform that could be reused - existing webhook or outbound HTTP infrastructure, the trigger/event system (create_instance_trigger, list_instance_trigger_events), the email path, any push/queue layer. Prefer extending something over inventing a new subsystem.
3. Evaluate the delivery options and recommend one or more:
   • user-registered outbound webhook (URL per instance or per account) fired on run completion
   • an event feed an MCP client can long-poll or subscribe to, so a client can await completion without busy-polling
   • email or existing notification channel as a fallback for humans
4. Define the payload: instance id, run id, driver, terminal status, stop reason, iteration count, spend, a short summary, and links to anything the run produced (commits, issue comments, PRs).
5. Cover the operational questions: retries and backoff on delivery failure, at-least-once vs at-most-once, secret/signature so the receiver can verify the call, per-account rate limits, and whether a stalled run should fire a distinct event from a clean failure.
6. Note the security angle - an arbitrary user-supplied URL is an SSRF surface. What restrictions are needed.
7. Scope it: what is the smallest useful version, and what would a complete version cost.

Deliverable

One comment on this issue with the findings, a recommended design, and a phased scope proposal - smallest useful slice first. No implementation, no production code changes.

[serge-ivo]

Findings

The premise needs one correction: terminal notification already exists

When an autonomous agent run finishes, there is no way for an external client to find out. Nothing is pushed.

Something is pushed. notifyUser (workers/api/src/routes/push.ts:234) is called at a terminal
point in every driver:

Driver	Terminal notify
workflows/coding-session.ts	:779 (and :551 mid-run)
workflows/job-apply.ts	:309
workflows/browser-task.ts	:137
workflows/agent-loop.ts	:308
workflows/coding-watch.ts	:106

It carries a type, title, body and a deep link, and rides typed preferences with a 10-minute
dedupe window (lib/notifications.ts:46,98,132). A stall watcher also exists —
workflows/coding-watch.ts is the watch mode lifted out of CodingSessionWorkflow (#341) and
notifies from it.

So the gap is transport, not detection. Run completion is already detected, already
classified, already deduped, and already fanned out — to exactly one sink, aimed at a browser.
That is a much smaller and safer thing to fix than "build run-completion notification", and it
means the design question is where to add a second sink, not how to detect the event.

1. Where a run terminates — three layers, no single choke point

• Per-driver result construction. Each workflow builds its own terminal object
  (coding-session.ts:739, browser-task.ts:55, job-apply.ts:118, agent-loop.ts:189/197,
  pipeline-run.ts:163/237/249/256). No shared type, no shared writer.
• finishLoopRun (lib/agent-loop-store.ts:201) — two callers,
  agent-loop.ts:227 and coding-session.ts:224. Covers loop runs only.
• closeRun (lib/pipeline-runs.ts:84) — pipelines only, four call sites.

The nearest thing to a real join point is the notifyUser set above: five sites, all five
drivers, already reached at the moment the outcome is known.

A hole worth naming separately: grep -c logEvent workflows/coding-session.ts returns 0,
against 6 in browser-task.ts, 6 in job-apply.ts, 7 in pipeline-run.ts, 3 in
agent-loop.ts. The longest-running driver — the one this ticket is most about — writes nothing
to the unified trace. Its terminal record is a chat message (:227's comment says so outright).
That is the same absence as the unfiled coding-session.ts logError finding (0 calls, against
4/4/5/3 in its peers). It should be its own issue, because it makes §5's "distinguish a stall
from a clean failure" unanswerable exactly where it matters most.

2. What already exists and should be extended, not reinvented

lib/connection-deliveries.ts is a complete at-least-once delivery engine (#58 / migration
0058) and is the answer to almost all of §5:

• enqueueDelivery writes the outbox row before the attempt (:114)
• backoffSeconds (:31) — ~1m · 5m · 15m · 1h · 3h, MAX_ATTEMPTS = 5 (:23)
• idempotencyKey(connectionId, traceId, payload) (:62) — stable-stringified, so a duplicate
  emission inside one run collapses while a deliberate re-run does deliver
• markDelivered / markAttemptFailed / claimDelivery / dueDeliveries (:212,232,290,302)
• dead-lettering visible and replayable — listDeliveries, replayDelivery (:321,335), with
  GET …/connections/deliveries and POST …/deliveries/:did/replay
• drained by cron: runDueDeliveries at index.ts:237

It has exactly one producer today — lib/steps.ts:606, the pipeline emit step. Everything
above is already built, tested and running in production for agent-to-agent events, and is
reachable from nothing else.

Also present: routes/push.ts (web push + notifyUser), lib/notifications.ts (typed kinds,
preferences, dedupe), inbound webhooks (POST /v1/triggers/webhook/:token), and lib/ssrf.ts.

3. Recommendation

Add a second sink beside notifyUser, and make it enqueue onto the existing outbox. Not a new
subsystem, not a new detector, not a new retry loop.

Concretely: a run.finished fact emitted from the same five sites, routed by the same
agent_connections vocabulary that already carries lead.created and site.live, delivered by
enqueueDelivery. The routing filter language (matchesWhere) already exists, so "only notify me
on failures" is configuration rather than code.

For the "I am an MCP client waiting on a run" case, the outbox alone is not enough — a webhook
needs a public URL an MCP client does not have. The smallest thing that serves it is a cursor
read over the delivery/event rows (GET …/events?since=<cursor>), which #566 just gave
instance_messages a working pattern for. Long-poll can come later; a cursor read removes the
"is it done yet" loop without any new transport.

4. Payload

Instance id · run id · driver (coding / browser / apply / loop / pipeline) · terminal
status · stop reason · iterations · spend · short detail · deep link (codingSessionLink already
builds these) · traceId. The trace id matters most: the outbox already correlates a delivery
to its emitting run, and without it a webhook receiver cannot join the notification back to
anything.

5. Operational questions

Free from the outbox: at-least-once, backoff, idempotency, dead-lettering, replay, correlation.

Not present — the one real gap: there is no outbound signing anywhere.
grep -rn 'hmac|signature|X-Signature' lib/connections.ts lib/connection-deliveries.ts routes/triggers.ts returns zero hits. A receiver cannot currently verify a call came from us.
Any outbound-webhook slice must add HMAC signing with a per-endpoint secret; the platform already
envelope-encrypts secrets (lib/crypto.ts) so there is a home for it.

Stall vs clean failure should be distinct event types, not one type with a reason field — a
stall is actionable ("your machine went quiet") and a failure is not ("the provider returned
400"). This depends on the coding-session tracing hole above.

6. Security

Largely solved: lib/ssrf.ts checkPublicHttpsUrl (:30, https-only) and safeFetch (:97,
redirect-capped at 5) already guard every config-supplied URL in the connector layer. A
user-registered webhook URL must go through the same path — this is a reuse requirement, not new
work. Add per-account delivery rate limits, since a webhook is an outbound amplifier a
compromised account could aim at a third party.

Phased scope

Phase 1 — smallest useful. Emit run.finished from the five notifyUser sites onto the
existing outbox, routed by agent_connections. No new transport, no new URL surface, no SSRF
exposure, no signing needed. Delivers agent-to-agent chaining on run completion immediately.

Phase 2 — the MCP client's actual need. Cursor-paged event read (?since=). Removes
busy-polling with no new outbound surface. This is the slice that answers the ticket's own
motivating case.

Phase 3 — user-registered outbound webhook. Requires HMAC signing (new), per-account rate
limits (new), and safeFetch (reuse). Largest attack surface, so last.

Prerequisite, separate issue: coding-session.ts writes neither logEvent nor logError.
Until that is fixed, Phase 1 can report that a coding run ended but not reliably why.

Verified vs inferred

• Verified: every file:line above; the five notifyUser terminal sites; the logEvent
  counts (0/6/6/3/7) and logError counts (0/4/4/3/5); the outbox's full export surface and its
  single producer at steps.ts:606; runDueDeliveries on cron at index.ts:237; the zero-hit
  signing grep; ssrf.ts's two exported guards.
• Inferred: that notifyUser's five sites are a complete set of terminal points. They are
  complete across the five workflow files, but a run can also end by DO-level cancellation or by
  a workflow instance dying without running its own catch — neither path was traced, and
  Phase 1 should establish them before claiming full coverage. Also inferred: that no MCP client
  can receive a webhook. True of hosted ChatGPT/Claude connectors; a self-hosted client could.

[serge-ivo]

Correcting my own findings comment: two of the five "terminal notify" sites are not terminal, and a third is conditional

I traced the two things the comment above labelled inferred. Both turned up something, and the
first one invalidates the table the comment opens with. The ticket's premise mostly stands; my
correction of it was too strong.

The table is wrong for two rows and overstated for a third

• workflows/job-apply.ts:309 is not terminal. It sits inside for (let round = 0; round < 12; round++) (:274, closing :349), in a step named notify-${round}, with key
  apply-handoff:${taskId}:${reason}:${round} and titles "🙋 needs an answer" / "🔐 Verification
  needed" / "✋ needs a hand". It fires when a run pauses on a human.
• workflows/browser-task.ts:137 is the same shape — inside the round loop, browse-handoff:….
• Each of those two files contains exactly two notifyUser occurrences (the import and that one
  call). So the apply and browser drivers push nothing when a run finishes. That is the ticket's
  premise, verbatim, and it is true for them.
• workflows/agent-loop.ts:308 fires only when the run needs a human. Both it and the
  escalation branch are inside if (needsHuman(stop.reason)) (:296, :305), and
  needsHuman = statusFor(reason) === "needs_human" || failed || budget || no_progress
  (lib/agent-loop.ts:201-203). done → completed and cancelled → cancelled are both false, so
  a loop run that succeeds — or one the user cancels — notifies nobody.

What survives: workflows/coding-session.ts:779 ("notify-end") is genuinely terminal and
unconditional — it is inside the finally at :664 (closing :792), so it fires on the crash path
too; and workflows/coding-watch.ts:106 ("✅ Coder finished") is the watch-mode completion.

So the honest summary is: coding pushes on completion, the loop pushes only when it needs a human,
apply and browser push only on a pause. "The gap is transport, not detection" is right for coding
and wrong everywhere else.

Consequence: phase 1 as scoped would emit almost nothing

"Emit run.finished from the same five notifyUser sites" would emit on a pause for apply and
browser, on nothing at all for a successful loop run, and correctly only for coding. Phase 1 has to
hang off the terminal record, not the notification:

• finishLoopRun (lib/agent-loop-store.ts:201) — the single terminal write for loop runs,
  called from agent-loop.ts:227 and coding-session.ts:224, and reached from the finish step
  that runs after the try/catch/finally, so it covers the crash path.
• closeRun (lib/pipeline-runs.ts:84) for pipelines.
• apply and browser have no shared terminal writer at all — each builds its own result object
  (job-apply.ts:118, browser-task.ts:55). Giving them one is real work and belongs in phase 1's
  cost, not assumed away by it.

The two inferred items, now traced

(a) DO-level cancellation, and a workflow dying without running its catch.

• Cancellation is cooperative, not a kill. requestCancel sets cancel_requested = 1
  (lib/agent-loop-store.ts:186) and the workflow polls isCancelRequested (:193), so a cancel
  returns through the normal stop → finish path. grep -rn "terminate" workers/api/src finds no
  call to a Workflow instance's terminate() or pause() anywhere — every hit is an unrelated word
  in a comment or a test. There is no server-initiated kill that skips a workflow's finalization,
  so the completeness worry does not come from cancellation.
• A workflow instance dying mid-step is real, and the repo already says so.
  lib/agent-loop-store.ts:44-46: "a running run whose value is far in the past is stalled — the
  only queryable staleness signal there is, since a Workflow that dies mid-step leaves status at
  running forever." When that happens, neither finishLoopRun nor any notify runs. [bug] WorkflowInternalError, traced: it is not the objective failing and not terminal — one run filed its own death twice, and all 8 occurrences classify as unknown #546
  (WorkflowInternalError, 8 occurrences) is the live evidence that it happens.
  This is the one the ticket cares most about ("a run that stalls with a frozen lastProgressAt
  while still reporting running"), and no terminal-writer emit can ever cover it: nothing runs.
  The stall producer has to be a cron over lastProgressAt, a separate producer from the
  terminal one — not a sixth emit site. coding-watch.ts is the only stall watcher today and it is
  coding-only. That also settles §5's "should a stall fire a distinct event": it must, because it
  comes from a different producer.

(b) "No MCP client can receive a webhook." Overstated, and it should not be argued from. It is a
property of hosted connectors (ChatGPT / Claude), not of MCP: a self-hosted client on a reachable
host can receive one. It does not change the recommendation — the cursor read serves both kinds of
client and a webhook only some clients can use is a weaker default — but the phase-3 rationale
should say "hosted clients", not "MCP clients".

What still stands from the earlier comment

Unchanged and re-verified: lib/connection-deliveries.ts is the at-least-once engine to extend
(one producer today, lib/steps.ts:606; drained by runDueDeliveries at index.ts:237); there is
no outbound signing anywhere; lib/ssrf.ts already guards config-supplied URLs; and
coding-session.ts writes neither logEvent nor logError (0 calls against 6/6/3/7 and 4/4/3/5 in
its peers), which is what makes "stall vs clean failure" unanswerable for the longest-running driver
and should be its own issue.

Readiness

Not ready to implement as scoped. The investigation deliverable is only now correct: the
termination map above replaces the table in the previous comment, and phase 1 must be re-scoped
around terminal writers plus a separate stall producer. The open decision for the owner, which I did
not close: what apply and browser use as their terminal writer — they have none, and inventing
one is the actual cost of phase 1.

Verified vs inferred

• Verified: every file:line; the enclosing for and finally line ranges quoted above; the
  notifyUser occurrence counts per workflow file; needsHuman's definition; the cooperative
  cancel; the zero-hit terminate grep.
• Inferred: nothing further. I did not run a workflow to observe a death mid-step — the claim
  rests on the repo's own statement at agent-loop-store.ts:44-46 and on [bug] WorkflowInternalError, traced: it is not the objective failing and not terminal — one run filed its own death twice, and all 8 occurrences classify as unknown #546's occurrences, and it
  is labelled as such.