copilot-cli: process hangs indefinitely after final safe-output is submitted; MCP gateway SSE streams remain open until action-step timeout

[microsasa]

Summary

The Copilot CLI agent process can fail to exit after it has completed all
of its work, including a successful safe-output call. The MCP gateway
server-sent-event streams (one per registered MCP server) remain open
after the agent's last RPC, and the only backstop is the action-level
step timeout killing the runner. The full step budget is burned while
producing no additional value.

This is distinct from #39282: that PR addresses mid-task idle hangs
where the agent stops making progress; what is described here is a
post-completion hang where the agent finished its work, the
safe-output was processed successfully, and the issue was filed —
but the process never terminated.

Environment

• gh-aw v0.79.6 (pinned via gh extension install github/gh-aw@v0.79.6)
• engine: copilot (Copilot CLI driver, default)
• Model: claude-sonnet-4.6
• MCP gateway: ghcr.io/github/gh-aw-mcpg:v0.3.25
• Two MCP servers registered: github (via ghcr.io/github/github-mcp-server:v1.1.2) and safeoutputs (gh-aw built-in)
• safe-outputs: create-issue with max: 1
• Trigger: workflow_dispatch only
• Step timeout: GH_AW_TIMEOUT_MINUTES: 20 (gh-aw default)

Symptom

An agentic workflow runs to completion, the agent emits its full
markdown output, calls a single create_issue safe-output, the
safeoutputs MCP server returns success — and then the Copilot CLI
process sits idle for the remainder of the 20-minute step budget. At
T+20m the action wrapper force-kills the runner.

The created issue does land (visible in the repository immediately,
before the hang ends). So this is not a missing-output failure; it is
purely a "process refuses to exit" tail.

Reproducibility

Non-deterministic. In one observed cycle, several structurally
identical agentic workflow dispatches (same engine, same gateway image,
same model, same MCP servers, same safe-outputs: create-issue
configuration, dispatched seconds apart) produced N-1 clean exits and 1
post-completion hang. The hanging run's prompt scaffolding was
identical to the clean runs; only the agent-generated text in the
create_issue body field differed across runs.

We do not yet have a deterministic trigger. Re-running with the exact
same inputs sometimes hangs and sometimes does not.

Timeline evidence (representative hung run)

All times relative to step start (T+0):

T	event
T+27s	MCP gateway: tools/list calls return for both servers
T+27s	MCP gateway: per-server SSE streams open (routed:github, routed:safeoutputs)
T+12m08s	MCP gateway: initial SSE streams close (server-side, duration=11m41s, "empty body")
T+12m08s	MCP gateway: new SSE streams immediately re-open (same servers)
T+12m45s	Agent's only tool call: create_issue → safeoutputs MCP server
T+12m45s	safeoutputs returns success
T+12m45s	Agent stdio log: agent emits its final markdown summary
T+12m45s	Agent stdio log: cuts off mid-output — no [copilot-harness] attempt N: process exit event line, no Stopping containers, no [INFO] Agent session state preserved
...silence...
T+20m00s	Action wrapper: ##[error]The action 'Execute GitHub Copilot CLI' has timed out after 20 minutes
T+20m00s	MCP gateway: Shutdown signal received, initiating graceful shutdown
T+20m00s	MCP gateway: final SSE responses close (duration=8m19s, "empty body" for both servers)

In a clean comparison run from the same cycle:

• Agent stdio log continues past the final markdown summary with the
  expected Changes / Duration / Tokens lines, then
  [copilot-harness] attempt 1: process exit event exitCode=0, then
  container shutdown lines.
• MCP gateway shutdown timestamp is consistent with the agent exiting
  normally, not a forced timeout.

So the difference is at the Copilot-CLI / SDK driver layer: in the hung
case, after the final create_issue returns success and the agent
emits its closing markdown, the driver never reaches its normal
"session complete → close MCP clients → exit process" path.

What is innocent

• The MCP gateway itself is operating correctly — it serves the
  request, returns the response, and waits for the client to close the
  stream. It only shuts down when the runner kills it.
• The safe-output flow itself is correct — the issue gets created, the
  safeoutputs.jsonl line is valid, the create_issue MCP call
  returns {"result":"success"}.
• The github-mcp-server is unused in the hung run's RPC log past
  T+27s (only tools/list ever called) — i.e. the hang persists even
  though only one of the two MCP servers was actively used.

Hypothesis

The Copilot CLI / SDK driver has at least one code path (rare,
non-deterministic) where, after the model emits its final assistant
message + tool result and the SDK has nothing more to send, the driver
fails to:

1. Close its outbound MCP SSE clients (gateway sees stranded SSE
   streams).
2. Resolve the top-level harness sendAndWait promise (no process exit event line is ever emitted).

#39282's watchdog (5-min idle on SDK events) would catch this case
if the driver still considers itself "mid-turn" — but only by
force-disconnecting, not by reaching the natural completion path. The
classification under that PR's pipeline would be
GH_AW_AGENTIC_ENGINE_IDLE_HANG rather than a successful exit, even
though the agent's work did complete successfully and the safe-output
was persisted.

So #39282 is a generic safety net for both mid-task and
post-completion hangs, but it doesn't address the underlying SDK
driver bug — it only bounds the damage. The proper fix would be in
whatever path the SDK driver takes after the final assistant-message-
with-tool-result is processed.

What would help

• Confirmation that the SDK driver has a known code path where the
  final-tool-result response can leave the session in limbo (not
  errored, not completed).
• Diagnostics we could enable on our side to capture more of the SDK's
  internal state when the hang occurs (env var to dump session state
  on step timeout, for example).
• Status / merge plans for Add Copilot SDK idle-hang watchdog and classify GH_AW_AGENTIC_ENGINE_IDLE_HANG #39282 (or a successor). The watchdog
  pattern there is general enough to bound this failure mode even if
  it doesn't fix the root cause.

What we can provide

• Full agent artifact from a representative hung run (MCP gateway
  logs, agent stdio, safe-output JSONL, agent_usage.json,
  agent_output.json, rpc-messages.jsonl, firewall logs). Happy to
  share privately if needed — the public-issue version is summarized
  above.
• A scrubbed reproducer workflow if useful, though we currently cannot
  guarantee a deterministic repro on our side.