[bug] A rejected relay socket retries forever and the console shows only an unexplained amber dot

[serge-ivo]

What happens

Each instance gets its own relay socket (packages/cli/src/commands/runner/relay.ts). On close
it reconnects with exponential backoff capped at 30s and never gives up — including when the
server is rejecting it deliberately:

ev.code === 4409  → " — run `pags up --force` to take over"

A 4409 (another runner already holds this instance+node) therefore loops forever at 30s
intervals. The other instances on the same machine stay connected, so:

• Terminals / Settings report the machine as online.
• That one agent reports not attached — amber, "machine online, agent not attached".
• The console never surfaces the reason or the remedy, even though the CLI knows both: it prints
  the close code and the hint to its own stdout, which nobody is watching.

The user experience is an agent that is silently unusable with no explanation, on a machine that
looks healthy. Every runner-backed tool (repo_tree, repo_read_file, repo_git,
repo_remote) fails with "No runner is connected" while the dot says the machine is up.

Fix

The close code is already known client-side. Report it:

• Persist the last relay rejection per (instance, node) — code + reason — and expose it on
  /v1/terminals/nodes and /v1/instances/:id/runtime/status.
• Render it where the amber dot is: "another runner holds this agent on RLs-MacBook-Air.local —
  run pags up --force", not just "agent not attached".
• Consider surfacing repeated 4409s in pags up's own summary line rather than only as a scrolled
  log line.

Related

Same amber symptom, different cause: #236 (the runner's instance list is frozen at startup). The
two are indistinguishable in the UI today, which is the actual problem — there is no way for a
user to tell which one they have.

[serge-ivo]

Scope note vs #229: that ticket's acceptance criteria include "a 4409 conflict does not create an infinite noisy reconnect loop for auto-discovered instances". This issue is the broader and already-live problem — a 4409 on an explicitly listed instance loops today, and regardless of cause the console surfaces only an unexplained amber dot. The CLI already knows the close code and the remedy; nothing carries it to the UI.

[serge-ivo]

Code audit update: the relay retry loop part appears fixed in the CLI: runner relay HEAD conflicts now call onConflict and stop retrying on 4409. The issue still has a valid remaining gap: persist/expose the last relay rejection in node/runtime status so the UI can show a clear 'runner attached elsewhere' reason instead of a generic offline/stale state. Keeping open, narrowed to the API/UI surfacing work.

[serge-ivo]

Fixed in ec272a8 — the server now says why, and the console shows it.

#229 already made the 4409 terminal CLI-side (it stops reconnecting and marks the instance blocked instead of looping at 30s forever). This closes the other half: the console never learned the reason.

Derived server-side, so no CLI change and no version skew. "Is there a registered node" + "is its heartbeat fresh" + "is a socket live" already separates every case:

state	remedy
attached	—
never-registered	pags up
runner-offline	pags up
machine-online-agent-detached	pags up --force

That last row is the one that matters and the reason a boolean wasn't enough: the machine is demonstrably alive — still heartbeating, its other agents connected — so plain pags up is the wrong advice. diagnoseAttachment is pure and tested, including that a missing or malformed heartbeat reads as stale rather than fresh (treating it as fresh would tell a user with no runner at all to use --force).

The Settings panel renders the message plus the one command that fixes it, under the existing amber line.