Skip to content

[bug] pags up does not self-heal after a wake: registration never rides the reconnect path, and the machine is locked out of its own relay slot by the socket it left behind #497

Description

@serge-ivo

What the owner saw

  ✓ Browser      running on your computer
  ✓ Secure link  connected to ProAgentStore
  ✗ ProAgentStorenot registered (retries automatically)

  Setting things up… this takes a few seconds. Keep this window open.
  Status: Relay connected

pags up (v0.4.45) on RLs-MacBook-Air.local, after the laptop had been asleep. Their question
was "is it supposed to come back by itself when I wake the laptop?" — and the answer the code
gives is yes, that is the design (the relay reconnects with 1s→30s backoff, relay.ts:288-293;
the heartbeat retries every 30s and even has a recovery message, relay.ts:126). It does not
fully work, for two independent reasons, and the pane cannot tell the user which one they have.

Separate ticket for "start pags up with no terminal open" (a LaunchAgent — a product decision,
not this bug): #___.


Part 1 — the mechanism

1a. Registration does not ride the reconnect path

registerRuntime is called from exactly two places (grep -n registerRuntime packages/cli/src/commands/runner/relay.ts → 37, 55, 162):

  • relay.ts:55for (const id of instanceIds) await registerRuntime(id);, once, at startup.
  • relay.ts:162 — inside the 20s discovery poll, and only for instances in
    diffMembership(...).attach, i.e. ones not already attached.

openRelaySocket (relay.ts:184-311) has no reference to it. So when a socket drops and
reconnects — the wake case — POST /v1/instances/:id/runtime is not re-issued. And when the
startup call fails, it is caught and logged and never tried again:

// relay.ts:50-53
} catch (e) {
    const msg = e instanceof Error ? e.message : String(e);
    writeError(`register ${id.slice(0, 8)}… failed: ${msg}`);
}

Two consequences, both verified in the route:

  1. A registration lost at startup is lost forever. Start pags up while the network is still
    coming up (exactly what happens if you run it right after a wake or a boot): fetch failed,
    no instance_runtime_nodes row for this machine, and nothing ever retries. The relay socket,
    which does retry, comes up seconds later — producing precisely the screenshot's asymmetry:
    secure link connected, ProAgentStore not registered.
  2. resumeSessionsForNode never runs on a wake. It lives inside the register route
    (workers/api/src/routes/instances.ts:365), so this machine's suspended coding sessions are
    reactivated only by a fresh pags up, not by a socket reconnect.

Note the heartbeat is not a substitute. POST …/runtime/heartbeat only UPDATEs
(instances-runtime.ts:868-891); with no node row for this machine the node-scoped UPDATE matches
0 rows, while the second UPDATE on instance_runtimes is not node-scoped — so the failed
machine's heartbeat still refreshes whichever machine owns the shared default row, and returns
200. Silent by construction.

Fix (cheapest, CLI-only): give openRelaySocket an onOpen(instanceId) callback and call
registerRuntime(id) from it. The upsert is idempotent, so re-registering on every (re)connect is
safe, it inherits the socket's existing backoff for free, and it makes resumeSessionsForNode run
on wake — which is what the multi-machine doc already promises ("the machine's own suspended
sessions reactivate on reconnect", platform-docs/coder-multi-machine.md:116).

1b. On wake the machine is locked out of its own relay slot — measured, in production

This is the part that makes recovery impossible rather than merely incomplete.

RelayDO.handleConnect decides whether an incumbent socket is alive by whether a send throws:

// workers/api/src/relay-do.ts:76-91
if (existing.length > 0 && !force) {
    let alive = false;
    for (const ws of existing) {
        try { ws.send("ping"); alive = true; break; } catch { /* dead */ }
    }
    if (alive) {
        return closeWithReason(4409, "Another runner is already connected. Use --force to take over.");
    }
}

Nothing anywhere requires the pong to come back — webSocketMessage drops "pong" on the floor
(relay-do.ts:111) and no timer, alarm or last-seen is kept. A peer that has stopped existing at
the application layer — a slept laptop — therefore still counts as alive.

Measured against production, api.proagentstore.online, on an instance I own, using a
throwaway node name (pags-probe-…, never registered, so no DB row and no routing effect):

step result
holder connected, node pags-probe-xeg1dj /status{"connected":true}
holder SIGKILLed, reconnect same node immediately accepted — a hard-killed runner does NOT conflict
holder SIGSTOPped (frozen = the shape of a slept laptop), reconnect same node 4409 "Another runner is already connected. Use --force to take over."
same, 47s later 4409 again; /status still {"connected":true}
same, still frozen, polled once a minute still 4409 at t+3m (probe still running at time of writing)
same, with &force=1 accepted

So: the machine's own pre-sleep socket holds its slot; /status reports it connected the whole
time; and the only thing that gets in is --force. Note the SIGKILL row — this is specifically a
frozen peer, not a dead process, which is why restarting pags up normally works and waking a
laptop does not.

On the client side, a 4409 is terminal for the life of the process:

// relay.ts:274-279
if (ev.code === 4409 && !force) {
    writeLine(`Relay conflict: ${instanceId.slice(0, 8)}… is connected on another machine — run \`pags up --force\` here to take it over.`);
    closed = true;
    onConflict?.(instanceId);
    return;
}

onConflict adds the id to blocked (relay.ts:73), and diffMembership excludes blocked ids
from attach (membership.ts:63). Nothing ever removes an id from blockedgrep -n blocked packages/cli/src/commands/runner/relay.ts gives 61 (declare), 73 (add), 155 (pass);
there is no delete. That directly contradicts the comment that documents the intent:

membership.ts:46-48 — "They are NOT dropped from the eligible set, so clearing the block (the
other machine disconnects, or the user runs --force) lets the next pass attach."

The block cannot clear, so even after the zombie expires and the slot is free, that agent stays
detached until the user restarts the CLI.

Live evidence this is happening on the owner's account (GET /v1/terminals/nodes, read
2026-08-11):

machine relay sockets live rows reading status: online
RLs-MacBook-Air.local (v0.4.45) 6 of 19 19
Sergeys-Mac-mini.local (v0.4.45) 18 of 18 18

Six of the Air's thirteen detached agents are pinned to the Air (bound: true), including
iTerm2 Operator, kitty Operator, Terminal Operator, tmux Operator, FWS platform,
Chess coder. Its pags up is running and will never reattach them.

Fix (server, and the one that actually restores automatic recovery): make the incumbent
liveness test evidence-based — send the ping and require a pong within a short deadline before
rejecting. handleConnect becomes async; resolve a promise keyed to that socket from
webSocketMessage when "pong" arrives; no pong in ~1.5-2s → treat the incumbent as dead, close
it, accept the newcomer. This keeps #237/#229's promise intact: a genuinely live second runner
answers instantly and still gets its 4409, so the "endless noisy reconnect against a real
conflict" that #237 was filed about does not come back. The runner answers a ping before it parses
anything else (relay.ts:238), and a command in flight does not queue behind it, so a busy runner
still pongs promptly.

Fix (client, complementary and cheap): make blocked clearable, as its own comment already
says it is. On each 20s discovery pass, for blocked ids only, GET /v1/relay/:id/status
(one cheap request, no socket) and unblock when it reports connected: false. Log the unblock
once. This is not a return to the 30s reconnect spam — it is one status GET per poll, and it is
the difference between "recovers by itself when the conflict ends" and "recovers when you notice
and restart".

Rejected alternative: carry machineId (#379) in the relay handshake and let a matching
machine always displace itself. It is a real option and it would fix the wake case, but it needs a
CLI change (version skew — the owner is on 0.4.45 and would not get it), it leaves /status still
reporting a zombie as connected, and it does nothing for the frozen-socket case on a machine whose
id file was reset. The pong check fixes all of those server-side, with no CLI release.

Also worth fixing with the same handle: handleStatus (relay-do.ts:140-156) uses the same
send-doesn't-throw test, which is why /status said connected: true for a peer that was gone.
diagnoseAttachment returns attached / "Connected." on relayConnected alone
(lib/runtime-attachment.ts:55-57), so during the zombie window the console tells the user the
agent is connected while every runner call goes to a socket that will never answer and dies on the
120s relay timeout (relay-do.ts:32). Same family as #380, different cause.


Part 2 — the pane, which cannot report any of this

2a. The registration light is a latch that can never clear

state.registration is derived entirely from string-matching the child's stdout
(up.ts:147-203); there is no timer and no polling. Only two lines set it to "failed":

  • up.ts:184trimmed.includes("Another machine"). Nothing in the codebase ever prints that
    string
    grep -rn "Another machine" across the repo returns only up.ts:184 itself. Dead
    branch.
  • up.ts:190trimmed.includes("fetch failed"), which sets
    state.lastEvent = "PAGS registration failed".

And only three lines set it back to "registered"up.ts:163 ("WebSocket relay") and
up.ts:178 ("Runtime registered" / "CONNECTED") — all three of which are matched by
relay.ts:90 and relay.ts:93, printed exactly once, at startup. After startup, no line the
runner ever emits can turn the ✗ back into a ✓.

So the pane's ✗ survives the recovery. In particular, the most likely producer of the owner's
screenshot is the heartbeat, not registration:

// relay.ts:121-127
if (failure && !heartbeatFailing) {
    heartbeatFailing = true;
    writeError(`Heartbeat failed: ${failure} — the console will show this machine as OFFLINE until it recovers. ...`);
} else if (!failure && heartbeatFailing) {
    heartbeatFailing = false;
    writeLine("Heartbeat recovered — this machine reads as online again.");
}

On wake, the first heartbeat fails with undici's fetch failedup.ts:190 fires → the pane
says "not registered" for a heartbeat failure, with the remedy text "(retries
automatically)". Thirty seconds later the heartbeat recovers and prints so — and
"Heartbeat recovered — this machine reads as online again." matches no branch in up.ts, not
even the catch-all error regex at up.ts:198 (no "error", no "failed"). The pane stays ✗ forever
while the thing it names is fine.

I could not determine from the screenshot alone which of the two the owner hit — a startup
register failure (1a, permanent) or a heartbeat blip (2a, self-healed). Both render identically.
That indistinguishability is itself the defect; the only way to tell is to press l and read the
scrollback for register … failed: vs Heartbeat failed: / Heartbeat recovered.

Fix, cheapest first:

  1. up.ts:184 — delete the dead branch, or point it at the message that actually exists
    ("Relay conflict:", relay.ts:275). Today a 4409 matches nothing in handleOutput: not
    the conflict branch (case differs — the message says "another machine", lowercase) and not the
    error regex (contains neither "error" nor "failed"), so the user is never told the one thing
    the CLI knows and the one command that fixes it.
  2. relay.ts:90 — stop printing "Runtime registered with PAGS ✓" unconditionally. It is emitted
    after the register loop whether or not a single register succeeded, so the pane can show
    ✓ ProAgentStore for a machine that registered nothing. Print the truth:
    Runtime registered: N/M agents.
  3. Give the heartbeat its own state instead of borrowing registration's, and clear it on
    "Heartbeat recovered".
  4. Better than all of the above: have the runner emit one explicit machine-readable status line
    (e.g. PAGS-STATUS registration=ok|fail relay=ok|fail reason=…) and have the TUI render that,
    instead of inferring product state from prose that a later commit will reword.

2b. "this takes a few seconds" is unbounded

tui.ts:89 prints "Setting things up… this takes a few seconds. Keep this window open." on
every render where connected is false (tui.ts:65). There is no elapsed-time input to
printStatus and no timer — so a state that is permanent (1a) is described as taking a few
seconds, indefinitely. Show elapsed time and change the wording past ~30s:
Still connecting — 4m elapsed. Press l for logs.

2c. ProAgentStorenot registered — the label column is one char too narrow

// tui.ts:72
console.log(pad + icon + " " + w(label.padEnd(13)) + d(note));

"ProAgentStore".length === 13, so padEnd(13) adds nothing and the label runs into the note.
"Browser" (7) and "Secure link" (11) both pad fine, which is why only this row is affected.
Fix: padEnd(15), or derive it — Math.max(...labels.map(l => l.length)) + 2.


Not the cause — checked

What I could not reproduce

I cannot sleep the owner's laptop, so the wake path end-to-end is inferred, not observed. What is
measured is the piece it turns on: a frozen peer holds the relay slot and forces 4409 (table
above), and the CLI's handling of 4409 is terminal (code, cited). The fetch failed → "not
registered" latch is read from code, not observed live.

Acceptance criteria

  • Sleep a laptop with pags up running for ≥5 minutes, wake it: within one backoff cycle
    every agent is attached again and /v1/terminals/nodes shows sockets live for all of them,
    with no keypress and no --force.
  • RelayDO.handleConnect rejects with 4409 only after the incumbent has answered a ping;
    a non-answering incumbent is closed and replaced. Unit test both directions.
  • A blocked instance is re-attached automatically once the conflict clears, without
    restarting the CLI.
  • Kill the network for 2 minutes with pags up running, restore it: the pane returns to
    ✓ ProAgentStore by itself.
  • Start pags up with the network down, bring it up: registration completes without a
    restart.
  • pags up started against an unreachable API shows ✗, not ✓, on the ProAgentStore row.
  • A 4409 renders in the pane with its remedy, not only in l logs.
  • Labels align: no ProAgentStorenot.

Regression risk

  • The pong deadline is the risky one. Too short and a real, live runner loses its slot to a
    second pags up; too long and every conflicting connect pays it. 1.5-2s is far above the
    runner's ping handling (relay.ts:236-238, answered before any parsing). The test that catches
    a regression is a relay-do case where a responsive incumbent still produces 4409 — that is
    [bug] A rejected relay socket retries forever and the console shows only an unexplained amber dot #237's shipped promise and it must keep passing.
  • relay-do.test.ts cannot catch this class today. Its MockWebSocket.send throws when
    closed (relay-do.test.ts:30-33), which encodes the very assumption production contradicts —
    a mirror of the implementation cannot falsify the implementation's premise. Any fix here needs
    a test whose fake socket can be silent as well as closed. Same blind spot as Both testing blind spots are closed — what remains is a fetch-deadline floor in safeFetch, then two ratchets (deadline + responsive-label) #438.
  • Re-registering on every reconnect increases POST /runtime volume by roughly one call per
    instance per reconnect. The route is an upsert, but it also runs resumeSessionsForNode and
    (only under force) suspendSessionsFromOtherNodes — confirm the non-force path stays
    side-effect-free for sessions that are already active.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions