Skip to content

[bug] /runtime/status answers "attached · Connected." for an instance whose pin points at a dead machine — the pin-blind fallback overrides the pin-aware resolution (#238) #380

Description

@serge-ivo

The contradiction, measured on one instance at one moment

cda75e28… (Heartfull tmux), pinned to RLs-MacBook-Air.local, which is offline; the live socket is
on Mac (full diagnosis in the node-identity issue).

GET /v1/instances/cda75e28…/runtime/status
  relay:      { "connected": true, "runnerNode": "Mac", "live": false }
  attachment: { "state": "attached", "message": "Connected.", "remedy": null }

POST /v1/instances/cda75e28…/tools/terminal_list_targets
  → "No runner is connected for this agent — run `pags up` …"

The endpoint whose job is "is this agent's runner up, and if not, why" answers Connected. for an
agent that cannot reach a runner at all.

Cause — the pin-blind fallback overrides the pin-aware resolution

workers/api/src/routes/instances.ts:761-799:

const liveRuntime = await getLiveRuntime(c.env, instanceId, session.uid);  // pin-aware → NULL here
const runtime     = liveRuntime ?? (await requireRuntime());              // default row → node "Mac"

const relayIsConnected = await relayConnected(c.env, instanceId, runtime.runner_node);   // asks "Mac" → TRUE
const attachment       = diagnoseAttachment({ relayConnected: relayIsConnected,});    // → "attached"

getLiveRuntime (routes/instances-runtime.ts:720) is getBoundRunnerConn + a row lookup, so it
returns null for exactly the reason routing fails: pinned, and the pinned machine is dead. The
fallback then re-asks the question without the pin, against a different machine, and that answer
wins.

The fallback is deliberate — its comment says it exists "to describe a registration that exists but is
not live". The bug is that its node is then fed to the liveness check and to diagnoseAttachment,
so "here is a registration that is not live" turns into "connected".

relay.live: false is the sole surviving trace that pin-aware resolution failed, and nothing reads
it
: runnerReading (store/console/src/lib/runnerPanel.ts) takes relay.connected only, so the
Runner card prints Status: Online.

Why it matters more than a wrong dot

diagnoseAttachment was added by #237 precisely so the console could stop showing "a symptom with no
cause and no remedy" — "the CLI knew the reason and the remedy and printed them to a terminal nobody
was watching."
On this instance it has both facts available (pin is dead; another node is live) and
reports the one state that has no remedy attached, because it was handed a boolean computed about a
different machine.

The console therefore says three things at once about one instance:

surface source says
Runner card headline /runtime/status relay.connected Online
pinned-machine warning /runner-node nodesDetail ⚠ pinned machine offline
every tool call getBoundRunnerConn no runner connected

Only the last two are true, and they are the two the user cannot see from the tab they are on.

Suggested fix

Answer for the route that will actually be taken:

  • when getLiveRuntime returns null, report not attached, and resolve attachment from the
    pin — the reason (pinned to X, offline since …) and the remedy (repin to Y, which is live) are
    both already computable from data this handler holds;
  • keep the fallback row for describing the registration, but never derive relay.connected or the
    attachment state from a node the pin excludes;
  • have runnerReading fail closed on relay.live === false until the above lands, so the card stops
    printing Online over a dead pin.

This is the same class as #238 ("/runtime/status reports the stale default row's node"), which
introduced getLiveRuntime. The pin-aware resolution landed; the line below it still overrides it.

Files: workers/api/src/routes/instances.ts:761-799,
workers/api/src/routes/instances-runtime.ts:720-730,
workers/api/src/lib/runner-availability.ts (diagnoseAttachment),
store/console/src/lib/runnerPanel.ts (runnerReading).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions