`instance_runtime_nodes.status` is a write-once "online" column — every machine an instance has ever seen reports online forever, including one last seen four days ago

[serge-ivo]

instance_runtime_nodes.status is a write-once "online" column — every machine an instance has ever seen reports online forever, including one last seen four days ago

Observed

instance_runtime_status(bd43f4de-ef35-4051-bdec-43f8571414a1), production, 2026-08-15. Four node rows, one machine actually connected:

runnerNode	runnerVersion	lastSeenAt	reported status
RLs-MacBook-Air	0.4.50	2026-08-14 23:13:08	online
Sergeys-Mac-mini.local	0.4.45	2026-08-12 07:43:42	online
RLs-MacBook-Air.local	0.4.47	2026-08-11 21:40:20	online
Mac	0.4.45	2026-08-10 20:27:28	online

Three of the four have not been seen in 2–4 days. All four say online. Reading the status field, an operator (or an agent) answering "which of my machines is up?" gets four yeses and one right answer.

Mechanism — the per-node table has no writer for any status but "online"

Verified. workers/api/src/routes/instances-runtime.ts:868-891:

export async function updateRuntimeStatus(env, instanceId, userId, status, runnerNode?) {
	const node = normalizeRunnerNode(runnerNode);
	if (node) {
		await env.DB.prepare(
			`UPDATE instance_runtime_nodes SET status = ?1, last_seen_at = …
			   WHERE instance_id = ?2 AND user_id = ?3 AND runner_node = ?4`)
			.bind(status, instanceId, userId, node).run();
	}
	await env.DB.prepare(`UPDATE instance_runtimes SET status = ?1, …`)…
}

The per-node UPDATE runs only when a runnerNode argument is supplied. Grep-complete list of the eight call sites:

routes/instances.ts:549         "online",  normalizeRunnerNode(body.runnerNode)   ← the only one with a node
routes/instances.ts:844          effective                                        (no node)
routes/instances.ts:893          "offline"                                        (no node)
routes/instances-tasks.ts:110    "online"                                         (no node)
routes/instances-tasks.ts:112    "offline"                                        (no node)
routes/instances-tasks.ts:116    "offline"                                        (no node)
routes/instances-tasks.ts:605    "offline"                                        (no node)
routes/instances-tasks.ts:608    "offline"                                        (no node)

Exactly one call site passes a node, and it passes "online". Every offline write reaches instance_runtimes and never instance_runtime_nodes. So a node row's status is set once, at registration, and never changes — and GET /v1/instances/:id/runtime (routes/instances.ts:433) serialises that raw column straight out through runtimeNodeResponse, with no liveness check:

return c.json({ runtime: runtime ? runtimeResponse(runtime) : null, nodes: nodes.map(runtimeNodeResponse) });

Why the routing code is fine and this still matters

Routing does not trust this column, deliberately. CLAUDE.md states it: resolution goes through getBoundRunnerConn → getLiveRunnerConn, "live-checked, not the DB status column which isn't cleared on disconnect". And the probe path of the same route (routes/instances.ts:840-887, ?probe=1) does the pin-aware live check that #380 built.

So work is routed correctly. What is wrong is the reporting: the un-probed nodes[] array is the answer instance_runtime_status gives over MCP, and it publishes a column the rest of the codebase knows is meaningless. The console does not read it — it uses /v1/instances/:id/runner-node (nodesDetail) and /v1/terminals/nodes, per store/console/src/lib/runnerPanel.ts:5,41 — which is why this has stayed invisible: the one consumer of the stale field is the debugging surface.

What to do — cheapest first

1. Do not serialise the column. Compute each node's status from last_seen_at against the same recency window routes/instances.ts:826 already uses for the single-runtime answer ("A runner heartbeats every 30s … if it was seen…"), or omit status from nodes[] entirely and let lastSeenAt speak. Either is honest; publishing a write-once field as a live one is not.

2. Or make the column true: pass the node through on the offline paths. More invasive — the offline writers at instances-tasks.ts do not know which node failed, and guessing would mark the wrong machine down. Prefer (1).

3. Say which node is live. instance_runtime_status already returns runtime.runnerNode for the default row; the useful field is a per-node live: boolean from relayConnected, which the probe path computes and the list path does not.

Alternatives considered and rejected

• Prune stale node rows. They are the "Runs on" tile list and a pin target — config.runnerNode can legitimately name a machine that is currently off. Deleting history to fix a status field would break the pin UI.
• Fix it in the MCP tool. The field is wrong at the route; the MCP tool is a pass-through, and the next consumer would inherit it.

Acceptance criteria

• No API response reports a node online whose lastSeenAt is older than the heartbeat window.
• On the instance above, exactly one node reads live.
• Test: two node rows, one heartbeated now and one 3 days ago; assert the response distinguishes them.

Regression risk

config.runnerNode pins survive a machine being offline — a status change must not make a pinned-but-off machine disappear from the "Runs on" tiles or from /v1/terminals/nodes. Test that a pinned offline node is still listed, just not as online.

Recorded non-reproduction

The observation that started this was "my_instances pins config.runnerNode: "RLs-MacBook-Air" while the live node is "RLs-MacBook-Air.local", and list_errors shows 409s reading Runner node is not connected: RLs-MacBook-Air". That is not what is happening, and the pin is not broken.

• normalizeRunnerNode (lib/runtime-nodes.ts:1-3) is trim().slice(0,120) — no .local stripping — so the two ARE different nodes with different relay DOs (bd43f4de…:node:RLs-MacBook-Air vs …:node:RLs-MacBook-Air.local, both present in the response above).
• But the currently-connected node for this instance IS RLs-MacBook-Air, exactly matching the pin. The .local row is a stale 08-11 registration — which is only confusing because it also says online, which is this issue.
• The 409s in list_errors (2026-08-14 21:18, and Runner node is not connected: Mac on 08-13) are on different instances (f8ddc272, e4d2d031), whose own node rows I did not enumerate. They are consistent with [bug] A session stamped to an offline machine makes the Coding banner tell an owner already running pags up to run pags up (#531 AC 2) #537 (a session stamped to an offline machine) rather than with a normalisation bug.

One physical laptop registering under two hostnames is real and is arguably worth its own decision, but it is a pags up hostname-resolution question, not a matching bug in the API.

Related: #537, #440 (a transport failure stored as durable state), #380 (the pin-aware liveness answer on the probe path).

[serge-ivo]

Landed in f126db6, fixed forward in fcfbce0 — the first commit was green everywhere and did nothing in production. Both deployed.

Confirmed the BA's grep myself

Eight updateRuntimeStatus call sites, exactly one passes a runnerNode (routes/instances.ts:549), and it passes "online". Every offline write reaches instance_runtimes and never the per-node table. normalizeRunnerNode is trim().slice(0,120) — no .local stripping. The recorded non-reproduction stands; I did not re-derive it.

What shipped

runtimeNodeResponse derives a node's status from its heartbeat instead of publishing the column. HEARTBEAT_FRESH_MS already existed in lib/runtime-attachment.ts claiming to be "the same window the status probe uses" while four call sites re-inlined both the number and the D1 timestamp parse; heartbeatFresh() is now that one definition and all four use it. A stored offline still wins, so the derivation only ever moves an answer toward offline, and every node stays listed — a pin may legitimately name a machine that is switched off.

Option 3 (a per-node live from relayConnected) was not taken: this is a serialiser, and making it async puts a Durable Object fetch per node behind a list endpoint. ?probe=1 is where the live check belongs and already is.

The first commit shipped a dead fix — how

runtimeNodeResponse(row, now = Date.now()) took an injectable clock so a test could pin it. Both callers are nodes.map(runtimeNodeResponse), and Array.prototype.map passes (element, index, array) — so now was 0, 1, 2, 3, and heartbeatFresh evaluated 0 - <a real timestamp> < 90_000, true for every row ever written. Every unit test passed now explicitly, so the suite measured a call shape nothing in the codebase makes.

Caught by checking the live API after the deploy: four nodes, still all online. The fix is the signature — one argument, which map's extra arguments cannot corrupt — not the two call sites, which would have left the trap armed for the third. The guard is now rows.map(runtimeNodeResponse), and it fails on the shipped code with exactly the production symptom.

Verified live, after the second deploy

node                       status    lastSeenAt (UTC)      age
RLs-MacBook-Air.local      online    2026-08-15 00:47:05   0:00:20
RLs-MacBook-Air            offline   2026-08-14 23:35:40   1:11:45
Sergeys-Mac-mini.local     offline   2026-08-12 07:43:42   2 days, 17:03:43
Mac                        offline   2026-08-10 20:27:28   4 days, 4:19:57

Exactly one node reads live, and all four are still listed with their relay names — both acceptance criteria, on the instance the issue reports.

Also

instances-runtime.ts lost the two serialisers and RuntimeRow to lib/ (the route file is under a size ratchet and these are pure functions of a D1 row). The type had to move too: left behind, the new lib module imported a routes module back and import-graph.test.ts failed on the cycle — the guard doing its job on a split that looked finished. Both are re-exported, so no importer changed; the pin came down 50 lines rather than being banked.