instance_runtime_status.runtime publishes a raw status and blends two machines — #570's fix reached nodes[] only

[serge-ivo]

instance_runtime_status.runtime reports an offline machine as online, and is a composite of two machines

Measured live 2026-08-15 across 22 instances with capabilities.runtime != null (82 node rows).

Two instances state a falsehood inside a single response

12ebf1f0 (Coder Home) and 6d2d9401 (Facebook Friends):

field	value
runtime.runnerNode	RLs-MacBook-Air.local
runtime.status	online
runtime.lastSeenAt	2026-08-15 10:11:58
runtime.runnerVersion	0.4.51, capabilities include coding.repo-write
nodes[] same node	offline, lastSeenAt 2026-08-14 23:36:04 — 10h36m stale
only live node	Sergeys-Mac-mini.local, 0.4.45, no coding.repo-write

nodes[] is correct.

Mechanism, traced

• workers/api/src/routes/instances.ts:433 — runtime: runtimeResponse(runtime), nodes: nodes.map(runtimeNodeResponse).
• workers/api/src/lib/runtime-response.ts:26 — runtimeResponse publishes status: row.status raw. :63-71 — runtimeNodeResponse applies the 90s heartbeatFresh derivation. fcfbce0 (instance_runtime_nodes.status is a write-once "online" column — every machine an instance has ever seen reports online forever, including one last seen four days ago #570) fixed only the second.
• updateRuntimeStatus (routes/instances-runtime.ts): the node UPDATE is filtered AND runner_node = ?4; the instance_runtimes UPDATE is WHERE instance_id = ?2 AND user_id = ?3 — no node filter. So the Mac mini's heartbeat (instances.ts:549) refreshes the shared row's status/last_seen_at while runner_node, runner_version and capabilities keep the last registrant's values.

The row is therefore not stale — it is two machines blended: one machine's liveness under another machine's identity and capability list.

Scope

• runtime.status === "online" on 22 of 22. No offline case observed — the same write-once-online shape instance_runtime_nodes.status is a write-once "online" column — every machine an instance has ever seen reports online forever, including one last seen four days ago #570 fixed, still live on this field.
• runtime.lastSeenAt disagrees with its own named node on 7 of 22.
• probe:true bypasses the derivation entirely (instances.ts:813-898, effective = online || recentlySeen), so the more thorough call is the less truthful one.

runtime-response.ts:38-52 already documents this exact bug class. It was applied to one of the two functions it describes.

Why this misleads a user into a wrong action

This is the field an owner reads to decide whether to run pags up. It currently says the machine with coding.repo-write is online when that machine has been gone for ten hours and the live one cannot write to repos. A run dispatched on that basis fails for a reason the status screen denies.

Acceptance criteria

1. runtimeResponse derives status from heartbeat freshness, as runtimeNodeResponse does. Neither may publish a stored status raw.
2. runtime stops blending machines: either it is scoped to one node, or the field set it publishes comes from the same row its lastSeenAt does. State which, and why.
3. probe:true is at least as truthful as the un-probed call. A path that skips a correctness derivation in the name of thoroughness is the defect, not a shortcut.
4. A guard asserts every runtime-status publisher applies the derivation — denominator stated per ADR 0002 (the number of publishers found, not a hand-listed pair). instance_runtime_nodes.status is a write-once "online" column — every machine an instance has ever seen reports online forever, including one last seen four days ago #570's fix passing while this one survived is exactly what a per-function test cannot catch.
5. Red demonstrated on today's code for the 2-of-22 case.

Verified vs inferred

• Verified: all values above read live; every file:line; the two UPDATE statements and their differing WHERE clauses; the 22/22 and 7/22 counts.
• Inferred: nothing load-bearing. The blend is read directly off the two SQL statements.

[serge-ivo]

Fixed and verified in production.

Commits: 35301fe (the fix) and d8cc2bb (a regression the first one introduced — see below).

What changed

AC1 — the derivation moved into runtimeResponse, which runtimeNodeResponse now composes. There is one serialiser, so there is no bypass. A stored offline still wins, so it can only move the answer toward offline.

AC2 — scoped to one node, at the write. updateRuntimeStatus's instance_runtimes UPDATE now carries the node filter its per-node sibling always had, so the shared row's liveness can only be refreshed by the machine it names. runner_node = '' (the pre-0030 default) still matches, or every registration older than that migration would go permanently unheard-from. Fixing the write rather than reconciling on read: the read-side repair leaves the corruption in the row and needs redoing at the next publisher.

Also: last_seen_at stops advancing on an offline write. It means "when we last heard from this runner" and was being stamped at the moment we concluded we had not.

AC3 — the probe passes the node it actually probed (both the success and the failure path), and echoes the timestamp the write left instead of now. An always-fresh stamp made heartbeatFresh true by construction, which is precisely how the more thorough call became the less truthful one. That also gives instance_runtime_nodes.status its first offline writer — #570's premise was that exactly one call site passed a node and it passed "online".

AC4 — the guard is workers/api/src/lib/status-domain.ts + status-domain.test.ts, built once for this issue and #590. Denominator: 14 status columns, 52 declared values, 4 unwritable by application code, and it asserts nothing takes a decision on any of them. Allowlist ships empty. It would have caught #570, this, and #590 — including lib/runner-client.ts:73, which routes on WHERE status != 'offline', a value that had no writer at all until this change.

AC5 — red demonstrated: reverting the one-line derivation fails 4 of the 9 new cases in lib/runtime-response.test.ts, including the measured 10h36m fixture.

Verified live (post-deploy, 22 instances with a runtime)

	before	after
runtime.status == "online"	22 of 22	20 of 22
runtime disagrees with its own named node	7 of 22	0 of 22

The two now reporting offline are exactly the two this issue names — 12ebf1f0 (Coder Home) and 6d2d9401 (Facebook Friends) — both now offline for RLs-MacBook-Air.local, agreeing with that machine's own node row.

Probe vs un-probed, 8 sampled: 8 agree, 0 disagree.

A regression this fix introduced, caught in production and fixed forward (d8cc2bb)

heartbeatFresh parses a stored stamp as `${s.replace(" ", "T")}Z`, because D1 writes datetime('now') without a zone. The probe echoed new Date().toISOString(), which already ends in Z — so it became …ZZ, Date.parse returned NaN, and the row read as never heard from. Harmless while the status was published raw; the moment 35301fe made the status derive from that stamp, the probe started reporting offline for machines it had just successfully reached. Measured: probe status=offline lastSeen=2026-08-15T10:54:51.917Z — an offline verdict carrying a four-second-old timestamp, self-contradictory on its face.

d1Timestamp() now owns that shape next to the derivation that consumes it, and a guard walks the source for any last_seen_at: built with toISOString. Every fixture in the test file was already written in the D1 shape, which is exactly why 12 green tests could not see it.

Not done

routes/instances-tasks.ts has four updateRuntimeStatus call sites that pass no node. They resolve their runtime from the default row and write back to it, so they are self-consistent and were not part of the measured blend — but they cannot mark a node down. Left alone deliberately: outside this issue's files, and a separate change.