Single-runtime assumption corrupts status files of workloads managed by other runtimes

[JAORMX]

Summary

When multiple container runtimes coexist (e.g., podman and go-microvm), running any CLI command (thv list, thv status, thv stop, thv restart) with one runtime active permanently corrupts the status files of workloads managed by the other runtime, marking them as unhealthy.

This happens because the architecture assumes a single runtime owns all workloads. There is no concept of runtime ownership -- status files, labels, RunConfig, and core.Workload carry no information about which runtime created a workload.

Reproduction

1. Start MCP servers using the default podman runtime:

   thv run some-mcp-server
   thv list  # shows running

2. Switch to the go-microvm runtime:

   export TOOLHIVE_RUNTIME=go-microvm
   thv list

3. Switch back to podman:

   unset TOOLHIVE_RUNTIME
   thv list  # podman servers now show as "unhealthy"

The podman containers are still running fine, but their status files have been overwritten.

Root Cause

The bug lives in the file-based status manager's reconciliation logic during ListWorkloads and GetWorkload.

The chain of events

1. Factory.Create() creates a single runtime (pkg/container/factory.go:109-110). If TOOLHIVE_RUNTIME=go-microvm, only the go-microvm client is initialized.

2. fileStatusManager.ListWorkloads() reads status files for ALL workloads (pkg/workloads/statuses/file_status.go:288-289) regardless of which runtime created them.

3. It queries only the active runtime (file_status.go:283): f.runtime.ListWorkloads(ctx) -- the go-microvm runtime returns zero podman containers.

4. mergeRuntimeAndFileWorkloads() calls handleRuntimeMissing() (file_status.go:1128-1135) for every workload that exists in status files but not in the active runtime's container list.

5. handleRuntimeMissing() writes WorkloadStatusUnhealthy to the status file on disk (file_status.go:967-974):

   if fileWorkload.Status == rt.WorkloadStatusRunning || fileWorkload.Status == rt.WorkloadStatusStopped {
       contextMsg := fmt.Sprintf("workload %s not found in runtime, marking as unhealthy", workloadName)
       if err := f.SetWorkloadStatus(ctx, workloadName, rt.WorkloadStatusUnhealthy, contextMsg); err != nil {
           return core.Workload{}, err
       }
       fileWorkload.Status = rt.WorkloadStatusUnhealthy
   }

Why runtime ownership is not tracked

None of the data structures carry runtime type information:

Data Structure	Runtime field?	Notes
workloadStatusFile (file_status.go:186-192)	No	Only status, context, timestamps, PID
core.Workload (pkg/core/workload.go:15-52)	No	Only a Remote bool for remote vs container
RunConfig (pkg/runner/config.go)	No	Deployer field has json:"-" -- not persisted
Container labels (pkg/labels/labels.go:46-53)	No	No runtime-type label
runtime.ContainerInfo (pkg/container/runtime/types.go:49-69)	No	No runtime identifier

Affected Commands

All commands that call ListWorkloads or GetWorkload have write-on-read side effects:

Command	Entry point	Side effect
thv list	file_status.go:275 via mergeRuntimeAndFileWorkloads	Marks all workloads not in active runtime as UNHEALTHY
thv status <name>	file_status.go:195 via validateRunningWorkload	Marks specific workload as UNHEALTHY if not in active runtime
thv stop --all / --group	Lists workloads first	Marks non-active-runtime workloads UNHEALTHY before stopping
thv restart --all / --group	Lists workloads first	Same as stop

What is NOT affected

• go-microvm recoverState() does NOT harm podman processes. The go-microvm state directory (~/.config/toolhive/gomicrovm/vms/) is completely separate from podman state, and cleanupOrphanedRunner() only reads PIDs from go-microvm-specific state files.
• thv serve does not do periodic reconciliation that would trigger this -- the damage only happens on explicit CLI commands.
• Marking unhealthy does not kill containers or proxy processes -- it only corrupts the status file. The actual workloads keep running.

Possible Fix Directions

1. Track runtime ownership: Add a runtime-type field to status files (and/or container labels / RunConfig). During reconciliation, skip workloads that belong to a different runtime.

2. Read-only list path: Separate the read-only listing from the write-on-read validation. thv list should not mutate state as a side effect.

3. Multi-runtime aware reconciliation: Query all available runtimes during reconciliation, not just the active one. Only mark a workload unhealthy if it's missing from the runtime it was created with.

[JAORMX]

Found a second trigger of this write-on-read corruption that needs no second runtime: ListWorkloads snapshotted the runtime before the status files, so any thv list running while a workload starts could observe "file running, runtime missing" and permanently persist unhealthy for a healthy workload — handleRuntimeMissing writes it and validateWorkloadInList never revalidates a non-running file status, so nothing recovers it. This is what intermittently failed the vmcp and core e2e shards; it is reproduced deterministically in TestFileStatusManager_ListWorkloads_DoesNotPoisonWorkloadStartingDuringList and narrowed by #6076 (snapshot reorder plus a guarded write).

#6076 removes that trigger but does not fix the root cause documented here — the list path still writes on read — so the durable fix remains the read-only list path this issue proposes.

One detail from #6076 worth recording here, because it shows how sharp the edges are: the reorder alone would have introduced a new instance of this same class. With files read first, a slow thv list that brackets a thv rm (file read before the removing write, merge write after the file delete) would call SetWorkloadStatus(unhealthy) on a deleted file — and SetWorkloadStatus creates missing files, resurrecting the removed workload as a ghost unhealthy entry. The old ordering could not do that geometrically. #6076 therefore ships a guard that re-checks under the file lock and skips if the file is gone or no longer says running/stopped. Two writers remain unguarded (handleRuntimeMismatch, isProxyUnhealthy), though both fire only when the container is present so their exposure is unchanged — migrating them to the same guard is a sensible follow-up, and a further argument for removing writes from the read path entirely.