vMCP dispatch aborts instead of falling back to Legacy on an inconclusive Modern probe

[jhrozek]

Summary

dispatch in pkg/vmcp/client/client.go aborts the entire backend call when the Modern MCP revision probe returns an error, instead of falling back to the Legacy protocol path. A backend that is up and serving Legacy is therefore reported unhealthy whenever its Modern server/discover probe happens to land on a transient status.

Root cause

Merge 885e8b1 ("Handle per-backend MCP revision in vMCP (client-side dual protocol)") routed ListCapabilities through the new dispatch seam. On a revision-cache miss dispatch probes first, and treats any probe error as fatal:

rev, cached := h.cachedRevision(target.WorkloadID)
if !cached {
    probed, err := h.probeRevision(ctx, target)
    if err != nil {
        return wrapBackendError(err, target.WorkloadID, "probe revision")  // aborts; no Legacy fallback
    }
    rev = probed
}

probeRevision (client.go:1060) deliberately returns its error uncached for the two inconclusive classes, so as not to poison the revision cache:

• errModernAuth — HTTP 401/403/407
• errModernTransient — HTTP 408/429/5xx, mid-stream read failure, or any transport error (connection refused, timeout)

See pkg/vmcp/client/modern.go:273-281 for where those statuses become the sentinels.

An inconclusive probe is absence of evidence about the backend's revision. Turning it into a hard failure is wrong on its own terms — and it is a regression: before 885e8b1, ListCapabilities was a plain Legacy initialize + enumerate with no probe gate, so a backend that was serving simply succeeded.

Impact

ListCapabilities is both the health-check probe (pkg/vmcp/health/checker.go:99) and the startup aggregation call (pkg/vmcp/aggregator/default_aggregator.go:95). A single transient probe therefore marks a healthy backend unhealthy, which the existing readiness machinery amplifies:

• pkg/vmcp/server/status_reporting.go + pkg/vmcp/health/monitor.go gate vMCP Ready on all backends being Routable, so one stuck probe delays readiness.
• pkg/vmcp/core/core_vmcp.go (ListTools) filters unhealthy backends out of the tool list, producing an empty tools list.

(Those two are pre-existing amplifiers — see #3100 — not part of this defect.)

The concrete nondeterministic case: a ToolHive proxy is listening while the MCP container behind it is still starting, so it answers the probe with 502/503. The probe consumes that window and aborts before the Legacy initialize — which would have run after the container came up — is ever attempted.

Symptoms

vMCP e2e suites flake on bring-up:

• test/e2e/thv-operator/virtualmcp/ — VirtualMCPServer Yardstick Base and Global ExcludeAllTools fail in [BeforeAll], either while aggregating tools or waiting on the Ready condition.
• Core vmcp e2e bucket — failing-spec count swung 1 → 12 → 12 across three runs of an identical commit.

Fingerprint in a failing run's logs: "probe revision" plus backend_unavailable on a backend that is actually up.

Expected behaviour

An inconclusive probe should fall back to Legacy — the pre-dual-era default — without caching the revision, so that:

• a backend reachable on the Legacy path is served rather than failed;
• a transient outage never pins a revision;
• a genuinely Modern backend is corrected by the next call's re-probe.

A truly-down backend still fails fast, because the Legacy initialize hits the same dead socket.

Scope note

This does not cover the dial tcp …: connect: connection refused signature — there the socket is genuinely dead and the Legacy path fails identically; the health monitor's retry interval absorbs it. The defect here is specifically the 5xx/408/429/auth window that 885e8b1 made fatal.

Related

• Unblock main CI: fix vMCP nil-registry panic and CRD docs drift #5999 — earlier fixes for 885e8b1 fallout (nil-registry panic, CRD docs drift)
• vMCP backend revision cache has no success/TTL re-probe (stale on stateless redeploy) #5992 — revision cache has no success/TTL re-probe (adjacent, different defect)
• Add core health monitoring infrastructure for vmcp backends #3100 — readiness/health-filtering amplifiers