bug(operator): VirtualMCPServer inline config.telemetry silently ignored; GET /metrics returns MCP 406

[RaviTharuma]

Bug description

On operator-managed VirtualMCPServer (observed on v0.42.1), setting:

spec:
  config:
    telemetry:
      enablePrometheusMetricsPath: true

is accepted by the API and stored on the CR, but the operator does not
render it into the vmcp ConfigMap. The running process never registers a
Prometheus handler on /metrics.

GET /metrics with Accept: text/plain (or Prometheus default) then falls
through to the MCP streamable-HTTP handler and returns:

HTTP/1.1 406 Not Acceptable
Content-Type: application/json

{"jsonrpc":"2.0","id":"server-error","error":{"code":-32600,"message":"Not Acceptable: Client must accept text/event-stream"}}

That makes Prometheus ServiceMonitors look broken even though the CR “enabled”
metrics. This blocked collecting Go runtime memory series requested on #5860.

Root cause (code)

cmd/thv-operator/pkg/vmcpconfig/converter.go normalizeTelemetry only reads
spec.telemetryConfigRef → MCPTelemetryConfig. Inline spec.config.telemetry
returns nil after #4819 (“Remove deprecated inline telemetry field from CRDs”),
while:

• the CRD still exposes spec.config.telemetry (embedded vmcp config type)
• docs still said the inline field “still works” for VirtualMCPServer
• there is no status condition or validation error when inline is set without a ref

When config.Telemetry is nil at serve time, pkg/vmcp/server/server.go skips
mux.Handle("/metrics", …) and the MCP catch-all owns the path.

Steps to reproduce

1. Install operator + CRDs v0.42.1 (or current main with the ignore behavior).

2. Create a VirtualMCPServer with only inline:

   apiVersion: toolhive.stacklok.dev/v1beta1
   kind: VirtualMCPServer
   metadata:
     name: demo
   spec:
     groupRef:
       name: <existing-group>
     incomingAuth:
       type: anonymous
     config:
       telemetry:
         enablePrometheusMetricsPath: true

3. Inspect the operator-managed ConfigMap (*-vmcp-config): no telemetry: key.

4. curl -H 'Accept: text/plain' http://<vmcp-svc>:4483/metrics → 406 JSON-RPC as above.

5. curl http://<vmcp-svc>:4483/health → 200 {"status":"ok"} (health is registered).

Expected behavior

Either:

1. Documented preferred path works and is discoverable: requiring
   spec.telemetryConfigRef + MCPTelemetryConfig is fine, but inline must
   not be a silent no-op (status Warning condition and/or validation), and docs
   must not claim inline still works for the operator; or
2. Deprecated inline continues to apply until removed from the CRD, with a
   deprecation log / condition pointing at telemetryConfigRef.

When Prometheus path is actually enabled, GET /metrics should return
Prometheus text (HTTP 200), not MCP 406.

Actual behavior

Silent ignore of inline telemetry → no /metrics route → MCP 406 on scrape.
No condition on the VirtualMCPServer warns that telemetry was dropped.

Environment

• ToolHive operator + vmcp image: v0.42.1
• Single-node Kubernetes
• Four VirtualMCPServers (backend counts ~12–17); no private hostnames in this report

Workaround

Use the shared config resource:

apiVersion: toolhive.stacklok.dev/v1beta1
kind: MCPTelemetryConfig
metadata:
  name: vmcp-prometheus
spec:
  prometheus:
    enabled: true
---
apiVersion: toolhive.stacklok.dev/v1beta1
kind: VirtualMCPServer
metadata:
  name: demo
spec:
  telemetryConfigRef:
    name: vmcp-prometheus
    serviceName: demo
  # ...groupRef, incomingAuth, etc.

Related

• vmcp: all VirtualMCPServer instances grow 2-3x in memory over ~24h regardless of client activity #5860 (memory growth) — maintainer asked for Prometheus series; blocked until
  metrics actually scrape
• Remove deprecated inline telemetry field from CRDs #4819 (intentional removal of inline telemetry shims) — left CRD field + docs
  claiming “still works”, producing this footgun

Proposed fix

Restore deprecated inline fallback when TelemetryConfigRef is unset (ref
still wins), log a migration warning, fix docs, and cover with unit tests.
PR forthcoming.

[ChrisJBurns]

@RaviTharuma Thanks for the issue, although not sure if this is something we want to really fix given its deprecated and the use of the supported method is working.. As you described in the PR description the workaround is the use of MCPTelemetryConfig - in fact this isn't just a workaround, but the long term solution.

Is there any other reason I may be missing that would require us to fix the deprecated code?

[reyortiz3]

Hello @RaviTharuma, thanks for filing this issue. I also agree with @ChrisJBurns, I think we shouldn't bring back something that's being deprecated, so I would suggest to close #6277 and rather implement the option 1 above:

Documented preferred path works and is discoverable: requiring
spec.telemetryConfigRef + MCPTelemetryConfig is fine, but inline must
not be a silent no-op (status Warning condition and/or validation), and docs
must not claim inline still works for the operator; or

Would you be interested in opening a PR for this?

[RaviTharuma]

You're not missing another reason. Restoring the deprecated inline path was only to stop the silent no-op /metrics 406; the supported spec.telemetryConfigRef + MCPTelemetryConfig path is the right long-term solution.

I'll close #6277 and follow option 1: keep inline telemetry a no-op for the operator, but make that visible (Warning condition and/or validation) and stop docs from claiming inline still works for operator VirtualMCPServer.

[RaviTharuma]

Agreed — if the supported path already works and the old one is deprecated, closing this rather than restoring deprecated behavior.