Skip to content

Metrics migration steps aren't performable/verifiable under the operator #6434

Description

@amirejaz

Summary

Step 1 of the migration in docs/observability.md says to point a scraper at the
diagnostics port and confirm metrics arrive. Under the operator today that step isn't
actually performable: mcpserver_controller.go:1467 declares only the proxy
containerPort, and :1561 only the proxy ServicePort. The diagnostics listener is
genuinely up on 9464 (or whatever prometheusPort resolves to) — it's just
undiscoverable. A PodMonitor has no named port to key on, and prometheusPort is
intentionally absent from MCPTelemetryConfigSpec (see the drift-table justification
in telemetry_drift_test.go), so there's no field to even declare it explicitly.

Separately, step 2 ("set metricsOnTransportPort: false and confirm nothing else was
still scraping it") isn't verifiable either. The startup WARN reports configuration,
not usage — it fires because the flag is on, not because a scraper actually hit the
endpoint. And the transport-port /metrics mux registration outranks the /
catch-all, so those requests bypass the telemetry middleware entirely; there's no
counter anywhere that would tell you whether traffic there is real.

Proposed fixes

  1. Discoverability: add a named containerPort (e.g. metrics) on the workload
    pod when EnablePrometheusMetricsPath is set, so a PodMonitor can target it by
    name. Leave the Service alone — deliberately, per the existing docs guidance that
    the diagnostics port must not be routed through anything internet-facing.
  2. Verifiability: wrap the transport-port /metrics handler (mounted at
    pkg/runner/runner.go around the mountPrometheusHandlerOnTransportPort call)
    with a request counter, so step 2 has something concrete to check — "requests to
    the transport-port copy since startup: N" — before an operator turns it off.

Neither blocks #6371; raised there as informational.

Raised by @jhrozek on #6371:
#6371 (review)

Part of #6271. Related: #6384.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageIssue needs initial triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions