MCPRemoteProxy forwards `X-Forwarded-Proto: http` to TLS upstreams, causing infinite redirect loops

[aron-muon]

Summary

When MCPRemoteProxy runs behind a TLS-terminating load balancer (e.g. an AWS ALB / any ingress that terminates TLS and forwards to the pod over plain HTTP), the transparent proxy sends X-Forwarded-Proto: http to the remote upstream. Upstreams that enforce HTTPS by redirecting when X-Forwarded-Proto != https then return a 301 to the same URL on every request, and the proxy follows it until it hits the redirect cap — so the upstream MCP session never initializes and all tool traffic fails.

This is the same class of bug that X-Forwarded-Host was already removed for in setXForwardedHeaders (third-party servers echoing the forwarded value into redirect URLs), but X-Forwarded-Proto is left intact and triggers it just as effectively.

Environment

• toolhive operator + proxyrunner v0.30.0
• MCPRemoteProxy, transport: streamable-http
• Kubernetes, fronted by an ALB that listens on HTTPS:443, terminates TLS, and forwards to the proxy pod over HTTP (target group is plain HTTP — the standard pattern).
• Remote upstream is a third-party MCP server whose gateway (nginx) does an exact-match if ($http_x_forwarded_proto = "http") { return 301 https://...; }.

Root cause

pkg/transport/proxy/transparent/transparent_proxy.go:

// ... X-Forwarded-For and X-Forwarded-Proto are kept so remote backends can
// still log the client IP and scheme. ...
func (p *TransparentProxy) setXForwardedHeaders(pr *httputil.ProxyRequest) {
	pr.SetXForwarded()
	if p.isRemote {
		pr.Out.Header.Del("X-Forwarded-Host")
	}
}

httputil.ProxyRequest.SetXForwarded() derives X-Forwarded-Proto from pr.In.TLS — i.e. the pod-local inbound connection, which is plain HTTP because the LB already terminated TLS. The original client→LB scheme (https, which the LB does pass in the inbound X-Forwarded-Proto header) is discarded, since httputil strips client-supplied X-Forwarded-* before Rewrite runs. So a remote upstream always receives X-Forwarded-Proto: http.

xforwarded_test.go actually encodes this as expected behavior:

assert.Equal(t, "http", pr.Out.Header.Get("X-Forwarded-Proto"))

Reproduction

1. Deploy an MCPRemoteProxy for a remote upstream that 301-redirects HTTP→HTTPS based on X-Forwarded-Proto.
2. Put it behind any TLS-terminating LB/ingress (ALB, etc.) so the pod is reached over HTTP.
3. Observe repeated proxy logs:

   following HTTP redirect from remote MCP server; consider updating the server URL
   status=301 from=https://upstream/mcp to=https://upstream/mcp redirect_number=1..10
   

   followed by readiness initialize not successful and no usable upstream session.

Direct confirmation against such an upstream:

Request to upstream /mcp	Result
X-Forwarded-Proto: http	301 (loops)
X-Forwarded-Proto: https	passes (401/200)
X-Forwarded-Proto absent	passes

Why existing knobs don't help

• trustProxyHeaders only influences the SSE response-processor's self-URL/PRM construction (sse_response_processor.go); it does not change the outbound X-Forwarded-Proto (verified: PRM is correct, the loop persists with it both true and false).
• headerForward.addPlaintextHeaders rejects X-Forwarded-Proto as a restricted header (failed to create middleware of type header-forward: header "X-Forwarded-Proto" is restricted and cannot be configured for forwarding).

So there is currently no configuration-level workaround; the value is hard-coded by the inbound connection scheme.

Proposed fix

For remote upstreams, set X-Forwarded-Proto to the scheme of the actual upstream connection (the remoteURL/target scheme), symmetric with the existing X-Forwarded-Host removal:

func (p *TransparentProxy) setXForwardedHeaders(pr *httputil.ProxyRequest) {
	pr.SetXForwarded()
	if p.isRemote {
		pr.Out.Header.Del("X-Forwarded-Host")
		// The inbound hop is whatever scheme the proxy was reached on (often
		// plain HTTP behind a TLS-terminating LB). Reflect the scheme of the
		// real upstream connection so upstreams that redirect on
		// X-Forwarded-Proto != https don't loop.
		if u, err := url.Parse(p.targetURI); err == nil && u.Scheme != "" {
			pr.Out.Header.Set("X-Forwarded-Proto", u.Scheme)
		}
	}
}

Alternatively, honor the trusted inbound X-Forwarded-Proto (e.g. from the LB) when trustProxyHeaders is enabled, instead of always overwriting it from pr.In.TLS.

Happy to open a PR if the direction looks right.

[jstar0]

I'd like to work on this if the proposed direction is acceptable. I can keep the change focused on the remote proxy X-Forwarded-Proto handling and add the corresponding regression coverage.

I'll wait for assignment or maintainer confirmation before starting the PR, per the contribution guide.

[amirejaz]

@aron-muon Thanks for the thorough write-up and root-cause analysis, assigning to you since you've already got the fix sketched out. And thanks to @jstar0 for volunteering too, appreciate the enthusiasm!