TokenValidator JWKS registration deadlocks permanently after the first ErrNotReady

[danbarr]

Bug description

Correction (edited): the original version of this issue claimed shipped ToolHive releases were affected and left the 5-second first-fetch failure unexplained. Both were wrong. The trigger is a jwx v3.1.0 behavior change that bypasses ToolHive's CA-aware HTTP client for JWKS fetches, tracked separately in #6219. OSS releases pin jwx v3.0.13 and are not affected as shipped. This issue now tracks the retry deadlock in ensureJWKSRegistered, which is what converted that single failed first fetch into a permanently dead validator.

ensureJWKSRegistered (pkg/auth/token.go:698) treats any error from Register as fatal and deliberately leaves jwksRegistered = false so the next call retries:

if err := v.jwksClient.Register(registrationCtx, v.jwksURL); err != nil {
    v.jwksRegistrationErr = fmt.Errorf("failed to register JWKS URL: %w", err)
    // Do NOT set jwksRegistered = true -- allow retry on next call
    return v.jwksRegistrationErr
}

But one of the errors it can get back is httprc.ErrNotReady, which is explicitly documented as non-fatal (httprc v3.0.6, errors.go):

When Add() returns this error, the resource IS in the backend's resource map and will continue to be fetched periodically in the background according to the refresh interval. The application can safely proceed - the resource data may become available later when a fetch succeeds.

Since the resource is already in httprc's map, every retry returns resource already exists (backend.go:29-33 checks the map before anything else) and jwksRegistered never becomes true. httprc's own test suite codifies exactly this trap (httprc_test.go, "retry logic must distinguish registration failures from ErrNotReady"). The retry logic is what converts a single slow or failed first fetch into a permanently dead validator.

Steps to reproduce

Any condition that keeps the first JWKS fetch from completing within the 5s registrationCtx budget triggers this: the jwx v3.1.0 client bypass in #6219, a slow IdP, or a transient network failure coinciding with the first authenticated request. Concretely:

1. Run a VirtualMCPServer (or any proxy using pkg/auth OIDC middleware) with incomingAuth.type: oidc against an external issuer, in a state where the first JWKS fetch fails or exceeds 5s.
2. Send a request with a Bearer token; it fails with resource registered but not ready.
3. Restore JWKS reachability and send more requests: every one fails with resource already exists, forever.

Expected behavior

ErrNotReady is handled as registered-but-pending: the validator proceeds and lets Lookup pick the key set up once a background fetch succeeds (jwk.Cache.Lookup returns a clean resource %q is not ready error until then). A transient first-fetch failure resolves itself on a later request.

Actual behavior

First request (5s, the registrationCtx budget):

Invalid token: failed to parse token: token is unverifiable: error while executing keyfunc:
JWKS registration failed: failed to register JWKS URL: failed to add resource to httprc.Client:
resource registered but not ready: context deadline exceeded

Every subsequent request, for the life of the pod:

Invalid token: failed to parse token: token is unverifiable: error while executing keyfunc:
JWKS registration failed: failed to register JWKS URL: failed to add resource to httprc.Client:
resource already exists

Environment

• Originally reproduced on stacklok-enterprise/vmcp:0.11.0, which bundles jwx v3.1.0 and so hits the trigger in JWKS fetches bypass the CA-aware HTTP client on jwx v3.1.0 #6219. The deadlock code path itself is identical in OSS main (jwx v3.0.13 / httprc v3.0.6 pins).

Additional context

Suggested fix, in ensureJWKSRegistered:

• Treat errors.Is(err, httprc.ErrNotReady()) as success: set jwksRegistered = true and let Lookup surface a not-yet-ready key set.
• Treat errors.Is(err, httprc.ErrResourceAlreadyExists()) as already-registered rather than an error, as defense in depth if the flag and httprc's map ever disagree.

Both sentinels wrap cleanly through jwx's %w chain, so errors.Is works from ToolHive's side.

[danbarr]

Root cause found and the issue body rewritten accordingly. The original claim that shipped OSS releases are affected was wrong: the trigger is a jwx v3.1.0 behavior change that bypasses ToolHive's CA-aware HTTP client for JWKS fetches, now split out as #6219. This issue now tracks only the ErrNotReady retry deadlock, which remains a genuine latent bug in pkg/auth and is what turned one failed fetch into a permanently dead validator.