Evict vMCP sessions when a backend is dropped

[tgrunnagle]

Summary

When a backend is removed from a vmcp.DynamicRegistry (a generation swap), any
already-open per-session connection to that dropped backend — most visibly a
long-lived SSE server-push stream — persisted until the owning client session
ended. New routing stopped correctly, but nothing reclaimed the live connection,
leaking a resource for the remaining lifetime of every affected session.

This reclaims those connections promptly by reusing the existing
lazy-eviction / RestoreSession machinery, rather than mutating the read-only
MultiSession in place (vMCP anti-pattern #10, "reconstruct, don't mutate"):

• ValidatingCache.RemoveMatching — predicate-based bulk eviction. The LRU's
  eviction callback only buffers evicted entries; every mutating path drains the
  buffer and invokes onEvict after releasing the cache lock, so slow teardown
  work in onEvict (such as closing a hung backend connection) never runs under
  the lock and cannot stall concurrent session creation on other keys.
• SessionManager.EvictStaleSessions — evicts every live session holding a
  backend absent from the current registry; onEvict closes its connections and
  the next request rebuilds the session minus the dropped backend via
  RestoreSession.
• reconcileSessionsOnRegistryChange — a background loop that watches the
  DynamicRegistry version counter and evicts on change, independent of status
  reporting, and a no-op for static registries.

Closes #6546

Type of change

• Bug fix
• New feature
• Refactoring (no behavior change)
• Dependency update
• Documentation
• Other (describe):

Test plan

• Unit tests (task test)
• Linting (task lint-fix)

Unit tests added for:

• RemoveMatching — happy path, no-match no-op, phase-2 re-check guard against a
  concurrent replacement, a -race concurrency smoke test, and a slow-onEvict
  test proving an unaffected key's Set completes while a slow eviction is in
  flight (i.e. onEvict is off the lock).
• EvictStaleSessions — evicts stale sessions while leaving unaffected sessions
  in place.
• reconcileSessionsOnRegistryChange — evicts on a registry version change and
  returns immediately for a static (non-dynamic) registry.

task lint-fix is clean and the affected packages' tests pass, including full
-race runs of pkg/cache and pkg/vmcp/server.

Changes

File	Change
pkg/cache/validating_cache.go	Add RemoveMatching predicate-based bulk eviction; run onEvict off the cache lock via buffer-and-drain on all eviction paths.
pkg/vmcp/session/factory.go	Add ParseBackendIDs, the single decoder for the MetadataKeyBackendIDs wire format; use it in filterBackendsByStoredIDs.
pkg/vmcp/server/sessionmanager/session_manager.go	Add EvictStaleSessions + referencesMissingBackend (using ParseBackendIDs); downgrade routine eviction logs from WARN/INFO to DEBUG.
pkg/vmcp/server/session_manager_interface.go	Add EvictStaleSessions to the SessionManager interface.
pkg/vmcp/server/session_reconcile.go	New background reconcile loop watching the DynamicRegistry version; poll interval injected as a parameter.
pkg/vmcp/server/server.go	Start/stop the reconcile loop for dynamic registries during Start.

Does this introduce a user-facing change?

No. Behavior is internal to vMCP session management; there is no API or
configuration change.

Special notes for reviewers

Accepted deviation from the issue's acceptance criteria. The issue lists
"Other backends in the same session are unaffected" as a done-when criterion.
That is not literally met here. The maintainer explicitly chose the "A'" approach
(reuse eviction + RestoreSession) over the issue's Option A (a targeted
per-backend CloseBackend), so evicting a session rebuilds it and its surviving
backends briefly disconnect and reconnect on the next request. This trade-off was
deliberately accepted to avoid adding a mutating method to the read-only
MultiSession (anti-pattern #10). Recommend updating the issue's acceptance
criteria to record this decision.

Residual window. Eviction operates on sessions already in the cache. A
session whose RestoreSession is in flight when the drop commits may open a
connection to the soon-dropped backend and escape a given pass; its stale
connection is then reclaimed on the next registry change, on checkSession
metadata drift, or ultimately at session end — i.e. it never regresses past the
original session-lifetime bound. This is the accepted best-effort of a polling
model and is documented at the method.

Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.77%. Comparing base (2533388) to head (4183f53).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6549      +/-   ##
==========================================
+ Coverage   78.69%   78.77%   +0.08%     
==========================================
  Files         777      778       +1     
  Lines       76797    76887      +90     
==========================================
+ Hits        60432    60571     +139     
+ Misses      16360    16311      -49     
  Partials        5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tgrunnagle]

Multi-Agent Consensus Review

Agents consulted: concurrency-correctness-reviewer, security-resource-lifecycle-reviewer, test-coverage-quality-reviewer, general-code-quality-reviewer

Consensus Summary

#	Finding	Consensus	Severity	Action
1	New test races with a pre-existing test on shared versionPollInterval	10/10	HIGH	Fix
2	Cache-wide lock held during synchronous sess.Close(), now targeted at just-dropped backends	9/10	HIGH	Fix
3	Duplicated comma-separated backend-ID parsing logic across two packages	8/10	MEDIUM	Fix
4	EvictStaleSessions INFO log contradicts this PR's own adjacent WARN\u2192DEBUG rationale	8/10	MEDIUM	Fix
5	RemoveMatching doc comment overclaims the locking guarantee; no test exercises a slow onEvict	7/10	LOW	Fix

Overall

This PR reclaims per-session backend connections in vMCP (notably lingering SSE streams) when a backend is dropped from a DynamicRegistry, via RemoveMatching bulk cache eviction, EvictStaleSessions, and a background reconciliation loop. The approach is sound and deliberately reuses the existing lazy-eviction/RestoreSession machinery instead of adding a mutating method to MultiSession, consistent with the reconstruct-don't-mutate convention. The core logic was traced end-to-end and holds up: the phase-1/phase-2 re-check in RemoveMatching correctly closes its TOCTOU window, the backend-ID identifier space matches between registry membership and session metadata, and a dropped backend's ID is genuinely excluded from session metadata on the next restore.

However, the new test in session_reconcile_test.go introduces a real data race against a pre-existing test in status_reporting_test.go (both mutate the shared package-level versionPollInterval under t.Parallel()) — reproduced directly with go test -race, and confirmed independently by three review agents. This breaks task test as submitted and needs to be fixed before merge. Separately, EvictStaleSessions's bulk eviction runs RemoveMatching's per-entry cache-lock-held Close() specifically against backends that were just dropped — the scenario most likely to involve a slow or unresponsive backend — which can stall the whole node-local session cache for up to one request-timeout per stale session in a reconciliation pass. That lock-holding pattern already exists for single-entry eviction elsewhere in the cache, but this PR meaningfully widens its blast radius; worth addressing here or as an immediate follow-up. The remaining findings are minor: duplicated backend-ID parsing logic across two packages, an INFO log level that contradicts this PR's own adjacent reasoning for a WARN→DEBUG downgrade a few lines above, and a doc comment that slightly overstates the locking guarantee it documents.

---

Generated with Claude Code

[jhrozek]

Reviewed via multi-agent pass (concurrency/code-quality, ToolHive conventions, security) covering the cache buffer-then-drain eviction rework and the vMCP session reconciliation on backend removal.

No blocking issues found:

• Lock ordering (mu -> evictMu) verified one-directional, no cycle
• RemoveMatching's phase-2 re-check correctly closes the TOCTOU window against concurrent Set
• SessionManager.EvictStaleSessions has a single implementer, updated atomically with test stubs
• Version-poll pattern matches the existing periodicStatusReporting idiom
• No security regression: eviction only reclaims idle cached connections (not a per-request auth re-check, which didn't exist before either); onEvict runs sequentially, no thundering-herd risk

Minor non-blocking note: RemoveMatching's phase-1 scan holds the cache lock for the full Keys()+Peek pass rather than per-key, so hold time scales with cache size, not match count — fine given this is documented as infrequent bulk reconciliation, not a hot path.