Add MCP conformance CI job for thv run proxy - #5806
Merged
Merged
Conversation
ToolHive proxies MCP servers, so spec compliance of its transport/proxy layer can be verified with the upstream MCP conformance suite. Add a per-PR job that packages the conformance repo's reference server dynamically (cloned at a pinned version, built into a container), runs it through `thv run --transport streamable-http`, and points the conformance tool at the proxied /mcp endpoint. The CLI proxy passes the active suite cleanly, so the baseline is empty and any drop is a real regression. A single CONFORMANCE_VERSION pins both the npm tool and the git server source (repo tags map 1:1 to npm versions). Also add a `task conformance` target for local runs. The k8s operator-proxy path is intentionally out of scope pending a fix to its Host-header rewriting. Closes #5803 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
JAORMX
approved these changes
Jul 15, 2026
1 task
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ToolHive is a proxy in front of MCP servers, so the fidelity of its
transport/proxy layer to the MCP spec can be measured with the upstream
MCP conformance suite.
This adds an automated, spec-authored compliance signal that regresses loudly —
feeding the protocol-currency epic (#5743).
conformancejob ine2e-tests.yml(reuses thebuild-binaryartifact; adds
actions/setup-node).test/conformance/run-conformance.shclones the conformance repo at a pinnedversion, builds its reference fixture server into a container, runs it through
thv run --transport streamable-http, and points the tool at the proxied/mcpendpoint, gated byexpected-failures.yaml(empty — the CLI proxy istransparent, so any drop is a real regression).
CONFORMANCE_VERSIONpins both the npm tool and the git serversource (repo tags map 1:1 to npm versions).
task conformancetarget for local runs.k8s operator-proxy conformance is intentionally out of scope pending a fix to its
Host-header rewriting; the stdio bridge needs a stdio fixture server. Both notedin
test/conformance/README.md.Closes #5803
Type of change
Test plan
Ran
task conformancelocally end-to-end: reference server built and started viathv run,activesuite reported 40/40 passed and "Baseline check passed".Verified trap cleanup removes the workload and the pre-run remove is idempotent.
Does this introduce a user-facing change?
No.
Generated with Claude Code