Skip to content

[Conformance] Report expected client terminal failures quietly#463

Merged
koic merged 1 commit into
modelcontextprotocol:mainfrom
koic:work_conformance_client_merged
Jul 19, 2026
Merged

[Conformance] Report expected client terminal failures quietly#463
koic merged 1 commit into
modelcontextprotocol:mainfrom
koic:work_conformance_client_merged

Conversation

@koic

@koic koic commented Jul 19, 2026

Copy link
Copy Markdown
Member

Motivation and Context

Negative auth scenarios such as auth/resource-mismatch and auth/scope-retry-limit are designed to drive the client into a terminal failure: the SDK correctly refuses a mismatched protected-resource metadata resource and stops after the scope step-up retry limit. The conformance harness asserts on the observed protocol exchange and already marks these scenarios as passed, but the client script had no top-level rescue, so each run leaked a raw Ruby backtrace to stderr inside a noisy "Client exited with code 1" block.

This change wraps the connect and scenario dispatch in a rescue that reports the expected AuthorizationError, RequestHandlerError, and Faraday::Error outcomes as a concise one-line reason on stderr and exits with status 0. The harness echoes the client's output only when the exit status is non-zero and does not consult the exit status for its verdict, so the expected failures no longer produce any output block at all; the one-line reason remains visible in manual runs and in the harness --output-dir stderr artifact. Any unexpected error still raises with a full backtrace and a failing exit status so genuine SDK bugs remain visible.

How Has This Been Tested?

Ran bundle exec rake conformance: the server suite passes 40/40 and the client suite passes 280/280 with no "Client exited with code" or "Stderr:" blocks in the output. Exercised auth/resource-mismatch and auth/scope-retry-limit individually via Conformance::ClientRunner; both pass with clean output.

Breaking Changes

None. Only the conformance client script output changes; no library code is affected.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

## Motivation and Context

Negative auth scenarios such as `auth/resource-mismatch` and `auth/scope-retry-limit` are
designed to drive the client into a terminal failure: the SDK correctly refuses
a mismatched protected-resource metadata `resource` and stops after the scope step-up retry limit.
The conformance harness asserts on the observed protocol exchange and already marks these scenarios as passed,
but the client script had no top-level rescue, so each run leaked a raw Ruby backtrace to stderr inside a noisy
"Client exited with code 1" block.

This change wraps the connect and scenario dispatch in a rescue that reports the expected `AuthorizationError`,
`RequestHandlerError`, and `Faraday::Error` outcomes as a concise one-line reason on stderr and exits with status 0.
The harness echoes the client's output only when the exit status is non-zero and does not consult the exit status
for its verdict, so the expected failures no longer produce any output block at all; the one-line reason remains
visible in manual runs and in the harness `--output-dir` stderr artifact. Any unexpected error still raises with
a full backtrace and a failing exit status so genuine SDK bugs remain visible.

## How Has This Been Tested?

Ran `bundle exec rake conformance`: the server suite passes 40/40 and the client suite passes 280/280
with no "Client exited with code" or "Stderr:" blocks in the output. Exercised `auth/resource-mismatch`
and `auth/scope-retry-limit` individually via `Conformance::ClientRunner`; both pass with clean output.

## Breaking Changes

None. Only the conformance client script output changes; no library code is affected.
@koic
koic merged commit 72400ca into modelcontextprotocol:main Jul 19, 2026
11 checks passed
@koic
koic deleted the work_conformance_client_merged branch July 19, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants