[Conformance] Report expected client terminal failures quietly#463
Merged
koic merged 1 commit intoJul 19, 2026
Merged
Conversation
## Motivation and Context Negative auth scenarios such as `auth/resource-mismatch` and `auth/scope-retry-limit` are designed to drive the client into a terminal failure: the SDK correctly refuses a mismatched protected-resource metadata `resource` and stops after the scope step-up retry limit. The conformance harness asserts on the observed protocol exchange and already marks these scenarios as passed, but the client script had no top-level rescue, so each run leaked a raw Ruby backtrace to stderr inside a noisy "Client exited with code 1" block. This change wraps the connect and scenario dispatch in a rescue that reports the expected `AuthorizationError`, `RequestHandlerError`, and `Faraday::Error` outcomes as a concise one-line reason on stderr and exits with status 0. The harness echoes the client's output only when the exit status is non-zero and does not consult the exit status for its verdict, so the expected failures no longer produce any output block at all; the one-line reason remains visible in manual runs and in the harness `--output-dir` stderr artifact. Any unexpected error still raises with a full backtrace and a failing exit status so genuine SDK bugs remain visible. ## How Has This Been Tested? Ran `bundle exec rake conformance`: the server suite passes 40/40 and the client suite passes 280/280 with no "Client exited with code" or "Stderr:" blocks in the output. Exercised `auth/resource-mismatch` and `auth/scope-retry-limit` individually via `Conformance::ClientRunner`; both pass with clean output. ## Breaking Changes None. Only the conformance client script output changes; no library code is affected.
atesgoral
approved these changes
Jul 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Negative auth scenarios such as
auth/resource-mismatchandauth/scope-retry-limitare designed to drive the client into a terminal failure: the SDK correctly refuses a mismatched protected-resource metadataresourceand stops after the scope step-up retry limit. The conformance harness asserts on the observed protocol exchange and already marks these scenarios as passed, but the client script had no top-level rescue, so each run leaked a raw Ruby backtrace to stderr inside a noisy "Client exited with code 1" block.This change wraps the connect and scenario dispatch in a rescue that reports the expected
AuthorizationError,RequestHandlerError, andFaraday::Erroroutcomes as a concise one-line reason on stderr and exits with status 0. The harness echoes the client's output only when the exit status is non-zero and does not consult the exit status for its verdict, so the expected failures no longer produce any output block at all; the one-line reason remains visible in manual runs and in the harness--output-dirstderr artifact. Any unexpected error still raises with a full backtrace and a failing exit status so genuine SDK bugs remain visible.How Has This Been Tested?
Ran
bundle exec rake conformance: the server suite passes 40/40 and the client suite passes 280/280 with no "Client exited with code" or "Stderr:" blocks in the output. Exercisedauth/resource-mismatchandauth/scope-retry-limitindividually viaConformance::ClientRunner; both pass with clean output.Breaking Changes
None. Only the conformance client script output changes; no library code is affected.
Types of changes
Checklist