Conversation
One validator served both providers, and it was wrong in both directions.
Too strict for OpenRouter. It required a numeric `confidence` on every
choice and score answer. OpenRouter's published schema (DecisionsChoiceAnswer,
DecisionsScoreAnswer in openrouter.ai/openapi.json) requires only `type`
plus `choice`/`score`; confidence is optional. A response the gateway
considers valid raised InvalidResponse.
Too lax for Typesafe. docs.typesafe.ai/api documents `probabilities` and
`confidence` as required on choice, and those plus `legend` on score. When
any of them was missing or the wrong shape the client substituted `{}` and
returned a typed answer with an empty distribution, so code thresholding
on probabilities read an empty map as a legitimate answer.
And too lax for everyone about values. `is_a?(Numeric)` accepts a `noul`
of 2, and JSON parses `1e999` into Float::INFINITY, which is Numeric and
survives every check downstream. A `choice` was taken as given even when
it was not one of the labels the question offered -- the one place a
wrong string does real damage, since that is what the application routes on.
So: providers declare `required_answer_fields`, the client enforces that
per provider, missing-but-optional comes back nil or {} while
present-but-malformed always raises, numbers must be finite and in range,
and a choice must be one the question offered. InvalidResponse now names
the field rather than only the question id.
Two existing tests asserted the confidence-is-required behaviour and one
asserted the silent fallback to {}; they now assert the contract each
provider actually publishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One validator served both providers, and it was wrong in both directions.
Too strict for OpenRouter. It required a numeric
confidenceon every choice and score answer. OpenRouter's published schema (DecisionsChoiceAnswer,DecisionsScoreAnswerinopenrouter.ai/openapi.json) requires onlytypepluschoice/score— confidence is optional. A response the gateway considers valid raisedInvalidResponse.Too lax for Typesafe.
docs.typesafe.ai/apidocumentsprobabilitiesandconfidenceas required on choice, and those pluslegendon score. When any was missing or the wrong shape the client substituted{}and returned a typed answer with an empty distribution, so code thresholding on probabilities read an empty map as a legitimate answer:And too lax for everyone about values.
is_a?(Numeric)accepts anoulof2, and JSON parses1e999intoFloat::INFINITY, which isNumericand survives every check downstream. Achoicewas taken as given even when it was not one of the labels the question offered — the one place a wrong string does real damage, since that's what the application routes on.The fix
Providers declare
required_answer_fields; the client enforces that per provider. Missing-but-optional comes backnilor{}, present-but-malformed always raises. Numbers must be finite and in range (noul,confidenceand probability values within 0..1, with a small tolerance for float slop). Achoicemust be one the question offered.InvalidResponsenow names the field, not just the question id.A provider naming a field the client doesn't model raises
ConfigurationErrorat construction, so a hand-written provider can't declare a guarantee nothing enforces.Behaviour change worth flagging:
Answers::Choice#confidenceandAnswers::Score#confidencecan now benilon OpenRouter. That's the honest answer for a field the provider doesn't guarantee, and better than rejecting valid responses — but it is a change.Tests
test/answers_test.rb, 22 cases, plus three existing tests updated: two asserted the confidence-is-required behaviour and one asserted the silent fallback to{}. They now assert the contract each provider actually publishes.This branch bundles three related defects because they're one 30-line method; splitting them would produce PRs that conflict head-on. Happy to split if you'd rather.
Draft: part of a security and API-coverage audit, opened for reference rather than as a request for immediate review. Independent of the other branches, each off
main. Suite green on Ruby 3.2.11, 3.3.8 and 3.4.8.🤖 Generated with Claude Code