Skip to content

fix: validate answers against each provider's actual contract - #8

Draft
simonx1 wants to merge 1 commit into
obie:mainfrom
simonx1:fix/answer-contract-validation
Draft

simonx1 wants to merge 1 commit into
obie:mainfrom
simonx1:fix/answer-contract-validation

Conversation

@simonx1

@simonx1 simonx1 commented Sep 18, 2026

Copy link
Copy Markdown

One validator served both providers, and it was wrong in both directions.

Too strict for OpenRouter. It required a numeric confidence on every choice and score answer. OpenRouter's published schema (DecisionsChoiceAnswer, DecisionsScoreAnswer in openrouter.ai/openapi.json) requires only type plus choice/score — confidence is optional. A response the gateway considers valid raised InvalidResponse.

Too lax for Typesafe. docs.typesafe.ai/api documents probabilities and confidence as required on choice, and those plus legend on score. When any was missing or the wrong shape the client substituted {} and returned a typed answer with an empty distribution, so code thresholding on probabilities read an empty map as a legitimate answer:

# 200 with {"type":"choice","choice":"billing","confidence":0.9}, no probabilities
response["dept"].probabilities   # => {}   (no error)

And too lax for everyone about values. is_a?(Numeric) accepts a noul of 2, and JSON parses 1e999 into Float::INFINITY, which is Numeric and survives every check downstream. A choice was taken as given even when it was not one of the labels the question offered — the one place a wrong string does real damage, since that's what the application routes on.

The fix

Providers declare required_answer_fields; the client enforces that per provider. Missing-but-optional comes back nil or {}, present-but-malformed always raises. Numbers must be finite and in range (noul, confidence and probability values within 0..1, with a small tolerance for float slop). A choice must be one the question offered. InvalidResponse now names the field, not just the question id.

A provider naming a field the client doesn't model raises ConfigurationError at construction, so a hand-written provider can't declare a guarantee nothing enforces.

Behaviour change worth flagging: Answers::Choice#confidence and Answers::Score#confidence can now be nil on OpenRouter. That's the honest answer for a field the provider doesn't guarantee, and better than rejecting valid responses — but it is a change.

Tests

test/answers_test.rb, 22 cases, plus three existing tests updated: two asserted the confidence-is-required behaviour and one asserted the silent fallback to {}. They now assert the contract each provider actually publishes.

This branch bundles three related defects because they're one 30-line method; splitting them would produce PRs that conflict head-on. Happy to split if you'd rather.


Draft: part of a security and API-coverage audit, opened for reference rather than as a request for immediate review. Independent of the other branches, each off main. Suite green on Ruby 3.2.11, 3.3.8 and 3.4.8.

🤖 Generated with Claude Code

One validator served both providers, and it was wrong in both directions.

Too strict for OpenRouter. It required a numeric `confidence` on every
choice and score answer. OpenRouter's published schema (DecisionsChoiceAnswer,
DecisionsScoreAnswer in openrouter.ai/openapi.json) requires only `type`
plus `choice`/`score`; confidence is optional. A response the gateway
considers valid raised InvalidResponse.

Too lax for Typesafe. docs.typesafe.ai/api documents `probabilities` and
`confidence` as required on choice, and those plus `legend` on score. When
any of them was missing or the wrong shape the client substituted `{}` and
returned a typed answer with an empty distribution, so code thresholding
on probabilities read an empty map as a legitimate answer.

And too lax for everyone about values. `is_a?(Numeric)` accepts a `noul`
of 2, and JSON parses `1e999` into Float::INFINITY, which is Numeric and
survives every check downstream. A `choice` was taken as given even when
it was not one of the labels the question offered -- the one place a
wrong string does real damage, since that is what the application routes on.

So: providers declare `required_answer_fields`, the client enforces that
per provider, missing-but-optional comes back nil or {} while
present-but-malformed always raises, numbers must be finite and in range,
and a choice must be one the question offered. InvalidResponse now names
the field rather than only the question id.

Two existing tests asserted the confidence-is-required behaviour and one
asserted the silent fallback to {}; they now assert the contract each
provider actually publishes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant