fix: validate answers against each provider's actual contract

CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## Unreleased
+
+- Answer validation follows each provider's published contract instead of one
+  shared guess. Typesafe documents `probabilities`, `confidence`, and `legend`
+  as required, and a response missing one now raises `InvalidResponse` rather
+  than being handed back with an empty Hash in its place. OpenRouter's schema
+  requires only `type` plus `choice`/`score`, and answers without a
+  `confidence` are no longer rejected there; `confidence` is `nil` instead.
+  Providers declare this with `required_answer_fields`.
+- Numeric answer fields are checked for finiteness and range. `noul`,
+  `confidence`, and probability values must be finite and within 0..1;
+  `{"noul": 1e999}` parses to `Infinity` and used to be accepted.
+- A `choice` that is not one of the question's own criteria raises
+  `InvalidResponse` instead of reaching application routing as an unknown
+  label.
+- `InvalidResponse` messages now name the field that was wrong, not just the
+  question id.
+
 ## 0.1.0 - 2026-09-18
 
 Provider-neutral release. One `Client`, two providers behind it.

README.md

@@ -124,6 +124,33 @@ Answers come back typed: `Answers::Noul` (`noul`, `probabilities`),
 `response.nouls`, `response.choices`, and `response.scores` return the answers
 of one type keyed the same way as `response.answers`.
 
+### What counts as a valid answer
+
+Each provider declares which answer fields its published contract guarantees,
+and the client holds it to that rather than to one shared guess.
+
+| | OpenRouter requires | Typesafe requires |
+| --- | --- | --- |
+| noul | `noul` | `noul` |
+| choice | `choice` | `choice`, `confidence`, `probabilities` |
+| score | `score` | `score`, `confidence`, `probabilities`, `legend` |
+
+A required field that is missing raises `InvalidResponse` naming the question
+and the field. A field the provider does not guarantee comes back `nil`, or
+`{}` for a map, rather than raising: OpenRouter's schema makes `confidence`
+optional, so `answer.confidence` can be `nil` there. A field that is present
+but the wrong shape always raises; it is never replaced with an empty value.
+
+Numbers are checked as well as typed. `noul`, `confidence`, and every value in
+a `probabilities` map must be finite and within 0..1 — `1e999` parses to
+`Infinity` and would otherwise sail through an `is_a?(Numeric)` check — and a
+`choice` must be one of the keys the question's `criteria` offered, so a label
+the application has no branch for cannot reach its routing.
+
+A provider written by hand declares its own contract with
+`required_answer_fields`; naming a field the client does not model raises
+`ConfigurationError` when the client is built.
+
 Score `probabilities` and `legend` are keyed by the wire's string level keys
 (`"0"`, `"1"`, ...), not by the criteria labels. Choice `probabilities` sum to
 approximately 1; treat them as calibrated, not normalized.

lib/ruby_decision_model/client.rb

@@ -35,6 +35,7 @@ def initialize(provider: nil, api_key: nil, model: nil, base_url: nil, timeout:
                    transport: nil, sleeper: ->(seconds) { sleep(seconds) }, retry: {},
                    random: -> { rand }, clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) })
       @provider = resolve_provider(provider, api_key: api_key, base_url: base_url)
+      validate_answer_contract(@provider)
       unless @provider.api_key?
         raise ConfigurationError,
               "api_key is required for #{@provider.name}: pass api_key: or set #{@provider.env_var}"
@@ -88,6 +89,23 @@ def resolve_provider(provider, api_key:, base_url:)
       end
     end
 
+    # A provider that requires a field the client does not model would be
+    # declaring a contract nothing enforces. Say so when the client is built
+    # rather than letting the guarantee quietly do nothing.
+    def validate_answer_contract(provider)
+      provider.required_answer_fields.each do |type, names|
+        known = ANSWER_FIELDS[type]
+        raise ConfigurationError, "#{provider.name} requires fields for unknown answer type #{type.inspect}" if known.nil?
+
+        unknown = names.map(&:to_s) - known.keys
+        next if unknown.empty?
+
+        raise ConfigurationError,
+              "#{provider.name} requires answer fields the client does not model: " \
+              "#{unknown.join(', ')} on #{type} answers"
+      end
+    end
+
     def default_transport
       lambda do |url:, headers:, body:|
         uri = URI.parse(url)
@@ -208,9 +226,9 @@ def parse_success(response_body, response_headers, questions)
 
         if answer_hash.is_a?(Hash) && answer_hash["type"] == expected_type
           begin
-            normalized[id] = normalize_answer(expected_type, answer_hash)
-          rescue MalformedAnswer
-            malformed << id
+            normalized[id] = normalize_answer(expected_type, answer_hash, question)
+          rescue MalformedAnswer => e
+            malformed << "#{id} (#{e.message})"
           end
         else
           missing << id
@@ -219,7 +237,7 @@ def parse_success(response_body, response_headers, questions)
 
       if malformed.any?
         raise InvalidResponse.new(
-          "malformed answer fields for: #{malformed.join(', ')}",
+          "malformed answer fields for: #{malformed.join('; ')}",
           answers: normalized
         )
       end
@@ -255,36 +273,106 @@ def request_id_from(headers)
 
     class MalformedAnswer < StandardError; end
 
-    def normalize_answer(type, hash)
+    # Every field an answer can carry, and the shape each one has to have.
+    # A field absent from this map is ignored; Response#raw still has it.
+    ANSWER_FIELDS = {
+      # Neither provider's schema has probabilities on a noul answer -- the
+      # value is the probability -- but it is read when one turns up rather
+      # than dropped, and Answers::Noul has carried the field since 0.0.1.
+      "noul" => { "noul" => :probability, "probabilities" => :distribution },
+      "choice" => { "choice" => :label, "confidence" => :probability, "probabilities" => :distribution },
+      "score" => { "score" => :number, "confidence" => :probability,
+                   "probabilities" => :distribution, "legend" => :map }
+    }.freeze
+
+    # Probabilities are floats off a model, so a distribution can land a hair
+    # over 1.0 without being wrong.
+    PROBABILITY_TOLERANCE = 1e-6
+
+    def normalize_answer(type, hash, question)
+      fields = ANSWER_FIELDS[type]
+      raise MalformedAnswer, "unsupported answer type #{type.inspect}" if fields.nil?
+
+      required = @provider.required_answer_fields.fetch(type, fields.keys)
+      values = fields.to_h { |name, shape| [name, read_answer_field(hash, name, shape, required)] }
+      check_choice_is_offered(values["choice"], question) if type == "choice"
+
+      build_answer(type, values)
+    end
+
+    def read_answer_field(hash, name, shape, required)
+      raw = hash[name]
+
+      if raw.nil?
+        raise MalformedAnswer, "#{name} is missing" if required.include?(name)
+
+        return %i[distribution map].include?(shape) ? {} : nil
+      end
+
+      case shape
+      when :probability then unit_interval(name, raw)
+      when :number then finite_number(name, raw)
+      when :label then label(name, raw)
+      when :distribution then distribution(name, raw)
+      when :map then raw.is_a?(Hash) ? raw : raise(MalformedAnswer, "#{name} is not an object")
+      end
+    end
+
+    def finite_number(name, raw)
+      raise MalformedAnswer, "#{name} is not a number" unless raw.is_a?(Numeric)
+      # JSON turns 1e999 into Infinity and some encoders emit NaN. Neither is
+      # an answer, and both survive every is_a?(Numeric) check downstream.
+      raise MalformedAnswer, "#{name} is not finite (#{raw})" unless raw.finite?
+
+      raw.to_f
+    end
+
+    def unit_interval(name, raw)
+      value = finite_number(name, raw)
+      unless value >= -PROBABILITY_TOLERANCE && value <= 1.0 + PROBABILITY_TOLERANCE
+        raise MalformedAnswer, "#{name} is outside 0..1 (#{value})"
+      end
+
+      value.clamp(0.0, 1.0)
+    end
+
+    def label(name, raw)
+      raise MalformedAnswer, "#{name} is not a string" unless raw.is_a?(String)
+      raise MalformedAnswer, "#{name} is empty" if raw.empty?
+
+      raw
+    end
+
+    def distribution(name, raw)
+      raise MalformedAnswer, "#{name} is not an object" unless raw.is_a?(Hash)
+
+      raw.to_h { |key, value| [key, unit_interval("#{name}[#{key.inspect}]", value)] }
+    end
+
+    # A choice the question never offered cannot be routed on, and reading it
+    # as a label the application knows is exactly the mistake this guards.
+    def check_choice_is_offered(choice, question)
+      return if choice.nil?
+
+      criteria = question.is_a?(Hash) ? (question["criteria"] || question[:criteria]) : nil
+      return unless criteria.is_a?(Hash)
+
+      offered = criteria.keys.map(&:to_s)
+      return if offered.include?(choice)
+
+      raise MalformedAnswer, "choice #{choice.inspect} is not one of the question's criteria (#{offered.join(', ')})"
+    end
+
+    def build_answer(type, values)
       case type
       when "noul"
-        noul = hash["noul"]
-        raise MalformedAnswer unless noul.is_a?(Numeric)
-
-        Answers::Noul.new(noul: noul.to_f, probabilities: hash_or_empty(hash["probabilities"]))
+        Answers::Noul.new(noul: values["noul"], probabilities: values["probabilities"])
       when "choice"
-        choice = hash["choice"]
-        confidence = hash["confidence"]
-        raise MalformedAnswer unless choice.is_a?(String) && confidence.is_a?(Numeric)
-
-        Answers::Choice.new(
-          choice: choice,
-          confidence: confidence.to_f,
-          probabilities: hash_or_empty(hash["probabilities"])
-        )
+        Answers::Choice.new(choice: values["choice"], confidence: values["confidence"],
+                            probabilities: values["probabilities"])
       when "score"
-        score = hash["score"]
-        confidence = hash["confidence"]
-        raise MalformedAnswer unless score.is_a?(Numeric) && confidence.is_a?(Numeric)
-
-        Answers::Score.new(
-          score: score.to_f,
-          confidence: confidence.to_f,
-          probabilities: hash_or_empty(hash["probabilities"]),
-          legend: hash_or_empty(hash["legend"])
-        )
-      else
-        raise MalformedAnswer
+        Answers::Score.new(score: values["score"], confidence: values["confidence"],
+                           probabilities: values["probabilities"], legend: values["legend"])
       end
     end
 
@@ -294,8 +382,5 @@ def question_type(question)
       question["type"] || question[:type]
     end
 
-    def hash_or_empty(value)
-      value.is_a?(Hash) ? value : {}
-    end
   end
 end

lib/ruby_decision_model/providers/base.rb

@@ -47,6 +47,23 @@ def reports_cost?
         false
       end
 
+      # The answer fields this provider's published contract declares
+      # required. The client raises InvalidResponse when one is missing
+      # instead of quietly substituting nil or an empty Hash, and leaves the
+      # rest nil when a provider does not send them.
+      #
+      # The default is the minimum that makes an answer an answer: the value
+      # field itself. A provider that guarantees more says so.
+      MINIMUM_ANSWER_FIELDS = {
+        "noul" => %w[noul],
+        "choice" => %w[choice],
+        "score" => %w[score]
+      }.freeze
+
+      def required_answer_fields
+        MINIMUM_ANSWER_FIELDS
+      end
+
       def base_url
         (@base_url || default_base_url).to_s.chomp("/")
       end

lib/ruby_decision_model/providers/typesafe.rb

@@ -33,6 +33,19 @@ def default_model
       def aliases
         ALIASES
       end
+
+      # https://docs.typesafe.ai/api documents probabilities, confidence, and
+      # legend as required on the answers that carry them, so a response
+      # missing one is a broken response rather than a sparse one.
+      REQUIRED_ANSWER_FIELDS = {
+        "noul" => %w[noul],
+        "choice" => %w[choice probabilities confidence],
+        "score" => %w[score probabilities confidence legend]
+      }.freeze
+
+      def required_answer_fields
+        REQUIRED_ANSWER_FIELDS
+      end
     end
   end
 end