Skip to content

fix: refuse unsendable questions before paying for them - #9

Draft
simonx1 wants to merge 1 commit into
obie:mainfrom
simonx1:fix/validate-questions-before-send
Draft

simonx1 wants to merge 1 commit into
obie:mainfrom
simonx1:fix/validate-questions-before-send

Conversation

@simonx1

@simonx1 simonx1 commented Sep 18, 2026

Copy link
Copy Markdown

ask checked that the questions hash was non-empty and sent whatever was in it.

client.ask(state: "x", questions: { "q" => "is this urgent?" })
# request body: {"model":...,"state":"x","questions":{"q":"is this urgent?"}}
# => MissingAnswers: missing or wrong-type answers for: q

That request was serialized, posted, and billed as input tokens, and came back as a report that the answer was missing for something that was never a question. An unknown type behaved the same way; a choice question with no criteria earned a 422.

Two ids that stringify to the same thing were worse than a wasted request. Answers come back keyed by the stringified id, so :a and "a" in one map are two questions, one answer, and a MissingAnswers naming whichever one lost.

The fix

Validate the map before building the body: each value a Hash, a type from the three the API defines, instructions present, and the criteria that type requires (1..255 options for choice, 2..10 levels for score — the same bounds the builders enforce, which do match the docs). Blank and colliding ids are refused. RequestError names the id and nothing goes out.

Hand-rolled question hashes are now held to the same rules as built ones, which is the point: the builders were the only validation and nothing made you use them. Questions.validate!(question, id:) is public for callers assembling questions elsewhere.

nil instructions are allowed through — EntryType in the official SDKs covers null for state, instructions and criteria alike.

Tests

test/question_validation_test.rb, 18 cases, each asserting the transport was never called. One existing test that asserted the send-then-fail behaviour for an unknown type now asserts it's refused first.


Draft: part of a security and API-coverage audit, opened for reference rather than as a request for immediate review. Independent of the other branches, each off main. Suite green on Ruby 3.2.11, 3.3.8 and 3.4.8.

🤖 Generated with Claude Code

`ask` checked that the questions hash was non-empty and sent whatever was
in it. `questions: { "q" => "is this urgent?" }` was serialized, posted,
billed as input tokens, and came back as MissingAnswers -- a report that
the answer was missing for a question that was never a question. An
unknown type behaved the same way, and a choice question with no criteria
earned a 422.

Two ids that stringify to the same thing were worse than a wasted
request. Answers come back keyed by the stringified id, so `:a` and `"a"`
in one map are two questions, one answer, and a MissingAnswers naming
whichever one lost.

Validate the map before building the body: each value a Hash, a type from
the three the API defines, instructions present, and the criteria that
type requires (1..255 options for choice, 2..10 levels for score, the
same bounds the builders enforce). Blank and colliding ids are refused.
RequestError names the id.

Hand-rolled question hashes are now held to the same rules as built ones,
which is the point: the builders were the only validation, and nothing
made you use them. Questions.validate! is public so a caller assembling
questions elsewhere can run the same check.

nil instructions are allowed through: EntryType in the official SDKs
covers null for state, instructions, and criteria alike.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant