Fix query9 validator accepting wrong answers and refusals (word-boundary match) - #79
Merged
Merged
Conversation
The fast-path check used a plain substring test, so "MI" matched inside common words such as determine, minimum and submitted. That made the validator accept refusals and answers naming a different state. Use word-boundary matching, which is already the semantics the fallback loop below applies via re.search(r'\b' + state + r'\b', ...). Refs ucbepic#78
marc-shade
added a commit
to marc-shade/DataAgentBench
that referenced
this pull request
Aug 2, 2026
Collaborator
|
Thanks, @marc-shade! We've merged your fix into the validator. Really appreciate it! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #78.
query_crmarenapro/query9/validate.pyused a plain substring test on its fast path:"MI"occurs inside common words in agent prose (determine,minimum,submitted,administrative), so the validator accepted refusals and answers that named a different state.This switches the fast path to word-boundary matching, which is already the semantics the fallback loop below it uses (
re.search(r'\b' + state + r'\b', ...)). One line, no new imports,reis already imported.Before / after
Same eight inputs, run against the validator before and after the change:
MIThe state is MI.the answer is mi(lowercase)I could not determine the answer.The answer is CA, at a MInimum.The submitted record shows TX.CAThe answer is Texas.All four false positives close; no true positive regresses, including the lowercase and in-prose forms.
Scope
Only this one file. I found it by running all 104
validate.pyfiles against two content-free probes (a refusal containing no numbers, and a string listing many candidate numbers). 85 resisted both; this was the only validator that accepted a refusal. The other 18 accept a numeric shotgun, which looks like the recall-over-precision tradeoff already noted in the paper rather than a defect, so I have not touched them.Happy to follow up with a small CI test that asserts every validator rejects a refusal, if that would be useful. It would currently need an allowlist for those 18.