Skip to content

Fix query9 validator accepting wrong answers and refusals (word-boundary match) - #79

Merged
Ruiying-Ma merged 1 commit into
ucbepic:mainfrom
marc-shade:fix/query9-word-boundary
Aug 2, 2026
Merged

Ruiying-Ma merged 1 commit into
ucbepic:mainfrom
marc-shade:fix/query9-word-boundary

Conversation

@marc-shade

Copy link
Copy Markdown
Contributor

Fixes #78.

query_crmarenapro/query9/validate.py used a plain substring test on its fast path:

if expected.upper() in llm_output_clean.upper():   # expected = "MI"

"MI" occurs inside common words in agent prose (determine, minimum, submitted, administrative), so the validator accepted refusals and answers that named a different state.

This switches the fast path to word-boundary matching, which is already the semantics the fallback loop below it uses (re.search(r'\b' + state + r'\b', ...)). One line, no new imports, re is already imported.

Before / after

Same eight inputs, run against the validator before and after the change:

LLM output before after
MI PASS PASS
The state is MI. PASS PASS
the answer is mi (lowercase) PASS PASS
I could not determine the answer. PASS fail
The answer is CA, at a MInimum. PASS fail
The submitted record shows TX. PASS fail
CA fail fail
The answer is Texas. fail fail

All four false positives close; no true positive regresses, including the lowercase and in-prose forms.

Scope

Only this one file. I found it by running all 104 validate.py files against two content-free probes (a refusal containing no numbers, and a string listing many candidate numbers). 85 resisted both; this was the only validator that accepted a refusal. The other 18 accept a numeric shotgun, which looks like the recall-over-precision tradeoff already noted in the paper rather than a defect, so I have not touched them.

Happy to follow up with a small CI test that asserts every validator rejects a refusal, if that would be useful. It would currently need an allowlist for those 18.

The fast-path check used a plain substring test, so "MI" matched inside
common words such as determine, minimum and submitted. That made the
validator accept refusals and answers naming a different state.

Use word-boundary matching, which is already the semantics the fallback
loop below applies via re.search(r'\b' + state + r'\b', ...).

Refs ucbepic#78
@Ruiying-Ma

Copy link
Copy Markdown
Collaborator

Thanks, @marc-shade! We've merged your fix into the validator. Really appreciate it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

query_crmarenapro/query9 validator accepts wrong answers and refusals (substring match without word boundary)

2 participants