Fix query8 validator rejecting the correct character spelling (apostrophe normalization) - #82
Merged
Ruiying-Ma merged 1 commit intoAug 2, 2026
Conversation
normalize() converted an apostrophe to a space, producing 'cockamamie s salesgirl', while the character check searched for the literal 'cockamamies salesgirl'. The correctly spelled name therefore never matched and only a misspelling passed. Drop apostrophes in normalize() instead. This is additive: the correct spelling and the curly variant now pass, and the previously passing no-apostrophe form still passes. Refs ucbepic#81
marc-shade
added a commit
to marc-shade/DataAgentBench
that referenced
this pull request
Aug 2, 2026
Collaborator
|
Thanks, @marc-shade! We've merged your fix into the validator. Really appreciate it! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #81.
normalize()turned an apostrophe into a space, so the ground-truth character name"Cockamamie's" Salesgirlnormalized tocockamamie s salesgirl, while the check searched for the literalcockamamies salesgirl. The correct spelling could never match, and only a misspelling passed.This drops apostrophes in
normalize()instead of spacing them, so the existing literal becomes reachable. The change is additive: nothing that passed before stops passing.Before / after
Every other clause held satisfied, varying only the character name:
"Cockamamie's" Salesgirl(correct, matches ground truth)“Cockamamie’s” Salesgirl(curly apostrophe)ground_truth.csvfield textCockamamies Salesgirl(misspelled)The curly
’case matters in practice because models frequently emit it rather than'.Why this shape of fix
The alternative was to normalize the expected literal at the call site, matching how the actress clause already does it (
normalize("Aaron, Caroline") not in llm_norm). That also works, but it would start rejecting the no-apostrophe spelling that passes today, which could regress an existing leaderboard submission. Fixingnormalize()avoids that.Found while adding a positive-control test that feeds each validator its own ground truth. 103 of the 104 accept it; this was the only one that did not.