bench: judged answer-correctness is the primary quality axis - #7
Merged
hallelx2 merged 3 commits intoJun 20, 2026
Conversation
… ran primary_quality() now prefers the LLM-judge verdict (answer_correct) over span-overlap F1: answer_correct -> abstention -> f1@k. An answer-first system (treewalk emits its own grounded answer, not the verbatim gold passage) scored near-0 on span-F1 even when its answer was correct, so ranking on F1 buried it. Unjudged configs fall back to f1@k unchanged. Adds a regression test.
summarize() now aggregates an answer block (judged_n, correct, faithful, answered, judge_usd) that the runner already wrote to records.jsonl but the report discarded. Adds an 'Answer correctness (LLM-as-judge)' section to both report.md and report.html, and the headline now states what Quality means (judged answer-correctness vs F1@k span-overlap).
… bm25) financebench_glm_judged.yaml now runs vectorless_treewalk vs vector_rag (local bge-small embeddings, no OpenAI key) vs bm25 over 20 docs, judged on GLM-4.6 — the comparison the launch reports on. All LLM work stays on GLM via z.ai.
Reviewer's GuideMakes LLM-judge answer-correctness the primary quality metric when available, surfaces judged answer metrics in markdown/html reports, and updates the FinanceBench judged config to a canonical treewalk vs vector_rag vs bm25 comparison. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In
report.summarize, you now exposejudged_nboth at the top level and inside theanswerblock; consider keeping it in a single place (likely underanswer) to avoid redundant state and reduce the chance of these drifting apart in future changes. - The
judgedflag logic (based onanswer.judged_n) and the narrative explaining what ‘primary quality’ means are duplicated and slightly diverging between the markdown and HTML paths; it may be worth centralizing this logic/text so the two outputs stay in sync as the definition evolves.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `report.summarize`, you now expose `judged_n` both at the top level and inside the `answer` block; consider keeping it in a single place (likely under `answer`) to avoid redundant state and reduce the chance of these drifting apart in future changes.
- The `judged` flag logic (based on `answer.judged_n`) and the narrative explaining what ‘primary quality’ means are duplicated and slightly diverging between the markdown and HTML paths; it may be worth centralizing this logic/text so the two outputs stay in sync as the definition evolves.
## Individual Comments
### Comment 1
<location path="src/vectorless_bench/metrics/citation.py" line_range="94-95" />
<code_context>
+ 2. **Correct abstention** for no-answer questions (when unjudged).
+ 3. **F1@k** span-overlap retrieval quality (the unjudged default).
+ """
+ if "answer_correct" in metrics:
+ return metrics["answer_correct"]
if "abstained" in metrics:
return metrics["abstained"]
</code_context>
<issue_to_address>
**issue (bug_risk):** Primary quality now mixes LLM-judge scores with F1/abstention within the same aggregate when only some rows are judged.
With this precedence, when only some queries are LLM-judged (e.g., first repeat only), `primary_quality` will use `answer_correct` for judged rows but fall back to `abstained`/`f1@k` for others. Since `summarize()` averages `quality['primary']` over all ok rows, this makes the efficiency frontier and `quality_per_1k_usd` combine different definitions of “quality” into a single scalar, which conflicts with the docs that describe primary quality as judged answer-correctness when `judged` is true.
Consider making primary quality consistently judged-only once answer-correctness is enabled. For example:
- Require that all rows in judged runs have `answer_correct`, or
- In `summarize()`, when `judged_n > 0`, compute primary quality as the mean of `answer_correct` over `judged` rows only, and don’t fall back to span-F1 for unjudged rows.
This keeps the primary quality axis semantically consistent and aligned with the documentation.
</issue_to_address>
### Comment 2
<location path="src/vectorless_bench/report.py" line_range="163-172" />
<code_context>
lines.append(f"Run: `{run_dir.name}` · k={k} · see `manifest.json` for full config.\n")
+ judged = any(r["answer"]["judged_n"] > 0 for r in results.values())
+ quality_def = (
+ "LLM-judged answer-correctness (the candidate answer vs the FinanceBench "
+ "gold answer, graded by the judge model)" if judged
</code_context>
<issue_to_address>
**issue (bug_risk):** HTML description of primary quality may not match the actual scalar used in the frontier when judged is partially populated.
Right now `judged` is true as soon as any row has `judged_n > 0`, but `primary_quality()` still falls back to abstention/F1 for unjudged rows. That means `quality['primary']` for a “judged” run can mix judge scores with span-F1/abstention for the same system while the HTML still labels the axis as pure LLM-judged correctness.
To keep the frontier semantics clear, consider either (a) defining `quality['primary']` to only use fully judged rows (or requiring all rows be judged in “judged” runs), and/or (b) tightening the `judged` condition (e.g., only when all systems/rows are judged) so the label reflects the actual metric.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+94
to
+95
| if "answer_correct" in metrics: | ||
| return metrics["answer_correct"] |
There was a problem hiding this comment.
issue (bug_risk): Primary quality now mixes LLM-judge scores with F1/abstention within the same aggregate when only some rows are judged.
With this precedence, when only some queries are LLM-judged (e.g., first repeat only), primary_quality will use answer_correct for judged rows but fall back to abstained/f1@k for others. Since summarize() averages quality['primary'] over all ok rows, this makes the efficiency frontier and quality_per_1k_usd combine different definitions of “quality” into a single scalar, which conflicts with the docs that describe primary quality as judged answer-correctness when judged is true.
Consider making primary quality consistently judged-only once answer-correctness is enabled. For example:
- Require that all rows in judged runs have
answer_correct, or - In
summarize(), whenjudged_n > 0, compute primary quality as the mean ofanswer_correctoverjudgedrows only, and don’t fall back to span-F1 for unjudged rows.
This keeps the primary quality axis semantically consistent and aligned with the documentation.
Comment on lines
+163
to
+172
| quality_def = ( | ||
| "LLM-judged answer-correctness (the candidate answer vs the FinanceBench " | ||
| "gold answer, graded by the judge model)" if judged | ||
| else "F1@k span-overlap for answerable questions / correct abstention for " | ||
| "no-answer questions" | ||
| ) | ||
|
|
||
| lines.append("## Efficiency frontier (the headline)\n") | ||
| lines.append("Quality is meaningless without its price. `quality_per_1k_usd` = " | ||
| lines.append(f"**Quality = {quality_def}.** " | ||
| "Quality is meaningless without its price. `quality_per_1k_usd` = " |
There was a problem hiding this comment.
issue (bug_risk): HTML description of primary quality may not match the actual scalar used in the frontier when judged is partially populated.
Right now judged is true as soon as any row has judged_n > 0, but primary_quality() still falls back to abstention/F1 for unjudged rows. That means quality['primary'] for a “judged” run can mix judge scores with span-F1/abstention for the same system while the HTML still labels the axis as pure LLM-judged correctness.
To keep the frontier semantics clear, consider either (a) defining quality['primary'] to only use fully judged rows (or requiring all rows be judged in “judged” runs), and/or (b) tightening the judged condition (e.g., only when all systems/rows are judged) so the label reflects the actual metric.
hallelx2
merged commit Jun 20, 2026
c54851e
into
halleluyaholudele/hal-70-bench-treewalk-endpoint-and-title-path
2 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The bench computed the LLM-as-judge verdict (
answer_correct/faithful/answered) per question, wrote it torecords.jsonl, then discarded it: the report ranked every system on F1@k span-overlap. An answer-first system (treewalk emits its own grounded answer, not the verbatim gold passage) scores near-0 on span-F1 even when its answer is correct — so treewalk looked like it scored ~0 in every judged run. The LLM judge ran, graded correctly, and the report threw the number away.Changes (3 sequential commits)
primary_quality(): rank on judgedanswer_correctwhen the judge ran (answer_correct -> abstention -> f1@k). Unjudged configs unchanged. + regression test.report.summarize()aggregates ananswerblock; new Answer correctness (LLM-as-judge) section inreport.md+report.html; headline states what Quality means.financebench_glm_judged.yaml→ canonical 3-way: treewalk vs vector_rag (local bge-small embeddings) vs bm25, 20 docs, GLM-4.6 judge.Verified
Closes HAL-322
Summary by Sourcery
Prioritize LLM-judged answer correctness as the primary quality metric and surface it prominently in reports, while updating the canonical FinanceBench judged benchmark configuration.
New Features:
Enhancements:
Tests: