fix(judge): collapse judge_llm.py onto judge.py's transport - #319
Open
vaibhavdabas16 wants to merge 1 commit into
Open
fix(judge): collapse judge_llm.py onto judge.py's transport#319vaibhavdabas16 wants to merge 1 commit into
vaibhavdabas16 wants to merge 1 commit into
Conversation
judge_llm.py declared itself a "drop-in replacement" for judge.py and then re-implemented _post_json, _build_user_msg, and the api_type dispatch. The copy drifted, and it is the module behind the published Reward-lenient column: - judge_context was absent from its judge_request() signature, so the kwarg run.py passes to the strict judge was a TypeError on this path. Imported rubric-based tasks were judged with the rubric withheld from the judge. - api_type "openai-responses" was folded in with "openai-completions" and POSTed to /chat/completions, which is not that API's endpoint. - api_type "google-generative-ai" raised NotImplementedError. The Gemini /v1beta/openai fix only ever landed in judge.py. - max_tokens was 800 against judge.py's 4096. The default judge is deepseek-v4-pro, a reasoning model that spends hidden tokens before emitting its verdict; truncation lands on the unparseable path. The module now owns only JUDGE_SYSTEM — the lenient rubric is the one thing that legitimately differs — and delegates to judge._build_user_msg and judge._run_judge. judge_request keeps its positional signature, so rescore.py's judge-function dict is unchanged, and gains judge_context and retries as keyword-only arguments matching the strict judge. The returned dict keeps its "rubric": "lenient" key. Two behaviour changes fall out of the collapse and are intended: retries now cover any transport exception rather than only HTTP 429/500/502/503, and a failed call reports "judge_call_failed: ..." as its reason instead of an empty string. Verdict parsing was already equivalent — both parsers were checked against the union of their test corpora plus fence, prefix-prose, and stringly-typed edge cases, and agreed on every one — so _parse_verdict is re-exported from judge.py and the existing lenient parser test keeps guarding this path. tests/test_lenient_judge_parity.py pins each of the four divergences, asserts the two rubrics differ only in the system prompt on the wire, and guards against the transport being copied back in. All nine fail against the previous judge_llm.py. Fixes TIGER-AI-Lab#300.
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
judge_llm.pydeclares itself a "drop-in replacement" forjudge.py(:20-22) and then re-implements_post_json,_build_user_msg, and theapi_typedispatch. That copy has drifted, and it is the module behind the published Reward-lenient column:judge.pyjudge_llm.py(before)judge_contextkwarg (passed atrun.py:592)TypeErroron this pathopenai-responses/responses(:240)openai-completions→/chat/completions(:188)google-generative-ai/v1beta/openai(:235-239)NotImplementedError(:195)max_tokensThe
judge_contextgap is the one with the quietest failure mode: imported rubric-based tasks were judged on the lenient path with the rubric withheld from the judge. The 800-token cap matters because the default judge isdeepseek-v4-pro, a reasoning model that spends hidden tokens before emitting its JSON line — truncation lands on the unparseable path.This reduces the module to its
JUDGE_SYSTEMprompt, which is the one thing that legitimately differs between the rubrics, and delegates tojudge._build_user_msg/judge._run_judge. 176 lines of duplicated transport become 28.judge_requestkeeps its positional signature, sorescore.py's judge-function dict (rescore.py:261-269, which calls both judges positionally) is unchanged. It gainsjudge_contextandretriesas keyword-only arguments matching the strict judge, and the returned dict keeps its"rubric": "lenient"key.Two behaviour changes fall out of the collapse and are intended — flagging them explicitly rather than burying them:
judge_call_failed: ...as its reason instead of an empty string.Both are
judge.py's existing behaviour; this PR adopts it rather than preserving the divergent copy. Happy to keep the old retry policy instead if you would rather that stayed put.Corpus
The lenient rubric is the V2 reward signal (
judge_llm.py:15-18), though the module is reachable fromclawbench-rescorefor any corpus.Test plan
tests/test_lenient_judge_parity.py(10 tests) pins each of the four divergences, asserts the two rubrics differ only in the system prompt on the wire (same URL, headers, user message, and token budget), and guards against the transport being copied back in.judge_llm.py: 9 of the 10 fail, each with the error the issue describes —TypeError: unexpected keyword argument 'judge_context', theopenai-responsescall never reaching/responses,assert None is Truefor Gemini, andmax_tokens800. A parity test that only passes after the change proves nothing, so this control mattered._parse_verdictimplementations were run over the union of their test corpora plus markdown-fence, prefix-prose, and stringly-typed edge cases (15 cases) and agreed on every one. So_parse_verdictis re-exported fromjudge.py, no published number moves, and the existingtest_lenient_judge_parse_verdict_is_tri_statekeeps guarding this path unchanged.test_host_tasks.py::test_checked_task_json_files_parse_and_validate[v1-lite], is a local Windows artifact (those task files are git symlinks, mode120000, that Windows checks out as text) and is green on CI.ruff checkandruff format --checkclean on both files.mainand against all four of my other open PRs (fix(batch): one truncated run-meta.json no longer discards the batch summary #312, fix(rescore): resolve paths from the workspace instead of a maintainer's home #313, fix(judge): a bad --judge model no longer discards a completed run #314, fix(runner): bound container wait so one wedged run can't stall a batch #316) withgit merge-tree.No network is touched by the tests: they monkeypatch
judge._post_jsonand assert on the captured payloads.Related issues
Fixes #300.