Skip to content

fix(judge): collapse judge_llm.py onto judge.py's transport - #319

Open
vaibhavdabas16 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:fix/collapse-lenient-judge-duplication
Open

fix(judge): collapse judge_llm.py onto judge.py's transport#319
vaibhavdabas16 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
vaibhavdabas16:fix/collapse-lenient-judge-duplication

Conversation

@vaibhavdabas16

Copy link
Copy Markdown

What does this PR do?

judge_llm.py declares itself a "drop-in replacement" for judge.py (:20-22) and then re-implements _post_json, _build_user_msg, and the api_type dispatch. That copy has drifted, and it is the module behind the published Reward-lenient column:

judge.py judge_llm.py (before)
judge_context kwarg (passed at run.py:592) yes absent from the signature — a TypeError on this path
openai-responses /responses (:240) folded in with openai-completions/chat/completions (:188)
google-generative-ai supported via /v1beta/openai (:235-239) NotImplementedError (:195)
max_tokens 4096 800

The judge_context gap is the one with the quietest failure mode: imported rubric-based tasks were judged on the lenient path with the rubric withheld from the judge. The 800-token cap matters because the default judge is deepseek-v4-pro, a reasoning model that spends hidden tokens before emitting its JSON line — truncation lands on the unparseable path.

This reduces the module to its JUDGE_SYSTEM prompt, which is the one thing that legitimately differs between the rubrics, and delegates to judge._build_user_msg / judge._run_judge. 176 lines of duplicated transport become 28.

judge_request keeps its positional signature, so rescore.py's judge-function dict (rescore.py:261-269, which calls both judges positionally) is unchanged. It gains judge_context and retries as keyword-only arguments matching the strict judge, and the returned dict keeps its "rubric": "lenient" key.

Two behaviour changes fall out of the collapse and are intended — flagging them explicitly rather than burying them:

  1. Retries now cover any transport exception, not only HTTP 429/500/502/503.
  2. A failed call reports judge_call_failed: ... as its reason instead of an empty string.

Both are judge.py's existing behaviour; this PR adopts it rather than preserving the divergent copy. Happy to keep the old retry policy instead if you would rather that stayed put.

Corpus

  • v2
  • v1
  • both
  • not applicable

The lenient rubric is the V2 reward signal (judge_llm.py:15-18), though the module is reachable from clawbench-rescore for any corpus.

Test plan

  • New tests/test_lenient_judge_parity.py (10 tests) pins each of the four divergences, asserts the two rubrics differ only in the system prompt on the wire (same URL, headers, user message, and token budget), and guards against the transport being copied back in.
  • The new tests were run against the previous judge_llm.py: 9 of the 10 fail, each with the error the issue describes — TypeError: unexpected keyword argument 'judge_context', the openai-responses call never reaching /responses, assert None is True for Gemini, and max_tokens 800. A parity test that only passes after the change proves nothing, so this control mattered.
  • Verdict parsing was checked for equivalence before collapsing it: both _parse_verdict implementations were run over the union of their test corpora plus markdown-fence, prefix-prose, and stringly-typed edge cases (15 cases) and agreed on every one. So _parse_verdict is re-exported from judge.py, no published number moves, and the existing test_lenient_judge_parse_verdict_is_tri_state keeps guarding this path unchanged.
  • Full suite: 209 passed, 3 skipped. The single failure, test_host_tasks.py::test_checked_task_json_files_parse_and_validate[v1-lite], is a local Windows artifact (those task files are git symlinks, mode 120000, that Windows checks out as text) and is green on CI.
  • ruff check and ruff format --check clean on both files.
  • Verified conflict-free against main and against all four of my other open PRs (fix(batch): one truncated run-meta.json no longer discards the batch summary #312, fix(rescore): resolve paths from the workspace instead of a maintainer's home #313, fix(judge): a bad --judge model no longer discards a completed run #314, fix(runner): bound container wait so one wedged run can't stall a batch #316) with git merge-tree.

No network is touched by the tests: they monkeypatch judge._post_json and assert on the captured payloads.

Related issues

Fixes #300.

judge_llm.py declared itself a "drop-in replacement" for judge.py and then
re-implemented _post_json, _build_user_msg, and the api_type dispatch. The
copy drifted, and it is the module behind the published Reward-lenient
column:

- judge_context was absent from its judge_request() signature, so the kwarg
  run.py passes to the strict judge was a TypeError on this path. Imported
  rubric-based tasks were judged with the rubric withheld from the judge.
- api_type "openai-responses" was folded in with "openai-completions" and
  POSTed to /chat/completions, which is not that API's endpoint.
- api_type "google-generative-ai" raised NotImplementedError. The Gemini
  /v1beta/openai fix only ever landed in judge.py.
- max_tokens was 800 against judge.py's 4096. The default judge is
  deepseek-v4-pro, a reasoning model that spends hidden tokens before
  emitting its verdict; truncation lands on the unparseable path.

The module now owns only JUDGE_SYSTEM — the lenient rubric is the one thing
that legitimately differs — and delegates to judge._build_user_msg and
judge._run_judge. judge_request keeps its positional signature, so
rescore.py's judge-function dict is unchanged, and gains judge_context and
retries as keyword-only arguments matching the strict judge. The returned
dict keeps its "rubric": "lenient" key.

Two behaviour changes fall out of the collapse and are intended: retries now
cover any transport exception rather than only HTTP 429/500/502/503, and a
failed call reports "judge_call_failed: ..." as its reason instead of an
empty string.

Verdict parsing was already equivalent — both parsers were checked against
the union of their test corpora plus fence, prefix-prose, and stringly-typed
edge cases, and agreed on every one — so _parse_verdict is re-exported from
judge.py and the existing lenient parser test keeps guarding this path.

tests/test_lenient_judge_parity.py pins each of the four divergences, asserts
the two rubrics differ only in the system prompt on the wire, and guards
against the transport being copied back in. All nine fail against the
previous judge_llm.py.

Fixes TIGER-AI-Lab#300.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

judge_llm.py has diverged from judge.py (no Gemini support, wrong responses endpoint, 800-token cap) — collapse the duplication

1 participant