A small, working harness for the thing that actually makes an agentic coding benchmark useful: proving a task is valid before anyone trains or evaluates on it, and grading a candidate patch on more than whether the tests turned green.
Every task in this repository ships with a base repository, a failing reproduction, a reference solution, a reproducible container, and a weighted rubric. The harness runs the whole thing end to end.
$ python3 -m harness verify retryutil-001
verify retryutil-001-negative-retry-after --------------------------
runner: local
BASE GOLDEN TEST
FAIL PASS tests/test_parser.py::TestHttpDate::test_past_date_clamps_to_zero
FAIL PASS tests/test_parser.py::TestHttpDate::test_naive_date_treated_as_utc
PASS PASS tests/test_parser.py::TestDeltaSeconds::test_plain_integer
...
VERIFIED -- the bug is real and the golden patch fixes it
Most evaluation tasks are published on trust: someone asserts that a bug is real and that the reference patch fixes it. Two failure modes follow, and both are silent.
An unverified task teaches the wrong thing. If a fail_to_pass test
already passes at the base commit, the task rewards a model for doing nothing.
If the base suite is not green to begin with, a model gets penalised for
breakage it never caused.
A test-only grade rewards the wrong patch. Tests answer did the behaviour change. They cannot answer was this the right change — and the single most important failure mode, a patch that edits the tests themselves, is invisible to the tests by construction.
This kit closes both gaps: verify refuses to bless a task that does not
demonstrate its own bug, and grade combines test outcomes with static
criteria that tests structurally cannot express.
git clone <your-fork-url> && cd agentic-eval-kit
pip install -r requirements.txt
python3 -m harness list # what tasks exist
python3 -m harness validate # schema check, runs nothing
python3 -m harness verify # prove every task is well formed
python3 -m harness grade retryutil-001 --patch examples/candidates/abs-shortcut.diffNo Docker required to try it — the default local runner uses the interpreter
you already have. Add --runner docker for the reproducible path.
tasks/retryutil-001-negative-retry-after is a real-shaped bug: an HTTP
Retry-After parser that returns a negative delay for dates in the past, and
crashes on dates without a timezone.
examples/candidates/ holds four patches for it. Every number below is real
output from python3 -m harness grade:
| Patch | Tests | Score | Verdict | Why |
|---|---|---|---|---|
GOLDEN |
10/10 | 100.0 | PASS | clamps to zero, and treats a missing zone as UTC |
partial-fix.diff |
9/10 | 73.7 | FAIL | clamps the sign but still crashes on naive datetimes — fixes half the report |
abs-shortcut.diff |
9/10 | 63.2 | FAIL | abs() instead of max(0.0, ...): a stale date becomes a long wait |
test-tampering.diff |
10/10 | 89.5 | FAIL | edited the assertions to match the buggy output |
The last row is the whole argument for rubrics. That patch makes every test
pass and scores 89.5. A test-only harness marks it resolved and ships the
bug. The no-test-tampering criterion is an auto-fail gate, so the verdict is
FAIL regardless of score:
FAIL [10] The patch does not edit tests or fixtures
patch modifies protected path(s): tests/test_parser.py
verdict: FAIL
- auto-fail criteria tripped: no-test-tampering
harness/ the CLI: loader, runners, evaluator, rubric scorer, report
loader.py structural validation with field-level error messages
runners.py local and docker backends behind one interface
evaluate.py test execution, JUnit parsing, verify + grade
rubric.py the seven check types and the scoring fold
tasks/<task-id>/
task.yaml problem statement, test command, fail_to_pass/pass_to_pass
rubric.yaml weighted criteria, auto-fail gates, pass threshold
Dockerfile pinned environment
golden_patch.diff the reference solution
repo/ the base repository, checked in at its buggy state
examples/candidates/ patches that exercise each rubric criterion
templates/ starting points for a new task
schemas/ JSON Schema for editor autocompletion
docs/ authoring guide, rubric design notes, case study
tests/ the harness's own unit tests
id: retryutil-001-negative-retry-after
title: parse_retry_after returns a negative delay for HTTP-dates in the past
difficulty: easy
problem_statement: |
Written as a bug report, not as instructions. It states observed behaviour
and expected behaviour; it never names the function to edit or the fix to
apply. A task that gives away the patch measures reading comprehension.
evaluation:
test_command: python3 -m pytest -q --no-header -p no:cacheprovider
fail_to_pass: # must FAIL at base, PASS after the golden patch
- tests/test_parser.py::TestHttpDate::test_past_date_clamps_to_zero
pass_to_pass: # must PASS in both runs
- tests/test_parser.py::TestDeltaSeconds::test_plain_integerThe loader rejects, with a message naming the field: an empty fail_to_pass,
a test listed in both groups, duplicates within a group, a rubric criterion
referencing a test the task never declares, weights that are not positive, and
an automated criterion with no rationale.
| Check | Catches |
|---|---|
tests |
the behaviour change itself |
forbidden_paths |
patches that edit tests, fixtures, or CI config |
forbidden_pattern |
a known-wrong idiom (abs() where a clamp belongs) |
required_pattern |
an invariant the tests cannot reach |
max_files_changed |
fixes that sprawl beyond the reported defect |
max_lines_changed |
rewrites dressed up as bug fixes |
manual |
judgement calls, scored by a human and excluded from the automated denominator |
Scoring folds these into one verdict: weighted percentage of automated
criteria, a pass_threshold, and an auto_fail list of criteria that force
FAIL no matter the score. Manual criteria never inflate or deflate the
automated score — they are reported separately as outstanding review.
See docs/RUBRIC_DESIGN.md for how criteria are chosen and docs/CASE_STUDY.md for this task's write-up.
The local runner is for authoring — fast, but it inherits whatever is on your
machine. Verify with Docker before publishing a task:
python3 -m harness verify retryutil-001 --runner dockerThe image supplies the interpreter and pinned dependencies; the repository is
bind-mounted at /workspace. Source stays out of the image, so re-running
after editing a patch costs a container start rather than a rebuild.
Containers run with --network none — a task that needs the network during
evaluation is not reproducible.
cp -r templates/task-skeleton tasks/myproject-001-short-slug
# fill in task.yaml, rubric.yaml, repo/, golden_patch.diff
python3 -m harness validate myproject-001
python3 -m harness verify myproject-001 --runner dockerThe full walkthrough, including the variant that clones a pinned upstream commit instead of vendoring the source, is in docs/TASK_AUTHORING.md.
MIT. The vendored retryutil package is written for this repository and is not
derived from any third-party project.