Gold-sanity gate: every bench check must pass on the reference solution before it may fail anyone

[ParallelEntrepreneur]

Goal

Every check we score agents against must first prove itself against a known-good solution: run each bench task's verification on the reference outcome before it may fail anyone — because a verifier that is wrong punishes good agents, and an LLM-judged check is wrong most often exactly where it looks most confident.

Current state (with pointers)

• Bench tasks (scripts/bench/tasks.json, docs/bench.md): fixed tasks with pass criteria, scored the same way every time. Nothing currently requires a task's checks to pass on a reference solution — a broken check (bad assertion, wrong fixture, impossible condition) fails every agent equally and reads as "agents can't do this task" instead of "the task is broken."
• The failure mode is measured, not theoretical: a majority of generated tests in one study failed against the gold patch itself, which an LLM judge alone cannot see. Any rubric/LLM-judged check Colonizer adds (findings quality, review thoroughness, red-team severity) inherits this blind spot unless gated.
• Neighbors: the held-out bench issue (suite structure), the trajectory monitor (process scoring), Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310 (the Evolver must not optimize against broken checks — garbage fitness in, garbage retained).

Requirements

1. Gold-sanity gate per task. For every bench task, a reference solution (or reference outcome state) exists that the task's checks must accept. Adding or editing a task requires a gate run: reference in → all checks green, or the task does not land. The gate runs in CI on every change to tasks, checks, or scoring — a broken gate blocks the bench change, not the agent change being measured.
2. LLM-judged checks get double gating. Any check involving model judgment (rubrics, quality scores, severity ratings) must additionally pass a discrimination test: it scores the reference high AND a deliberately degraded variant low, with the margin recorded. A judge that can't tell good from deliberately-bad is uncalibrated and doesn't ship — its verdicts would be noise billed as signal.
3. Reference maintenance. References live beside their tasks (scripts/bench/ layout), versioned with them; a task edit that invalidates its reference fails the gate until the reference is updated in the same change. Stale references are a first-class failure, reported as "task broken," never as agent scores.
4. Broken-task quarantine. A task whose gate starts failing in CI (fixture rot, environment drift) is quarantined automatically: excluded from scoring with a loud marker, filed as a task bug, and its recent scores annotated (not silently recomputed). Quarantine is visible in every report that would otherwise include the task.
5. Evolver guard. Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310's retain step may only use scores from gate-passing tasks; quarantined tasks contribute nothing to fitness. State this as an invariant with a test (a quarantined task in the input set changes no retention decision).

Test plan

• Unit: gate logic (green reference passes, broken reference blocks landing), discrimination test math (margin thresholds, degraded-variant construction rules), quarantine transitions.
• Adversarial fixtures: a task with a subtly wrong assertion (fails the reference — must block), a judge that scores everything high (fails discrimination — must block), a rotting fixture (quarantine path).
• CI: gate runs on bench-affecting changes; quarantine marker renders in reports.

Done when

• Every bench task has a reference that its checks accept, enforced at land time
• LLM-judged checks pass discrimination (reference high, degraded low, margin recorded)
• References versioned with tasks; stale reference = task broken, loudly
• Quarantine automatic, visible, annotated; Evolver invariant tested
• docs/bench.md documents the gate, the quarantine, and the margin rule

Out of scope

• Writing new bench tasks (content work); changing scoring math for passing tasks; the held-out rotation mechanics (separate issue — shares the gate).

Refs

• scripts/bench/, docs/bench.md, scripts/bench.mjs
• Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310