External calibration: SWE-bench Lite to Verified to Multilingual, with controls, then leaderboard

[ParallelEntrepreneur]

Goal

Calibrate colonies against the industry's standard external benchmark instead of only our own fixed tasks: run SWE-bench Lite, then Verified, then Multilingual through colonies, with the anti-hacking controls and gold-sanity gate our own eval work demands — and publish the scores.

Why these three, why this order

• SWE-bench Lite first (300 instances): the cheapest full-protocol run (same harness, F2P/P2P) to shake down the plumbing — dataset fetch, colony-per-task launch, patch extraction, scoring — before spending on the full sets.
• Verified second: the standard 500-task calibration everyone compares against. Known validity caveats apply (leakage channels, flawed instances per the recent audits) — which is exactly why our anti-hacking controls and gold-sanity gate (Gold-sanity gate: every bench check must pass on the reference solution before it may fail anyone #330) must wrap it, not the raw harness.
• Multilingual third, and the real prize: 300 tasks across 9 languages including Go, Rust, JavaScript, and TypeScript — our sandbox presets' own stacks (Node, Python, Rust, Go). Python-only Verified can't tell us whether colonies hold up where our users actually work; Multilingual can, per language.
• Leaderboard submission last: publish colony scores once a calibrated run exists. External, comparable numbers are comms and recruiting, not vanity — but only after the controls make them honest.

Current state (with pointers)

• Bench (scripts/bench.mjs, docs/bench.md): fixed internal tasks; held-out split and gold-sanity gate are separate issues and are prerequisites here, not followers — no external run happens without them.
• Red-team runs (crates/colonizer/src/redteam.rs): the in-house adversarial surface; external suites complement it (real-world issues vs our hunters).
• Cost tracking (spend.rs, per-colony budgets): external runs cost real money per task (the Multilingual baseline cites a $2.50 cost limit per task) — every run needs a budget envelope up front.
• Neighbors: held-out bench issue, gold-sanity gate issue (Gold-sanity gate: every bench check must pass on the reference solution before it may fail anyone #330), trajectory monitor (clean-resolved reporting for external runs), Harness-aware cost routing and per-colony harness x model x dollar logging (colony-report, bench) #296 (cost attribution per task).

Requirements

1. Harness adapter, staged. Stage 1 (Lite): dataset fetch (HuggingFace id or local path — the SWE-bench CLI accepts both), one colony per task (issue text as the task, repo snapshot as the worktree base), patch extraction from the colony's branch, F2P/P2P scoring in a fresh environment the colony never touched. Stage 2 (Verified): same pipe, full set. Stage 3 (Multilingual): plus per-language sandbox stacks (our presets already cover Node/Python/Rust/Go — wire them, don't rebuild) and per-language score reporting.
2. Controls mandatory, not optional. Every external run applies: anti-hacking environment controls (fresh single-commit reconstruction, concealed eval artifacts, no network answer sources), the gold-sanity gate on any added/adapted check, and trajectory-monitor scoring (report clean vs raw resolution alongside). A run missing any control is labeled uncalibrated in the report, never compared.
3. Budget envelope. Each run declares per-task and total spend caps up front (Multilingual-scale: hundreds of tasks × model cost); the run stops cleanly at the cap with partial results reported, not extrapolated. Caps live in the run record next to scores.
4. Score record. Per run: suite + version, colony configuration (module, models, settings pins), per-task outcomes (visible/held-out where applicable, clean/hacked flags), cost, and the controls checklist. Stored with bench results, comparable across runs; the Evolver (Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310) may consume calibrated external scores as fitness only with controls attached.
5. Leaderboard submission. After one fully-controlled Verified (or Multilingual) run: submit with the controls documented alongside the score. No submission off a Lite-only or uncontrolled run.

Test plan

• Adapter unit tests (dataset parsing, patch extraction from colony branches, scoring math, budget-cap stop).
• Dry run: 5-task Lite slice end to end, controls verified firing (e.g. a deliberately leaked answer is caught, not scored).
• Manual review of the first full Lite run before authorizing Verified spend.

Done when

• Lite run complete with controls + score record (the plumbing proof)
• Verified run complete, calibrated, with clean-vs-raw reporting
• Multilingual run complete with per-language breakdowns on our stacks
• Budget envelopes declared + enforced on all three; partial results honest
• Leaderboard submitted once, with controls documented
• docs/bench.md documents the external suites alongside internal tasks

Out of scope

• Multimodal (no vision surface in colonies); SWE-agent legacy scaffolding; synthetic task generation (separate SWE-smith issue); training models on the data (evaluation only).

Refs

• scripts/bench.mjs, scripts/bench/tasks.json, docs/bench.md
• Gold-sanity gate, held-out bench, trajectory monitor issues; Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310, Harness-aware cost routing and per-colony harness x model x dollar logging (colony-report, bench) #296

---

_{Credited to @colonizer-settlers as co-author on this issue.}

[ParallelEntrepreneur]

Colonizer colony f89da331 on host archlinux (f8832acd-9abe-4979-91c9-cf9a08e2ca28) claimed issue #331: starting a second colony on it duplicates its work.