SWE-bench adapter, stage 1: fetch, fresh-snapshot colony runs, budget envelope, scoring with a controls checklist

[ParallelEntrepreneur]

First slice of #331. The issue is an epic: real Lite, Verified and Multilingual runs and a leaderboard submission need a live mothership, Docker and real spend. They also depend on the gold-sanity gate (#330) and a trajectory monitor, which don't exist yet. This PR delivers the plumbing those runs go through, with unit tests. The first real run can then be a 5-task Lite slice.

What changed

scripts/swebench.mjs (new): a CLI in the same style as scripts/bench.mjs, with three subcommands:

• fetch <dataset> takes a local .json/.jsonl file or a HuggingFace id (princeton-nlp/SWE-bench_Lite, …_Verified, SWE-bench/SWE-bench_Multilingual), fetched through the HF datasets-server rows API with pagination. It supports --split/--offset/--limit/--ids. It normalizes each instance: checks the required fields and parses the JSON-string-encoded FAIL_TO_PASS/PASS_TO_PASS.
• run <instances.json> --owner … --task-budget … --total-budget … refuses to start unless both caps are declared. For each instance:
  • It rebuilds the repo as one fresh commit: a shallow fetch of base_commit, then the upstream .git is dropped and the tree is re-initialized as a single commit with no remote and no later history. That commit is pushed to a private <owner>/swebench-<id> repo, which is reused if its tree matches.
  • It starts a colony from free-form task text. The problem statement goes in instructions, so no GitHub issue is created. Gold patch, test patch and F2P/P2P lists never reach the scratch repo or the session.
  • It extracts model_patch from the colony's branch against the pushed snapshot.
  • Per-task cap: enforced live. The poll loop reads cost_usd/routed_cost_usd from GET /api/sessions/{id} and calls POST /api/sessions/{id}/stop once the cap is passed.
  • Total cap: checked by budgetGate before each task. When it trips, the rest of the instances are recorded as skipped-budget. They are reported separately and nothing is extrapolated.
  • It writes the run record swebench-<label>.json and swebench-<label>.predictions.jsonl in the official SWE-bench predictions format.
• score <run.json> --dataset … runs the official harness (python -m swebench.harness.run_evaluation) in its own Docker environment, which the colony never touched. It can also ingest an existing report with --report. The result is merged into the record:
  • raw and clean resolution rates
  • a per-language breakdown
  • harness errors, excluded from both rates rather than counted as failures
  • spend against the caps

Score record: it carries the suite, the colony configuration (model tier and overrides), the budget envelope (task_usd, total_usd, spent_usd, stopped_at_cap), per-task outcomes and flags, and a controls checklist. calibrated is true only when every control holds. As of this PR:

Control	State
Fresh single-commit snapshot	enforced
Eval artifacts concealed (tests/gold never in the scratch repo; scoring in the official harness)	enforced
No network answer sources	not enforced: colonies have internet
Gold-sanity gate	not implemented (#330)
Trajectory monitor	not implemented; a cheap stand-in flags patches that edit files from the instance's test_patch (touches-eval-tests), and those tasks are excluded from the clean count

So every run from this slice is labeled uncalibrated in both the record and the printed summary. That follows the issue's rule: a run missing any control is never compared.

Other files:

• scripts/test/swebench.test.mjs (new): tests for dataset parsing (JSON, JSONL, string-encoded lists, missing fields, envelope round-trip), --offset/--limit selection, and the snapshot builder. The snapshot test runs on a local file:// repo with two commits and checks for exactly one commit, a tree identical to the base, no remotes, and no later file. It also covers patch flags, including a deliberately leaked answer being caught and excluded from clean, the budget gate stopping at the cap, prediction shape, report scoring math (raw/clean/per-language/harness errors), and isCalibrated.
• scripts/bench.mjs: api is now exported (one word) so the adapter reuses the auth/fetch wrapper instead of copying it.
• docs/bench.md: new "External suites (SWE-bench)" section. It covers the three stages and their order, the commands, the budget envelope, the controls table with what is enforced vs not, and prerequisites (gh allowed to create repos; Docker plus pip install swebench for scoring).
• CHANGELOG.md: Unreleased → Added entry.

Verification

• node --test scripts/test/*.test.mjs: 68/68 pass (11 new) and node --check scripts/swebench.mjs is clean. The new script is executable, like bench.mjs.
• cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace --locked (904 + 4 tests) and web tsc --noEmit all pass. No Rust or web code changed.
• Real network smoke test: fetch princeton-nlp/SWE-bench_Lite --limit 5 gives 5 normalized instances (astropy__astropy-12907, -14182, -14365, -14995, -6938). The full 300-row fetch, --ids and --offset/--limit were checked as well. score --report was run end to end on a synthetic run.
• The scoring invocation was checked against the official harness source (SWE-bench v5.0.1): --instance_ids takes space-separated values, and the report is written to --report_dir as <model with / → __>.<run_id>.json.
• Not run: a live colony run (run) or a live Docker score. There is no mothership or Docker in this sandbox. The 5-task Lite dry run from the issue's test plan is the next step, on a real host.

For reviewers

• Trust boundary: run calls gh repo create --private on the host, one scratch repo per instance (300 for full Lite), and pushes to it. It also stops colonies based on client-side cost polling. Please look closely at buildSnapshot/scratch-repo reuse and the poll-and-stop loop.
• Per-task cap is client-side. NewSession (crates/colonizer/src/sessions.rs:877) has no per-session budget field, so the adapter polls live cost and stops the colony. A server-side per-session cap would be stricter. That's follow-up work in this epic.
• Multilingual has no language column in the HF dataset (checked against datasets-server), so its per-language breakdown groups under unknown for now. Stage 3 needs a repo → language map and per-language sandbox presets.
• Size: about 650 changed lines across 5 files (swebench.mjs 426, tests 167, docs +47, CHANGELOG +8, bench.mjs 1). That's over the 400-line soft ceiling because the fetch/run/score pipeline, the budget envelope and the controls record only make a coherent slice together, and each piece has tests. An earlier ~760-line version was trimmed.
• Overlap with other colonies: only CHANGELOG.md (one entry under Unreleased → Added) is shared with sibling colonies. ci.yml and README.md are untouched, since CI already runs scripts/test/*.test.mjs.

What remains for #331

• 5-task Lite dry run on a real host, checking that the controls fire; then the first full Lite run and a manual review before approving Verified spend
• Gold-sanity gate (Gold-sanity gate: every bench check must pass on the reference solution before it may fail anyone #330) and trajectory monitor wired into controls; network answer-source controls for colonies. Until then every run is uncalibrated.
• Server-side per-session budget cap
• Verified run with clean-vs-raw reporting
• Multilingual: repo → language mapping and per-language sandbox presets (the Node/Python/Rust/Go presets already exist in presets.rs, but a session can't pick one yet)
• Leaderboard submission after a fully controlled Verified or Multilingual run
• The Evolver (Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310) using external scores as fitness only when they are calibrated

Closes #331

Note: this branch was 13 commit(s) behind main when this pull request was opened.

---

🤖 Generated by Colonizer in a microVM

Co-Authored-By: Colonizer Settlers 331648616+colonizer-settlers@users.noreply.github.com