Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline

[ParallelEntrepreneur]

Goal

Close the loop the harness is two-thirds of the way to owning: red-team raids find defects, the bench verifies fixes — add the missing middle, an offline evolution loop that diagnoses failures, proposes harness/prompt/config changes, and retains only what beats the baseline in reproducible evaluations. It develops the harness; it never runs inside production colonies.

Current state (with pointers)

• Find side: red-team runs (crates/colonizer/src/redteam.rs, docs/red-team.md) with hunter findings in findings.jsonl (protocol §6.6); validation pending (#211).
• Verify side: the bench (scripts/bench.mjs, docs/bench.md) — fixed tasks a change must survive, scored the same way every time; pairs with scripts/colony-report.mjs (what happened inside colonies: cost, tools, questions, silences).
• Missing middle: today a human reads both and hand-writes the fix. Nothing proposes a prompt/module/config change from a red-team finding or a bench regression, and nothing re-runs the bench to confirm the proposal. The pieces exist but the loop is manual.
• Neighbors: Validate hunter findings with an orchestrator before filing; autofix and merge behind an independent PR review #211/Security-hunter modules: Strix and Shannon, downloaded on demand, findings filed as validated issues #216 (validated findings become the evolver's input feed), Harness-aware cost routing and per-colony harness x model x dollar logging (colony-report, bench) #296 (cost logging — evolved changes must be cost-aware, not just score-aware).

Requirements

1. Offline loop, four stages. (a) Diagnose: ingest validated red-team findings + bench failures + colony transcripts and cluster them into failure classes (which prompt/module/setting is implicated, with evidence links). (b) Propose: generate candidate changes scoped to one surface each (a prompt paragraph, a module default, a docs correction — never code + prompt + config in one candidate). (c) Evaluate: run each candidate through the bench (fixed tasks, same scoring) plus the cost accounting from Harness-aware cost routing and per-colony harness x model x dollar logging (colony-report, bench) #296. (d) Retain: keep a candidate only if it beats the baseline on score without regressing cost past a stated bound; everything else is discarded with its scorecard kept for the record.
2. Never in production. The loop runs offline (operator-invoked, then scheduled at most daily): it reads logs, transcripts, and findings, and writes proposals (diffs + scorecards) for human review. It launches no colonies, edits no live config, opens no PRs by itself. Its output is a proposal queue the operator approves item by item — the same review posture as colony output at publish.
3. Consumes the harness as a library. Drive the bench and report tooling through their existing entry points (scripts/bench.mjs, colony-report.mjs) rather than reimplementing scoring. Proposals reference the exact bench tasks + seed so any result reproduces from the command line.
4. Baseline discipline. Every proposal records: baseline score (commit-pinned bench run), candidate score (same tasks, same commit + candidate), delta per task (no hiding a regression inside an average), cost delta (tokens + dollars per the spend record), and the failure classes it claims to address. A proposal that improves the average while regressing any single task is flagged, not retained silently.
5. Proposal format. One file per proposal: the diff, the claim (which failure class, from which findings), the scorecard (baseline vs candidate per task + cost), and a review checklist (docs updated? pins affected? audit implications per docs/audit.md?). Approved proposals become normal PRs through the existing review path.
6. Start narrow. First instantiation: prompt-only candidates for one agent module, driven by one bench task family. Generalizing to module defaults and config comes only after the loop has retained three proposals whose bench gains reproduce on re-run.

Test plan

• Unit: diagnose-clustering on fixture findings (known classes recovered), scorecard math (per-task deltas, cost bounds, flag-on-single-regression), proposal serialization round-trip.
• Reproducibility: same proposal + same commit re-run yields the same verdict (recorded twice in the PR that builds it).
• Manual: one full loop on a real red-team finding end to end, proposal reviewed by a human, bench gain confirmed by hand.

Done when

• Four-stage loop implemented and documented (diagnose → propose → evaluate → retain)
• Proposals are diffs + scorecards for human approval; loop launches nothing and edits nothing live
• Baseline/candidate/delta/cost recorded per task; single-task regressions flagged
• First instantiation scoped to prompt-only candidates ships with three reproduced retentions
• Docs page merged describing the loop, its safety posture, and how to run it

Out of scope

• Autonomous application of proposals (explicitly not this — approval stays human); the validation pipeline (Validate hunter findings with an orchestrator before filing; autofix and merge behind an independent PR review #211); cost-routing changes (Harness-aware cost routing and per-colony harness x model x dollar logging (colony-report, bench) #296 — consumed, not built, here).

Refs

• scripts/bench.mjs, docs/bench.md, scripts/colony-report.mjs
• crates/colonizer/src/redteam.rs, docs/red-team.md, docs/audit.md
• Validate hunter findings with an orchestrator before filing; autofix and merge behind an independent PR review #211, Security-hunter modules: Strix and Shannon, downloaded on demand, findings filed as validated issues #216, Harness-aware cost routing and per-colony harness x model x dollar logging (colony-report, bench) #296

---

_{Credited to @colonizer-settlers as co-author on this issue.}