Trajectory monitor: clean vs hacked resolution scoring for bench and Evolver fitness

[ParallelEntrepreneur]

Goal

Score how a colony worked, not just what it produced: audit the recorded trajectory (commands, file/network/git access) for shortcut patterns and report clean vs hacked resolution alongside pass/fail — so gaming the checks is measured, and the Evolver loop optimizes a fitness function that can't be gamed by construction.

Current state (with pointers)

• The event log already records everything a trajectory audit needs: tool calls and results, file reads/writes, subagent spawns, questions, turn ends (docs/protocol.md §2/§3, docs/agent-events.schema.json, crates/colonizer/src/protocol.rs, events.rs). colony-report.mjs narrates trajectories; nothing scores them.
• Scores today are outcome-only: bench pass/fail, findings counts, "done" claims. An agent that mines history for answers, edits tests to fit, or shapes the verification environment scores identically to one that did the work — the reward-hacking literature shows this is not hypothetical (history-mined patches, test-rewriting shims, verifier-aware edits), and fixed benchmarks concentrate the incentive.
• Neighbors: Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310 (Evolver — this monitor is its gaming-resistant fitness input), the held-out bench issue (outcome hardening; this is process hardening — both needed), origin-tagged events (Origin-tagged colony events (user/agent/watchdog/autonomy/scheduler/system) across log, transcript and logic #312 — attribution needs the tags), Validate hunter findings with an orchestrator before filing; autofix and merge behind an independent PR review #211 (reproduction requirement for findings).

Requirements

1. Shortcut pattern set, documented. A closed, versioned list of trajectory patterns that invalidate a success: repository-history mining for answers (suspicious git log/object reads beyond normal workflow), test-file edits that weaken assertions, verifier/shim-adjacent writes (test runners, config hooks, anything the scorer executes), network fetches of solution-shaped content during the task, and prompt-injection-shaped inputs acted on without flagging. Each pattern states its normal-workflow lookalikes (reading history to understand code is fine; extracting a patch from it is not) — the line is the point of the document.
2. Two scores per evaluated run. resolved (existing outcome: checks passed) plus clean_resolved (passed with no pattern fired) — and the gap between them is the headline metric, reported per task family and per agent/module. A monitor firing on a passing run downgrades it to hacked-resolved: counted separately, never silently, never averaged away.
3. Monitor runs on records, never live. The audit consumes the persisted event log after the turn/run ends (or on a settled transcript) — it cannot perturb the colony, cannot be bargained with mid-turn, and its own operation is itself logged. Live blocking stays with the existing gates (preflight, publish sanitizer, choice cards); this is scoring, not enforcement.
4. Calibrated, not paranoid. Every pattern ships with a false-positive budget evaluated against real colony transcripts: a pattern that fires on normal work more than the stated rate is demoted to advisory (reported, doesn't affect the score) until refined. The calibration set and rates are committed alongside the patterns — a monitor nobody trusts is worse than none, because it teaches operators to ignore scores.
5. Evolver feed. Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310 consumes clean_resolved (not raw pass rate) as fitness, plus the gap as a secondary signal: a proposal that raises raw scores while widening the gap is rejected. The scorecard format from Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310 gains the two columns.
6. Privacy of the mechanism (limited). The exact pattern list is operator-visible (this repo is open source; obscurity is not the defense — freshness and rotation are, per the held-out issue). Document that stance explicitly so nobody mistakes openness for weakness.

Test plan

• Unit: each pattern against fixture transcripts (fires on the exploit, silent on normal-work lookalikes), scoring math (resolved/clean/gap), calibration harness (false-positive rate measured on a labeled transcript set).
• Adversarial fixtures: history-mined patch, weakened assertion, shim-adjacent write, solution-shaped fetch — each must flip its run from resolved to hacked-resolved.
• Manual: run the monitor over a sample of real red-team/bench transcripts; review the gap report for false positives before enforcing anything on scores.

Done when

• Versioned pattern set merged with normal-workflow boundaries stated per pattern
• Clean/hacked resolution scoring implemented, gap reported per family/module
• Monitor runs post-hoc on persisted logs; its operation logged
• Calibration set committed; over-firing patterns demoted automatically by rate
• Evolver consumes clean scores; scorecard carries both columns
• Openness stance documented

Out of scope

• Live blocking on patterns (existing gates own enforcement); new bench tasks; changing event capture (the log already has what the monitor needs).

Refs

• crates/colonizer/src/protocol.rs, events.rs, docs/agent-events.schema.json
• scripts/colony-report.mjs, scripts/bench.mjs, docs/bench.md
• Offline evolver loop: diagnose failures, propose changes, retain what beats the bench baseline #310, Red-team synthesis step: merge hunters findings into one ranked report when the run ends #309, Origin-tagged colony events (user/agent/watchdog/autonomy/scheduler/system) across log, transcript and logic #312, Validate hunter findings with an orchestrator before filing; autofix and merge behind an independent PR review #211