Port schema-harness behavioral trace schema + predict–verify loop into codegraff trajectories

[justrach]

Summary

Port the behavioral trace schema and predict–verify control loop from the ARC-AGI-3 schema-harness trajectories into codegraff's .graff/trajectories, so our runs record what the agent believed and why it changed its mind — not just how fast it was. Then close the loop by feeding recomputable scores into the fleet/DGM machinery so the harness can improve over time automatically.

cc @yxlyx

Data source

Reference dataset: https://huggingface.co/datasets/schema-harness/arc-agi-3-schema-traces

50 ARC-AGI-3 gameplay trajectories (25 GPT-5.6 Sol @ 95.35% mean RHAE; 25 Claude Opus 4.8 / Fable 5 @ 98.98%, 183/183 levels). Each trajectory ships run.json, a streamed events.jsonl, sanitized session JSONL, per-level snapshots, the agent's own notes.md + world_model.py, and a stdlib-only score_trajectories.py that re-derives every score from events alone and verifies it against the manifests.

A perfect run (claude_fable_opus/claude-fable-5_max_ft09_100.0: 6 levels in 78 actions, 100% RHAE) was dissected for this analysis.

What the winning traces do that we don't

1. Rich, recomputable event schema (9 kinds, seq+ts)

run_started, turn_started (full env state + legal actions), text_delta (streamed reasoning), tool_started/tool_finished (with args), turn_committed (plan + reason), action_taken (action + resulting state), model_mispredicted (predicted vs actual + surprise message), run_finished (outcome summary). The trace alone is sufficient to recompute the score — the scorer is the audit.

2. Beliefs as files, re-injected every turn

The agent maintains notes.md from a fixed template — Action semantics (confirmed/guessed), Current level, Hypotheses to test, Confirmed facts — and the harness re-injects it into every user turn. Memory lives in the file, not the context window; hypotheses are explicitly labeled GUESS vs CONFIRMED so the agent never acts on speculation without saying so.

3. Executable world model + backtest ratchet

The agent writes world_model_vN.py implementing predict(state, action) → next_state, snapshotted per milestone (snapshots/cleared_level_N.py). A run_backtest tool replays all observed transitions through the model; the agent is not allowed to plan until the backtest is green over full history (observed progression: 3/3 → 10/10 → 24/24 → 43/43 → 44/44). Plans are simulated against the model (BFS) before any real action is committed.

4. Mispredict = hard stop

The harness checks each executed action against the model's prediction. On mismatch it emits model_mispredicted, drops the remainder of the committed plan, and forces model repair + re-backtest before acting again. ft09's entire exploration cost was 9 mispredicts ≈ 9 wasted actions out of 78. Every committed plan also carries a stated hypothesis (turn_committed.reason), making exploration deliberate.

5. Honest scope assessment

• What improves automatically: the agent's artifacts, within one run (the ratchet).
• What improves manually: the scaffold (prompt/tools/scorer), across runs, by humans reading traces. The dataset + re-scorer make that loop measurable.
• What never changes: model weights.
• Portability: the event schema, notes pattern, and predict–verify loop are task-general; the world-model/BFS simulation is ARC-specific. For coding tasks, the compiler/tests/linter are the verifier — no simulation needed. General form: state belief + expectation → act → verifier checks → surprise ? stop & repair : continue.

Where codegraff stands today

Piece	Location	Records
Perf trace	.graff/traces/<run>.jsonl (src/trace.zig)	startup_phase, api (ms, bytes, context_tokens, cache_read), tool (name, ms, result_bytes) — performance only
Trajectory	.graff/trajectories/<run>.jsonl	session, recipe (provider/model/effort + prompt_sha/toolset_sha), prompt, turn (ms, task, context_tokens, model_calls, tool_errors, ok), turn_error — stats only, zero content
Eval log	.graff/eval-log.tsv	one TSV line per eval; write-once, not recomputable
Scratch memory	codedb-pro memo tool	findings/plans/errors; unstructured, not re-injected per turn
Fleet / DGM	src/fleet.zig, examples/dgm_loop.py	prompt-variant tournaments + eval selection — the self-improvement machinery schema-harness lacks

Capability	schema-harness	codegraff	Gap
Tool calls with args	✅	❌ (name+ms only)	🔴
State at each turn	✅ full	❌	🔴
Plan + reason at commit	✅	❌	🔴 biggest miss
Expectation/mispredict events	✅	❌	🔴
Notes re-injected per turn	✅ templated	🟡 memo, unstructured	🟡
Run outcome event	✅	❌	🔴
Score recomputable from events	✅ stdlib scorer	❌	🔴
Perf telemetry	❌	✅ excellent	🟢 we win
Recipe reproducibility (shas)	🟡	✅	🟢 we win
Auto scaffold improvement	❌ manual	✅ fleet/DGM exists	🟢 we win

Proposal

1. Task-agnostic event schema — adopt the 9 event kinds in the trajectory writer; state/action/expect as opaque JSON; only kind/seq/ts mandatory. Extend src/trace.zig / trajectory writer; keep the existing recipe line.
2. Expectation field, not world models — any committed action may carry expect:; a contradicting observation logs mispredicted. Zero cost when unused; no simulation required.
3. Verifier as per-recipe hook — games get run_backtest, coding gets zig build test / the eval command; same stop-on-red-repair-resume contract, pluggable command.
4. Notes template per task class — CONFIRMED/GUESS/HYPOTHESES/FACTS sections, recipe-configurable, memo-backed so it survives compaction; strict mode for eval/benchmark runs, opt-in for interactive.
5. Recomputable scorer interface — per-task metric, one invariant: score must be recomputable from events alone (a scripts/score_run.py-style stdlib verifier, mirroring score_trajectories.py).
6. Close the loop — feed recomputed behavioral scores into fleet/DGM selection so scaffold variants evolve on decision-quality traces, not just perf. This is the "harness improves over time" piece schema-harness never built.

Suggested phases

• Phase 1: items 1–2 (schema + expectation/mispredict) — the foundation everything hangs on.
• Phase 2: items 3–4 (verifier hook + notes template).
• Phase 3: item 5 (re-scorer wired into graff --eval).
• Phase 4: item 6 (fleet/DGM integration).

Validation

Replay the dissected ft09 trajectory through the new schema and prove the emitted events reproduce its published RHAE score (100.0) — round-trip auditability from day one.

References

• Dataset: https://huggingface.co/datasets/schema-harness/arc-agi-3-schema-traces
• Dissected run: claude_fable_opus/claude-fable-5_max_ft09_100.0 (6 levels, 78 actions, 9 mispredicts, 100.0 RHAE)
• Scoring formula: per-level min(115, 100*(h/a)^2), weighted mean by level, completion cap → RHAE