Evaluations: a run cannot stop at the hand over, so measuring a change of the interview pays for the whole loop

[PierreMardon]

Status

Closed on 2026-10-05: done by #90, and verified on one run. corpus --stop-at-hand-over plays planning alone. What it saves is the execution, which is a third of a run and not most of it: 0.90 USD of 2.83 on average, over the 17 runs of the three campaigns. Before that, campaigns 2 and 3 cost 2.6 and 3.4 USD per run on average, the whole loop each time, to measure changes that all sit in planning.

What was seen

corpus takes --case, --runs, --jobs and --developer-model, and always plays a run to its end: planning, approval, slices, gates, review. A change that touches only planning, the interview or the blueprint, is measured at the price of the execution too.

Orders of magnitude from campaign 1: a full run costs 1.53 to 4.43 USD at list price, 2.61 on average. In the trial run, the planning session of 01-overdue-list cost 0.78 USD of the run's 1.80.

Sources

• The command line: evals/surface_evals/cli.py lines 141 to 145.
• Report of campaign 1, committed: usd.
• Raw data, out of git: .evals/trial-1/report.md.

Next

Worth adding when a campaign targets planning alone: an option of corpus that stops a run once the blueprint is handed over and the developer has read it, with the measures and the judgements that need no execution (the interview, the blueprint). The first campaign after the fixes needs full runs anyway, since it also reads the hand back at conformity.

[PierreMardon]

Done by #90, and played on a real session

#90, merged: corpus --stop-at-hand-over plays planning as a run played whole does, the sentence that agrees included, then launches nothing and runs no acceptance test.

• The outcome of such a run is stopped-at-hand-over.
• Every measure the execution gives is empty for it, not 0: the report never reads it as a run that failed to conform, nor counts it in a spread of the runs played whole.
• The judge is asked of the interview and of the blueprint as before. Of the stops it is asked under dimensions of their own, messages-in-planning and following-in-planning, whose scores never join those of the runs played whole.
• A new measure, planning_usd, is the cost both kinds of runs share.

What it saves is a third of a run, not most of it. Over the 17 runs of the three campaigns, the sessions of the execution cost 0.90 USD per run on average (0.56 to 1.82), of 2.83: planning is 68% of a run. The figure this issue gave, 0.78 USD of 1.80 for planning, came from the trial run alone.

Played once, on 06-borrow-limit (.evals/validation-76/, out of git):

• outcome stopped-at-hand-over, five stops: the need, two answers, one correction, the sentence that agrees;
• the plan left awaiting-approval, with no plan-approved in its journal, and approved_by_sentence false;
• no acceptance test and no gate run;
• 9 sessions, 1.61 USD;
• judged in 4 sessions, 0.27 USD: no reason of the judge faults a stop for a launch or a delivery that never came;
• the report says that one run played planning alone, shows planning_usd, and no row of the execution.

Status

Closed: done by #90, verified on one run.