Evaluations: the container has no `gh`, so the pull request steps are never played and every message is scored with that absence in it

[PierreMardon]

Status

Closed on 2026-10-05: done by #91, verified on one run. The image holds a stand-in gh: the pull request steps are played, kept and measured, and no message speaks of a missing gh any more. A real remote is still #8. Before that, unchanged over three campaigns. The measures and their sources are in the comments below, campaign by campaign.

What was seen

• No session of the evaluations, nor of the end to end tests, opens a draft pull request, refreshes its description or reads one: the image has no gh and the remote is a bare repository on disk.
• Every hand over says that gh is missing, at up to four stops of a run, and the judge counts it as noise and as a fact repeated (following.no-noise, messages.right-length).
• The hand back at conformity speaks of a pull request that does not exist: that part is a defect of the chain and has its own issue, and it was found only because the container has no gh.

So the scores of the messages hold three things at once: what the chain says badly, what it says because of the container, and what the container hides.

Sources

• Evaluations of the chain: whether it meets its goals, and what it is like to work with #58, "Risks": "The container has no gh: every hand over says so, which the judge counts as noise. That part of the low scores on the messages is the environment's."
• Report of campaign 1, committed: following.no-noise.
• The image: tests/e2e/Dockerfile.

Next

Not decided. Two ways, neither free:

• a stand-in gh in the image, which records what it is asked and answers as GitHub would for the few calls the chain makes: the pull request steps are then played, and the messages are scored on the path most developers live;
• a real remote, as Exercise the pull request steps and the host CI snippet against GitHub #8 asks.

Until then, read the scores of the messages knowing that part of them is the container's.

[PierreMardon]

Read again in campaign 2

Campaign 2: 01-overdue-list and 03-overdue-reminders, three runs each, on main with the four first fixes (#78, #79, #80, #81). Report kept: evals/reports/2026-10-03-e830bc236e9b/report.md. Raw data, out of git: .evals/campaign-2/. In the tables, a value is the mean over the three runs, with its lowest and highest when they differ.

With the hand back fixed (#66), the missing gh still shows in the scores, differently: every first hand over prints or points to the pull request description, every last message sends the developer to a script, and the judge faults both: "it sends them to run an internal script for the pull request description instead of giving the text". On the path most developers live, with gh, none of these sentences is written.

Status

Unchanged: confirmed, not decided.

[PierreMardon]

Done by #91: a stand-in gh in the image, played on a real run

#91, merged. The image of the end to end tests and of the evaluations holds a stand-in gh, first on the PATH (tests/e2e/toy/gh_stand_in.py, copied into the image). It answers as gh does for the calls the chain makes and a few a model may add (pr create, pr edit, pr list, pr view, pr ready, repo view, auth status), on the repository of the current directory, and keeps every call in the git directory of the project. Any other call gets a clear error and is kept as not played. What it does not imitate is said in tests/e2e/README.md: GitHub itself, so no CI, no review, no merge.

The harness reads what it kept. Five measures of the evaluations:

• pr_draft_at_hand_over: at every hand over, the branch has a pull request and it is a draft;
• pr_described: at the stops of planning, its description is what surface-status pr-body prints;
• pr_refreshed: the same at the stops of the loop;
• pr_marked_ready: a call marked it ready, which the chain must never do;
• gh_not_played: the calls the stand-in could not answer, which say how far to trust a run.

The nominal scenario of the end to end tests asserts the description, the draft, and that nothing marks it ready.

Played on one whole run, 06-borrow-limit (.evals/campaign-4/runs/06-borrow-limit/run-02/, out of git; conformant, 9 hidden tests of 9):

• two calls kept, gh pr create --draft with a title and a description, then gh pr edit --body-file -, and none that marks the pull request ready;
• at the hand over, at the reply to the sentence that agrees and at conformity, the branch has its draft, and its description is the one the state script prints;
• the hand over now reads "The branch is pushed and the draft pull request is open: https://github.com/stand-in/lending/pull/1", and the hand back at conformity "I pushed the branch and refreshed pull request End-to-end trial on a real feature of a real host project #1's description. [...] You can mark the pull request ready whenever you want, which will trigger your CI." No message speaks of a missing gh.

So the pull request steps are played, and the messages are scored on the path most developers live. What their scores become is for the campaign under way to say (#77).

The other way this issue named, a real remote, is still #8: one run by hand against GitHub, and the CI snippet of the guide.

Status

Closed: done by #91, verified on one run.