An interview recommendation rests on a claim that the host's README contradicts

[PierreMardon]

Status

Closed on 2026-10-05, not planned: not a fault of the chain. Read again in the raw data: the session had read the categories, and the sentence it wrote is true of the host. What the README contradicts is the judge's paraphrase of it. No fix. 06-borrow-limit still has one run: no later campaign plays it, and no session was played for this reading. The reading and its sources are in the comment below.

What was seen

06-borrow-limit, interview.not-already-answered 4 of 5. The judge: "The question itself was open: neither the need nor the code sets a limit. But the recommendation claims the specs and README make no distinction between members, while the README's Data section already declares the student and staff categories, so it steered toward a default the code argues against."

Passage: "Recommended: A, the specs name no distinction between members and the README rules treat them alike; 5 is a common library default, but the number is the developer's to confirm."

The question was the right one, and the simulated developer answered from its brief (3 for a student, 6 for staff), so the run was not harmed. A developer who answers in a word, as the prompt invites, would have taken the recommendation.

Sources

• Report of campaign 1, committed: interview.not-already-answered.
• Raw data, out of git: .evals/campaign-1/runs/06-borrow-limit/run-01/judge.json, run.json.

Suspected cause

None located. "Asking a question" (skills/surface-plan/SKILL.md line 57) asks for a recommendation with its reason in one line, and says nothing of what that reason must rest on.

Next

Watch the recommendations in the next campaign, with 06-borrow-limit in it: a second occurrence would make it worth a sentence in "Asking a question".

[PierreMardon]

Read again in the raw data: the sentence written is true, the judge paraphrased it

Read on 2026-10-05, in the streams and the plan folder of the run, .evals/campaign-1/runs/06-borrow-limit/run-01/. No session was played: whether it comes back on a new run is not measured.

The session had read the categories. Before it wrote interview.md, the stream of its first session (logs/01-need.jsonl) shows a read of the whole README.md, a read of lending/members.py and a cat data/members.jsonl. Its exploration.md says "Members have a category, student or staff", and lists as open "the limit itself (its value, and whether it is the same for every member or depends on the category)". The question offers it: "C. A limit per category (student and staff), numbers to be given."

The sentence it wrote is not the one the judge reports.

• Written: "Recommended: A, the specs name no distinction between members and the README rules treat them alike; 5 is a common library default, but the number is the developer's to confirm."
• The judge: "the recommendation claims the specs and README make no distinction between members".

Both halves of the written sentence hold against the host. The need is "Members should not be able to hold too many books at once." No rule of the "Rules" section of the README depends on the category, and in lending/ only members.py names category, to load it. The README tells members apart in its "Data" section and treats them alike in its "Rules": the session said the second, and the judge read the first.

It is not the slip of one pass. In the judgements of that run made again for #74, the same judge faults the recommendation in each of its three passes, twice under interview.not-already-answered and once under interview.questions-matter, and the other model in none. The sentence is easy to read as more than it says.

What is left is a disagreement with the pick. "it steered toward a default the code argues against" is said with the brief in hand. The host argues neither way, and the developer's answer, C with 3 and 6, overrode the recommendation, which is what the question is for. On 04-suspension the same kind of pick, one limit for everyone, is the one its brief holds: "Staff are suspended like students".

No other recommendation was found to rest on a claim the host contradicts. The 20 recommendations of the 18 runs on disk were read again for this, the trial and the three campaigns, where 11 runs ask at least one question. Most cite the specs, the README or the code; the others give a reason of judgement.

So no change of the prompts. A sentence saying that the reason of a recommendation must rest on what the specs or the code say would not have changed this run, whose reason did. A sentence asking to name what argues against the pick makes a recommendation that hedges, where the fact was already in option C.

What it shows of the evaluations: the rubric has no criterion on recommendations, so the remark landed under two criteria about questions.

Status

Not confirmed as a fault of the chain: the claim the README contradicts is the judge's paraphrase, not the sentence written. No fix. Not replayed: 06-borrow-limit still has one run.

[PierreMardon]

Closed

Not a fault of the chain: the sentence the session wrote is true of the host, and the claim the README contradicts is the judge's paraphrase of it, as the comment above shows. No fix, and none planned.

Two things this closure leaves as they are, neither a reason to keep it open: 06-borrow-limit still has one run, and the rubric has no criterion on recommendations.

[PierreMardon]

Campaign 4: the case played again, twice

06-borrow-limit was played twice in campaign 4 (.evals/campaign-4/, out of git). The question comes back in both runs, and neither recommendation rests on a claim the host contradicts:

• "Recommended: C, because the data already distinguishes student from staff; but a single number is simpler if you do not want the categories to differ."
• "Recommended: A, with 5 books: the simplest rule that answers the specs, which name no distinction between members."

The pick differs from one run to the other, and the developer answers by category in both.