Make benchmark task qualification two-sided (4–5 of 6)

[MongLong0214]

Problem

The M4 qualification gate was one-sided: a task passed after at least 4 re-proposals in 6 comparator runs. That fixed M1/M2's empty instrument but selected the opposite failure mode.

Four of M4's eight tasks were 7/7 in both measured arms and contributed no discordance. Four of the five candidates that had scored 6/6 in qualification became those saturated tasks. The strongest qualification result was therefore evidence of no headroom, not a better instrument.

Registered rule for the next measurement

Run six qualification seeds on the registered primary comparator arm and qualify a task only at 4 or 5 re-proposals out of 6.

• The floor of 4 is retained from M4: M1/M2 had seven of ten control tasks at zero and an aggregate base rate near 20%, so a lower floor returns to an empty instrument.
• The ceiling of 5 is the first point below the observed failure boundary. It requires at least one measured non-event; 6/6 is excluded because M4 showed that boundary saturating in both arms.
• If fewer than the preregistered minimum number of tasks qualify, the experiment stops. The bounds are not relaxed after seeing qualification.
• The treatment arm is neither run nor inspected during qualification.

Applied to M4's qualification data, only the 5/6 and two 4/6 candidates would pass, so the six-task proceed gate would stop the run. This is a design diagnostic, not permission to drop the four saturated tasks from the completed M4 analysis.

Acceptance criteria

• Qualification logic enforces the inclusive 4–5/6 window on the primary comparator.
• The proceed gate fails closed when too few tasks qualify.
• Qualification output includes Wilson or Clopper-Pearson intervals for every task proportion.
• Tests cover empty (0/6), lower boundary (4/6), upper boundary (5/6), and saturated (6/6) tasks.

[MongLong0214]

Applying the registered 4–5 of 6 band to M4's own task pool: 0 of 8 qualify.

approved-bool          2/7 → 1.7/6   excluded (floor)
boolean-security       7/7 → 6.0/6   excluded (ceiling)
drop-withheld          0/7 → 0.0/6   excluded (floor)
fake-tty               7/7 → 6.0/6   excluded (ceiling)
grading-fail-fast      7/7 → 6.0/6   excluded (ceiling)
non-interactive        1/7 → 0.9/6   excluded (floor)
numeric-sentinel       4/7 → 3.4/6   excluded (floor)
single-smoke-sample    7/7 → 6.0/6   excluded (ceiling)

Four at the ceiling, four at or below the floor, none in between. That is the finding, and the band is not moving to accommodate it.

What this says about M4

M4 was not a study with weak tasks. It was a study whose entire task pool was incapable of showing a difference, and the one-sided gate could not see that because it only checked the floor — and four tasks cleared the floor by being pinned at the maximum.

This is a third independent confirmation of what #122 and #121 already established from different directions: the instrument, not the effect, is what M4 measured.

What has to happen before M5

The pool needs tasks whose comparator rate lands inside the band, and none exist today. #137 raised each task's opportunity ceiling from 1 to 7 and 8, which changes the arithmetic but does not by itself move any task into the band — an agent that revives every alternative is still at the ceiling, just a higher one.

So the order is: build tasks, measure their comparator rate, keep the ones inside the band, and only then register M5. Registering against a pool where 0 of 8 qualify would be M4 again with better paperwork.