HARD_FPR_CEILING = 0.98 in sdk/benchmarks/attacks/ci_gate.py:53 lets the hard-negative slice fail on 39 of its 40 prompts and still report a pass.
Current gate output on dev:
| slice |
n |
false positives |
FPR |
ceiling |
| easy |
485 |
1 |
0.0021 |
0.02 |
| hard |
40 |
39 |
0.975 |
0.98 |
The ceiling was pinned just above the measured rate when the hard negatives landed in #150, which was the right call at the time: it makes the number a ratchet that can only move down. But nothing has moved it, and at 0.98 the assertion is satisfied by a detector that flags essentially every hard negative. It records a number rather than gating on one.
Two things worth separating here.
Whether 39/40 is even the right thing to gate on. These are prompts picked because they trip the patterns, so a high rate is expected and a ceiling near zero would be wrong. What is missing is a target: some rate below which we would call the hard-negative behaviour acceptable, with a plan to get there. Without that the ratchet has no destination.
And whether it belongs in a pass/fail gate at all. The alternative is demoting it to a reported metric that shows up in the gate output without voting on the result, and putting the pass/fail weight on the easy slice where 0.02 is a real constraint.
Related: the two patterns doing most of the damage are known. On the NotInject benign corpus, all 12 regex-only false positives came from developer_mode (7) and persona_replacement (5), neither of which looks at what surrounds the trigger word. Adding context guards there is the most likely way to actually move this number.
No fix proposed yet. This needs a decision on target-or-demote before the constant is worth touching.
HARD_FPR_CEILING = 0.98insdk/benchmarks/attacks/ci_gate.py:53lets the hard-negative slice fail on 39 of its 40 prompts and still report a pass.Current gate output on dev:
The ceiling was pinned just above the measured rate when the hard negatives landed in #150, which was the right call at the time: it makes the number a ratchet that can only move down. But nothing has moved it, and at 0.98 the assertion is satisfied by a detector that flags essentially every hard negative. It records a number rather than gating on one.
Two things worth separating here.
Whether 39/40 is even the right thing to gate on. These are prompts picked because they trip the patterns, so a high rate is expected and a ceiling near zero would be wrong. What is missing is a target: some rate below which we would call the hard-negative behaviour acceptable, with a plan to get there. Without that the ratchet has no destination.
And whether it belongs in a pass/fail gate at all. The alternative is demoting it to a reported metric that shows up in the gate output without voting on the result, and putting the pass/fail weight on the easy slice where 0.02 is a real constraint.
Related: the two patterns doing most of the damage are known. On the NotInject benign corpus, all 12 regex-only false positives came from
developer_mode(7) andpersona_replacement(5), neither of which looks at what surrounds the trigger word. Adding context guards there is the most likely way to actually move this number.No fix proposed yet. This needs a decision on target-or-demote before the constant is worth touching.