You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Empirical finding from the 2026-07-26 backfill adjudication (331 historical decisions, three repos).
The AI defect-finding confidence is calibrated to "is the defect real" — but close-correctness depends on a second, independent axis: is the PR salvageable and the author responsive. The data: 28 contributor PRs held on real, high-confidence defect findings (0.55–0.97) later MERGED — 17 as-is (finding refuted) and 11 after the author fixed the flagged defect (finding right, but a close would still have denied a real contribution). Meanwhile acted closes at the same confidences ran 96.0% correct. Same confidence, opposite close-correctness, split by salvageability.
Implication for the 99.5% goal: raising the confidence floor alone cannot get closes past ~96% — the residual error class is 'real defect, salvageable PR'. The close criterion should combine:
a salvageability score: author track record (AMS bridge already exists), defect class fixability (mojibake/dead-import = trivially fixable vs fabricated-test/scope-violation = disqualifying), PR freshness/responsiveness.
Requirements:
Define the salvageability signal as a deterministic, explainable score (no new AI call) computed at decision time and persisted in the decision record (schema bump).
Gate wiring: high-confidence defect + high salvageability → hold-with-guidance instead of close; high-confidence + low salvageability → close as today. Manifest knob for the salvageability threshold, following the config-as-code pattern.
The decision-audit rubric gains a reason category for the new boundary so future labels measure it.
Advances #8828.
Empirical finding from the 2026-07-26 backfill adjudication (331 historical decisions, three repos).
The AI defect-finding confidence is calibrated to "is the defect real" — but close-correctness depends on a second, independent axis: is the PR salvageable and the author responsive. The data: 28 contributor PRs held on real, high-confidence defect findings (0.55–0.97) later MERGED — 17 as-is (finding refuted) and 11 after the author fixed the flagged defect (finding right, but a close would still have denied a real contribution). Meanwhile acted closes at the same confidences ran 96.0% correct. Same confidence, opposite close-correctness, split by salvageability.
Implication for the 99.5% goal: raising the confidence floor alone cannot get closes past ~96% — the residual error class is 'real defect, salvageable PR'. The close criterion should combine:
Requirements:
src/**lines; golden-corpus entries (eval: versioned golden replay corpus gating every gate/prompt/model/config change #8832) for both sides of the new boundary.