fix(agent): AI CI-refutation — an AI-judgment failure never closes (or visibly blocks) a green-CI PR - #1331
Merged
Merged
Conversation
…(#ai-ci-refutation) A live dry-run audit of 44 metagraphed PRs found the AI reviewer false-closing ~60% of clean, green, mergeable contributor PRs: the dual free Workers-AI models share the same hallucination (e.g. inferring a JSON field is "required" from sibling data, or claiming "schema validation fails" / "the test count is wrong" on a PR whose CI is green), and the consensus/split blocker auto-CLOSES the PR. Grounding already feeds the finished CI status to the reviewer, but the model can ignore it — nothing enforced the deterministic ground truth. This makes the disposition enforce it: when the gate FAILED solely because of an AI-judgment blocker (ai_consensus_defect / ai_review_split) and the real CI is GREEN, the AI claim is refuted by the validator → the effective verdict is success → a clean + green PR MERGES. It is NOT routed to manual review (only the guardrail path holds), and it NEVER overrides a deterministic blocker (every blocker must be an AI-judgment code) or a red/unverified CI. The AI concern still surfaces in the review comment as an advisory; it simply no longer auto-closes a green PR. - agent-actions.ts: AI_JUDGMENT_BLOCKER_CODES + an effective `conclusion` that downgrades an AI-only failure to success when ciState==passed; threads through gatePassing, willClose, and the label-reason display (now reports verdict=success; CI green). - processors.ts: thread the gate blocker CODES into the planner, gated under the same grounding + convergence condition as the converged AI review that produces the defects (flag-off / non-convergence repo ⇒ codes omitted ⇒ byte-identical verdict).
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1331 +/- ##
=======================================
Coverage 95.23% 95.24%
=======================================
Files 185 185
Lines 20259 20270 +11
Branches 7286 7291 +5
=======================================
+ Hits 19294 19306 +12
Misses 378 378
+ Partials 587 586 -1
🚀 New features to boost your workflow:
|
…are the gate Completes the AI CI-refutation: the disposition (planAgentMaintenanceActions) merges a green-CI PR whose only gate failure was an AI-judgment blocker, but the review COMMENT was still rendered from the raw gate.conclusion=failure — so a merged PR would show a red "blocked/closed" headline + Gate panel row. reconcileGateEvaluationForGreenCi makes the comment render SUCCESS/advisory in that case, matching the action; the specific AI concern still surfaces from the advisory findings as a raised concern under the green verdict. To keep the two surfaces in lock-step and fully unit-testable (no uncovered branch in the processor): - advisory.ts: canonical AI_JUDGMENT_BLOCKER_CODES + isAiJudgmentOnlyFailure + reconcileGateEvaluationForGreenCi(gate, ciState, enabled). agent-actions now imports the shared code set instead of a local copy. - grounding-wire.ts: aiCiRefutationActive(env, repo) — the single grounding+convergence gate both the disposition and the comment use, so they can never disagree. Called as a plain function at each processor site (no inline && / ternary → no integration-only branch). - agent-actions.ts: the refutation is now gated by an explicit aiCiRefutationEnabled flag (unit-tested), and the blocker codes are resolved once so the nullish fallback is covered. - processors.ts: the comment block resolves live CI first, then reconciles the gate ONCE and uses it for both the panel rows and the comment body.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A live dry-run audit of 44 metagraphed PRs found the AI reviewer false-closing ~60% of clean, green, mergeable contributor PRs: the two free Workers-AI models share the same hallucination (e.g. inferring a JSON field is "required" from sibling data, or claiming "schema validation fails" / "the artifacts-test count is wrong" on a PR whose CI is green), and the consensus/split blocker auto-CLOSES the PR. Grounding already feeds the finished CI status to the reviewer, but the model can ignore it — and nothing enforced the deterministic ground truth.
This makes the engine enforce it, in both surfaces so they always agree:
planAgentMaintenanceActions): when the gate FAILED solely on an AI-judgment blocker (ai_consensus_defect/ai_review_split) and the real CI is green, the effective verdict issuccess→ a clean + green PR MERGES. Never manual (only guardrails hold), never overrides a deterministic blocker (a mixed failure still closes) or a red/unverified CI.reconcileGateEvaluationForGreenCi): the same condition renders the comment headline + Gate panel row as success/advisory instead of a contradictory red "blocked/closed". The specific AI concern still surfaces as a raised concern under the green verdict — visible, not silenced.Both gate on one shared predicate
aiCiRefutationActive(env, repo)(grounding AND convergence-allowlisted), so the action and the message can never disagree, and it's byte-identical for non-convergence repos.Closes #1331
Scope
src/only — pure helpers + flag-gated, branch-free call sitesValidation
npm run test:ci— exit 0; 4283 tests passnpm audit --audit-level=moderate— 0 vulnerabilities?? []defensive branch)agent-actions(consensus/split→merge, enabled-false→close, red-CI/mixed/deterministic/inconclusive→close, guardrail-still-held);rules(isAiJudgmentOnlyFailure + reconcile across enabled/ciState/code combos);grounding-wire(aiCiRefutationActive all 4 grounding×convergence arms)Safety