Skip to content

orb(review): a paused repo fires the ai_review_public_summary_missing alarm on every pass #9692

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

runAiReviewForAdvisory refuses to run on a paused repo — src/queue/ai-review-orchestration.ts:580-586:

if (
  args.mode === "paused" ||
  args.settings.aiReviewMode === "off" ||
  !reviewableAuthor ||
  !args.advisory.headSha
)
  return undefined;

The predicate that decides whether a review is expected does not know about that stop. aiReviewWillRun (src/queue/processors.ts:11072-11086) is built from authorBlacklisted, isFrozenForManualReview, autoReviewSkipReason, oneShotPriorReview and shouldStartAiReviewForAdvisory(...) — and neither shouldStartAiReviewForAdvisory (ai-review-orchestration.ts:284-310) nor resolvePublicAiReviewGateSkipReason (:361-379) takes a mode argument at all. The pass's mode is resolved once at src/queue/processors.ts:9820 and is "paused" whenever agentPaused, the global pause, the global freeze, or a forced self-host paused instance mode is set (src/settings/agent-execution.ts:57).

On a paused repo the publish pass still runs (the gate verdict is computed; only GitHub writes are suppressed), so: aiReviewWillRun is true, and aiReviewExpected = aiReviewWillRun (:11087); runAiReviewForAdvisory returns undefined immediately; and :11801if (aiReviewExpected && (aiReview?.persistable === false || !hasPublicReviewAssessment(aiReview?.notes))) — is therefore true on every pass, writing a github_app.ai_review_public_summary_missing audit event and calling capturePostHogReviewFailure(new Error("AI review did not produce a public summary; ...")) (:11816).

That check exists to catch "a fresh review was expected and did not happen" — see its own comment at :11795-11800. A deliberately paused repo makes it a permanent false positive: a recurring error in the operator's error-tracking channel plus a growing audit population that buries the real signal. Secondary effect: :11230 (if (aiReviewWillRun && pr.headSha)) starts review-evasion tracking for an "active review" that will never run.

The three sibling per-PR advisory helpers in this same file all fold the stop in as their first line: runContentLaneDeliverableCheckForAdvisory (:8868), runVisualVisionForAdvisory (:9062), runScreenshotTableVisionForAdvisory (:9349). ai_review is the one member whose will-it-run predicate does not know about pause. No test covers the paused case: test/unit/queue-2.test.ts:1592 and :1673 assert a 0 count for non-paused fixtures only.

Requirements

  • Add "paused" to PublicAiReviewGateSkipReason (ai-review-orchestration.ts:352-359) and return it from resolvePublicAiReviewGateSkipReason, threading mode: AgentActionMode in as a required field on that function's args (the module already imports AgentActionMode).
  • Evaluate the paused stop first, before skip_ai_review_requested, matching the order runAiReviewForAdvisory uses, so the reason a PR reports matches the reason the executor would report.
  • shouldRequirePublicAiReviewForAdvisory and shouldStartAiReviewForAdvisory accept and forward mode; every call site in src/queue/processors.ts (lines 8213, 8256, 8289, 11077) and the resolvePublicAiReviewGateSkipReason call site at :11190 passes the pass's already-resolved mode.
  • Consequence that must hold: on a paused repo aiReviewWillRun is false, so :11801 does not fire, capturePostHogReviewFailure is not called, and review-evasion tracking at :11230 does not start.
  • The paused pass must still be explained in the ledger: the existing github_app.ai_review_public_gate_skipped audit event at :11195-11213 fires with reason: "paused". Do not add a new event type.
  • No behaviour change for mode === "live" or "dry_run".

⚠️ Required pattern: extend the EXISTING named-reason gate (resolvePublicAiReviewGateSkipReason -> shouldRequirePublicAiReviewForAdvisory -> shouldStartAiReviewForAdvisory -> aiReviewWillRun), mirroring how runContentLaneDeliverableCheckForAdvisory (processors.ts:8868) folds pause into one guard. It does NOT satisfy this issue to add a bare mode !== "paused" && term inline in the aiReviewWillRun expression with no named skip reason, leaving the pass unexplained in the audit trail; to special-case the alarm at :11801 while leaving aiReviewWillRun (and therefore :11230) wrong; or to add a second, parallel pause predicate.

Deliverables

  • PublicAiReviewGateSkipReason includes "paused", and resolvePublicAiReviewGateSkipReason({ mode: "paused", ... }) returns "paused" even when other preconditions would also fail — precedence case in test/unit/ai-review-advisory.test.ts.
  • shouldStartAiReviewForAdvisory returns false for mode: "paused" on an otherwise fully-eligible PR — new unit case.
  • A named regression test drives a full processJob webhook pass against a repo with agentPaused: true and asserts select count(*) from audit_events where event_type = 'github_app.ai_review_public_summary_missing' is 0, mirroring test/unit/queue-2.test.ts:1592/:1673.
  • The same test asserts exactly one github_app.ai_review_public_gate_skipped row whose metadata_json carries "reason":"paused".
  • The same test asserts no review-evasion tracking row was started for the head SHA.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example adding "paused" to the reason union and its unit test without threading mode into aiReviewWillRun, so the end-to-end alarm still fires — does not resolve this issue.

Test Coverage Requirements

99%+ Codecov patch coverage, branch-counted, on src/**; both touched files are inside coverage.include. Both arms of every new conditional need a test (mode === "paused" and not), plus the changed aiReviewWillRun composition on a paused and a live fixture. The end-to-end paused-pass test is the required named regression test. Measure coverage unsharded with npm run test:coverage — CI shards under-report a file this large.

Expected Outcome

Pausing a repo stops producing a recurring "AI review did not produce a public summary" error and a per-pass audit row that mean nothing. A paused pass records exactly one github_app.ai_review_public_gate_skipped row naming paused, and the missing-summary alarm goes back to meaning only what it was built to mean.

Links & Resources

src/queue/ai-review-orchestration.ts:284-310, :344-394, :580-586; src/queue/processors.ts:9820, :11072-11087, :11183-11220, :11230, :11801-11826, :8868, :9062, :9349; src/settings/agent-execution.ts:51-60; test/unit/queue-2.test.ts:1592, :1673.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions