Skip to content

fix(review): add order-swapped stability check to the dual-AI tie-break pass #2997

Description

@JSONbored

Context

The dual-AI review combiner in src/services/ai-review.ts resolves disagreement between the two reviewers via a tie-break judge call (see the combine-strategy resolution around resolveAiReviewPlan and consensusDefectOf, where a consensus is only as strong as the weaker reviewer's confidence). That tie-break call presents both reviewers' outputs in a fixed order today. An LLM judge call is susceptible to position bias -- favoring whichever opinion appears first or second in the prompt regardless of content -- which means the current tie-break resolution could be an artifact of presentation order rather than a genuine adjudication of the disagreement.

Goal

Make the tie-break pass order-invariant by running it twice with the two reviewers' outputs swapped and only trusting a resolution that agrees across both orderings.

Requirements

  • When the dual-AI combiner reaches the existing tie-break judge call, run it twice: once with reviewer[0]/reviewer[1] in the current order, once with the order swapped.
  • Only accept the tie-break resolution if both calls converge on the same outcome (same reviewer favored / same consensus defect). Treat a swap-stable result as the trusted resolution.
  • If the two orderings disagree (position bias detected), fall back to the existing conservative default that applies today when tie-break is inconclusive -- do not add a new fallback behavior, reuse what is already there.
  • Bound the added latency/cost: this doubles the tie-break judge call only (not the two primary reviewer calls), and only fires in the code path where reviewers already disagree, not on every review.
  • Log when a swap-instability triggers the conservative fallback, so position-bias frequency is observable rather than silent.
  • Cover both branches (swap-stable acceptance, swap-unstable fallback) in tests, including a fixture pair of reviewer outputs constructed to be order-sensitive.

Deliverables

  • Order-swapped double tie-break call in the dual-AI combiner's disagreement path
  • Swap-stability check gating acceptance of the tie-break resolution
  • Reuse of the existing conservative fallback on swap instability (no new fallback path)
  • Observability signal for swap-instability occurrences
  • Tests for both swap-stable and swap-unstable cases using order-sensitive fixture reviewer outputs

Expected outcomes

  • Tie-break resolutions are provably invariant to reviewer presentation order in the covered test fixtures
  • Position-bias fallback rate is visible in logs/metrics rather than undetectable
  • No change in behavior or cost on the majority-path reviews where the two reviewers already agree

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

    Projects

    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions