Context
The dual-AI review combiner in src/services/ai-review.ts resolves disagreement between the two reviewers via a tie-break judge call (see the combine-strategy resolution around resolveAiReviewPlan and consensusDefectOf, where a consensus is only as strong as the weaker reviewer's confidence). That tie-break call presents both reviewers' outputs in a fixed order today. An LLM judge call is susceptible to position bias -- favoring whichever opinion appears first or second in the prompt regardless of content -- which means the current tie-break resolution could be an artifact of presentation order rather than a genuine adjudication of the disagreement.
Goal
Make the tie-break pass order-invariant by running it twice with the two reviewers' outputs swapped and only trusting a resolution that agrees across both orderings.
Requirements
- When the dual-AI combiner reaches the existing tie-break judge call, run it twice: once with reviewer[0]/reviewer[1] in the current order, once with the order swapped.
- Only accept the tie-break resolution if both calls converge on the same outcome (same reviewer favored / same consensus defect). Treat a swap-stable result as the trusted resolution.
- If the two orderings disagree (position bias detected), fall back to the existing conservative default that applies today when tie-break is inconclusive -- do not add a new fallback behavior, reuse what is already there.
- Bound the added latency/cost: this doubles the tie-break judge call only (not the two primary reviewer calls), and only fires in the code path where reviewers already disagree, not on every review.
- Log when a swap-instability triggers the conservative fallback, so position-bias frequency is observable rather than silent.
- Cover both branches (swap-stable acceptance, swap-unstable fallback) in tests, including a fixture pair of reviewer outputs constructed to be order-sensitive.
Deliverables
Expected outcomes
- Tie-break resolutions are provably invariant to reviewer presentation order in the covered test fixtures
- Position-bias fallback rate is visible in logs/metrics rather than undetectable
- No change in behavior or cost on the majority-path reviews where the two reviewers already agree
Context
The dual-AI review combiner in src/services/ai-review.ts resolves disagreement between the two reviewers via a tie-break judge call (see the combine-strategy resolution around resolveAiReviewPlan and consensusDefectOf, where a consensus is only as strong as the weaker reviewer's confidence). That tie-break call presents both reviewers' outputs in a fixed order today. An LLM judge call is susceptible to position bias -- favoring whichever opinion appears first or second in the prompt regardless of content -- which means the current tie-break resolution could be an artifact of presentation order rather than a genuine adjudication of the disagreement.
Goal
Make the tie-break pass order-invariant by running it twice with the two reviewers' outputs swapped and only trusting a resolution that agrees across both orderings.
Requirements
Deliverables
Expected outcomes