Skip to content

fix(review): order-swapped dual-AI tie-break stability - #3569

Merged
JSONbored merged 8 commits into
JSONbored:mainfrom
RealDiligent:fix/review-dual-ai-tiebreak-order-stability
Jul 5, 2026
Merged

fix(review): order-swapped dual-AI tie-break stability#3569
JSONbored merged 8 commits into
JSONbored:mainfrom
RealDiligent:fix/review-dual-ai-tiebreak-order-stability

Conversation

@RealDiligent

Copy link
Copy Markdown
Contributor

Summary

Test plan

  • npm run typecheck
  • npx vitest run test/unit/ai-review.test.ts

Closes #2997

RealDiligent and others added 6 commits July 6, 2026 00:31
Run the tie-break judge twice with reviewer order swapped on consensus disagreement, accept only swap-stable resolutions, and fall back to the existing combineReviews path on instability or judge failure. Adds observability via gittensory_ai_review_tiebreak_order_unstable_total.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Add targeted tests for parse catch, judge usage, stable inconclusive,
synthesis skip path, and remaining branch coverage on the JSONbored#2997 tie-break.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@RealDiligent
RealDiligent requested a review from JSONbored as a code owner July 5, 2026 17:27
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 5, 2026
@loopover-orb

loopover-orb Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-05 17:38:15 UTC

2 files · 1 AI reviewer · no blockers · readiness 80/100 · CI pending · blocked

⏸️ Suggested Action - Manual Review

Review summary
This PR adds an order-swapped dual-AI tie-break judge for the `consensus` combiner: on reviewer disagreement it runs the judge twice with reviewer presentation order swapped, only accepts the resolution if both calls agree on the same physical reviewer (guarding against position bias), and falls back to the existing conservative `combineReviews` on any instability, judge failure, or unparseable output. The stability-matching logic in `dualAiTieBreakVerdictsOrderStable` and the reviewer-index remapping in `mapDualAiTieBreakVerdictToCombineResult` are internally consistent (verified index correspondence: normal-order `reviewer_0`/swapped-order `reviewer_1` both map to the same physical review object), and the fail-safe/fail-closed defaults (missing AI binding, thrown provider errors, unparseable JSON, mismatched swap results) all correctly reduce to the pre-existing conservative path. Test coverage is extensive, exercising both stable and unstable branches, the fallback-model path, and the full integration flow through `runGittensoryAiReview`.

Nits — 5 non-blocking
  • src/services/ai-review.ts (buildDualAiTieBreakJudgeUserPrompt/runDualAiTieBreakJudgeCall): the retry count `3` and `max_tokens: 512` are unexplained magic numbers — consider naming constants like the existing `RELIABLE_FALLBACK_MODELS`/`BEST_REVIEW_MODELS` pattern elsewhere in this file.
  • The `mapDualAiTieBreakVerdictToCombineResult` policy of fully dismissing a flagged reviewer's blocker when the judge favors the clean reviewer ("trust the pass even when the other reviewer flagged") is a meaningful gate-disposition change — worth a one-line call-out in the PR description since it changes the historical fail-closed-on-disagreement behavior for this specific case, even though the swap-stability gate mitigates false positives.
  • Reviewer disagreement now doubles AI spend for that PR (up to 2 judge calls × 3 attempts × 2 models = up to 12 extra calls) — not a blocker since it's bounded and intentional, but worth confirming this is accounted for in daily neuron budget expectations.
  • console.warn on tiebreak_order_unstable (ai-review.ts:1833) is fine as structured logging consistent with the file's existing pattern, but double-check it isn't spamming logs on noisy/flaky judge models in production.
  • Consider extracting the retry-count (3) and max_tokens (512) into named constants near TIE_BREAK_JUDGE_SYSTEM_PROMPT for discoverability.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #2997
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ❌ 5/25 Preflight is holding this PR: the review lane is unavailable, so it is not ready for automated review.
Contributor workload ✅ 10/10 Author activity: 181 registered-repo PR(s), 37 merged, 0 issue(s).
Contributor context ✅ Confirmed Gittensor contributor RealDiligent; Gittensor profile; 181 PR(s), 0 issue(s).
Gate result ✅ Passing No configured blocker found.
Review context
  • Author: RealDiligent
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository registration is not available in the local Gittensory cache.
  • Public profile languages: JavaScript, Ruby, Svelte, TypeScript, Cuda, Markdown
  • Official Gittensor activity: 181 PR(s), 0 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Await review-lane availability.
  • Refresh registry data or choose a registered active repo.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.11%. Comparing base (eb8cb4f) to head (907897d).
⚠️ Report is 25 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3569      +/-   ##
==========================================
+ Coverage   93.09%   93.11%   +0.02%     
==========================================
  Files         301      301              
  Lines       31450    31546      +96     
  Branches    11483    11526      +43     
==========================================
+ Hits        29279    29375      +96     
  Misses       1517     1517              
  Partials      654      654              
Files with missing lines Coverage Δ
src/services/ai-review.ts 96.90% <100.00%> (+0.65%) ⬆️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

RealDiligent and others added 2 commits July 6, 2026 01:35
Exercise judge provider_error, unparsed judge exhaustion, and consensus
title case-insensitivity to clear the remaining 0.05% patch gap.

Co-authored-by: Cursor <cursoragent@cursor.com>
Refactor consensus title comparison for clearer branches and add tests
when tie-break judges omit consensusTitle on consensus verdicts.

Co-authored-by: Cursor <cursoragent@cursor.com>
@JSONbored
JSONbored merged commit e1abb8b into JSONbored:main Jul 5, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Development

Successfully merging this pull request may close these issues.

fix(review): add order-swapped stability check to the dual-AI tie-break pass

2 participants