Skip to content

fix(rag): align retrieval quality knobs with reviewbot — minScore 0.4 + bm25 rerank + title-led query (#GAP-2) - #1124

Merged
JSONbored merged 1 commit into
mainfrom
feat/counter-rag
Jun 23, 2026
Merged

fix(rag): align retrieval quality knobs with reviewbot — minScore 0.4 + bm25 rerank + title-led query (#GAP-2)#1124
JSONbored merged 1 commit into
mainfrom
feat/counter-rag

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Aligns the gittensory codebase-RAG retrieval seam (src/review/rag-wire.ts) with reviewbot's core config, fixing GAP-2 (the code half).

Before, the retrieve call used minScore: 0 (off) and no reranker, so low-relevance cosine "neighbours" were injected as "RELEVANT EXISTING CODE" — noise that itself drives false positives. The query also omitted the PR title, hurting recall vs. reviewbot (which embeds ${title}\n${diff}).

Changes

  • src/review/rag-wire.ts
    • buildRagQuery(files, title?) — PREPENDS the (trimmed) PR title to the query text; additive, omitted cleanly when blank/absent.
    • buildReviewRagContext(...) — always passes minScore: 0.4 + reranker: "bm25" to retrieveContext (matches reviewbot rag: { minScore: 0.4, reranker: "bm25" }). A caller-supplied reranker still wins (e.g. forcing "off"); minScore is fixed.
  • src/queue/processors.ts — feeds the PR title into the retrieval-context build at the (already flag+allowlist-gated) review call site.
  • test/unit/rag-wiring.test.ts — new assertions: title leads the query; retrieveContext invoked with minScore: 0.4 / reranker: "bm25"; caller override honored; blank-title handling.

Scope / safety

Code knobs ONLY. RAG stays per-repo opt-in (isRagEnabled(env) flag AND the convergence repo allowlist) — this does not force-enable RAG globally; the operator enables + warms the index via runtime config separately. Fully fail-safe path unchanged.

Note: Task A (repoint public-stats to the live audit_events ledger) was already shipped on main in #1114, so this branch contains only the RAG (Task B) work.

Verification

  • npx tsc --noEmit — clean (exactOptionalPropertyTypes + noUncheckedIndexedAccess)
  • npx vitest run3546 passed, 1 skipped (215 files); rag-wiring suite 16 → 21 tests

… + bm25 rerank + title-led query (#GAP-2)

The RAG retrieval seam used minScore 0 (off) and no reranker, keeping
low-relevance cosine 'neighbours' as 'relevant code' — itself a source
of false positives. Match reviewbot's core config: pass minScore 0.4 +
reranker 'bm25' on every retrieve call, and prepend the PR title to the
query text for recall parity (reviewbot queries `${title}\n${diff}`).

- rag-wire: buildRagQuery prepends the (trimmed) PR title; buildReviewRagContext
  passes minScore 0.4 + reranker bm25 (caller-supplied reranker still wins).
- processors: feed the PR title into the retrieval-context build.
- tests: assert the title leads the query + retrieveContext is called with
  minScore 0.4 / reranker bm25 (caller override honored).

Code knobs only — RAG stays per-repo opt-in (flag + allowlist); the operator
enables + warms the index via runtime config separately.
@dosubot dosubot Bot added the size:M label Jun 23, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.84%. Comparing base (6be41be) to head (292cbe4).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1124   +/-   ##
=======================================
  Coverage   94.83%   94.84%           
=======================================
  Files         155      155           
  Lines       18708    18711    +3     
  Branches     6772     6773    +1     
=======================================
+ Hits        17742    17746    +4     
  Misses        395      395           
+ Partials      571      570    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 46f3812 into main Jun 23, 2026
17 checks passed
@JSONbored
JSONbored deleted the feat/counter-rag branch June 23, 2026 23:41
JSONbored added a commit that referenced this pull request Jun 24, 2026
…_RAG=true) (#1126)

Flips GITTENSORY_REVIEW_RAG false->true so the reviewer retrieves related-code context (with the #1124 quality
knobs: minScore 0.4 + bm25 rerank + title-led query). Fail-safe: retrieval degrades to '' until the Vectorize
index warms (this flag also activates ingestion: the 6-hourly cron fan-out + the merged-PR incremental path).
Still gated by the GITTENSORY_REVIEW_REPOS allowlist. (#GAP-2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant