refactor(rag): extract the rag cluster into src/lib/rag/ (maturity X2)

[BigSimmo]

Summary

Phase 4 of the maturity backlog (docs/maturity-backlog-workorders.md → X2): the first real domain directory under src/lib, and the seam that unblocks directory-scoped boundary rules for the rest of the 197-file flat lib.

• Moved the 22-file rag cluster (rag.ts + 21 rag-*.ts) into src/lib/rag/ via git mv (renames preserved).
• Rewrote every importer: @/lib/rag* → @/lib/rag/rag* (51 sites) and the relative ../src/lib/rag* → ../src/lib/rag/rag* (worker + tests). The ../scripts/rag-offline-contract.mjs import was deliberately left untouched.
• Updated the maintainability-budget key, two config-path test fixtures (client-bundle boundary + worker-deploy trigger), the two readFileSync contract-test paths, docs/codebase-index.md, and rag path references across 13 maintained docs.

Pure moves + path rewrites — no logic change, so behaviour is identical.

Verification

Full local gate, all green (run as constituent commands rather than the verify:pr-local wrapper):

• npm run typecheck — pass.
• npm run test — 3012 pass; the only failure is the pre-existing container-only pdf-extraction-budget flake (a Python-subprocess deadline test with no rag dependency), which fails identically on clean origin/main — not a regression from this refactor.
• npm run lint — pass.
• npm run check:maintainability-budgets — pass (src/lib/rag/rag.ts 5143/5238).
• npm run docs:check-index — pass (new src/lib/rag/ indexed); npm run docs:check-links — pass (989 refs); npm run format:check — pass.
• UI verification not run: pure import-path rewrite, no rendered output changed (the only touched component file, display-text.ts, is a string helper — no JSX/markup).
• Retrieval/answer evals not run: no retrieval, ranking, selection, chunking, or answer-contract logic changed — the rag modules are byte-identical and only moved location, so the offline/live evals are unaffected. No dependency change.

Risk and rollout

• Risk: low — a large but purely mechanical diff (renames + import/path strings). Verified behaviour-preserving by the full type+test gate; git log --follow traces history through the renames.
• Rollback: revert the PR (single squash commit; no data or schema effect).
• Provider or production effects: None.

Clinical Governance Preflight

This is a byte-identical relocation of the rag modules (git mv + import-path rewrites); no clinical behaviour, data flow, or configuration changed. Each item is preserved by construction:

• Source-backed claims still require linked source verification before clinical use
• No patient-identifiable document workflow was introduced or expanded without explicit governance approval
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked when clinical decision-support behavior changed

Notes

Next natural step (separate PR) is X3 — decompose rag.ts behind the maintainability budget, now that its siblings already sit in src/lib/rag/. This PR intentionally does not touch logic.

---

Generated by Claude Code

Summary by CodeRabbit

• Documentation
  • Updated clinical safety hazard analysis, privacy/tenancy reviews, observability/SLO notes, and deployment architecture docs with clearer, up-to-date references.
  • Marked the completed retrieval-domain extraction work order as done.
• Refactor
  • Reorganized retrieval and answer-processing modules to align with the new internal layout (intended to keep endpoint behavior unchanged).
• Improvements
  • Enhanced clinical answer text sanitization and monitoring, including improved capture of hybrid retrieval failures and source-only degradation reasons.
• Tests
  • Updated test wiring and coverage to match the reorganized module structure.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cc545c08-57a4-48e9-8dd2-4fab740fded1

📥 Commits

Reviewing files that changed from the base of the PR and between 6171b2d and 19ef1bf.

📒 Files selected for processing (2)

• docs/maturity-backlog-workorders.md
• scripts/eval-retrieval.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• scripts/eval-retrieval.ts

---

📝 Walkthrough

Walkthrough

This PR moves RAG modules under src/lib/rag/, adds supporting RAG utilities, rewires application, script, worker, and test imports, updates maintainability and deployment references, and refreshes documentation citations and hazard-analysis content.

Changes

RAG domain-directory extraction

Layer / File(s)	Summary
RAG modules and entrypoint wiring src/lib/rag/*	Adds answer-text sanitization, comparison, context selection, contracts, document-summary loading, evaluation, diagnostics, provider handling, query guards, cache matching, and versioning modules; the rag.ts entrypoint and related helpers use the nested module paths.
Application and script call sites src/app/api/**, worker/main.ts, src/lib/*, scripts/*	Updates route, worker, shared-library, evaluation, cache-warming, and maintainability references to the reorganized RAG modules without changing surrounding behavior.
Tests and documentation tests/*, docs/*	Updates test imports and mocks, corrects RAG path citations, records the completed extraction, and expands clinical hazard and injection-threat descriptions.

Estimated code review effort: 3 (Moderate) | ~30 minutes

Possibly related PRs

• BigSimmo/Database#457: Related RAG hardening and module refactoring across shared RAG components.
• BigSimmo/Database#985: Also updates the repository maturity work-order tracker.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.94% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: moving the RAG cluster into src/lib/rag/ as part of X2.
Description check	✅ Passed	The description matches the template sections and includes summary, verification, risk, governance preflight, and notes with concrete details.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/maturity-phase-4-rag-extraction

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

src/lib/rag/rag-query-guard.ts (1)

19-34: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate decision tree between the two exported guards.

shouldShortCircuitUnsupportedSearch and isUnsupportedSoftTailAnalysis run the identical 4-condition chain, only inverting the first two branches' return value. Any change to the pattern guards must be mirrored in both functions with reversed polarity, which is easy to get wrong.

♻️ Suggested extraction of the shared classification

+type UnsupportedSearchClassification =
+  | "pattern_guard"
+  | "soft_tail_ineligible"
+  | "soft_tail_consumer"
+  | "soft_tail_confidence_gate_pass"
+  | "soft_tail_confidence_gate_fail";
+
+function classifyUnsupportedSearch(query: string, analysis: ClinicalQueryAnalysis): UnsupportedSearchClassification {
+  if (unavailableDocumentNoisePattern.test(query)) return "pattern_guard";
+  if (clearlyOutsideCorpusMedicalPattern.test(query) && analysis.documentTitleTerms.length === 0) return "pattern_guard";
+  if (!unsupportedSoftTailEligible(analysis)) return "soft_tail_ineligible";
+  if (clearlyNonClinicalConsumerPattern.test(query)) return "soft_tail_consumer";
+  return analysis.confidence <= 0.42 && analysis.expandedTerms.length <= 5
+    ? "soft_tail_confidence_gate_pass"
+    : "soft_tail_confidence_gate_fail";
+}
+
 export function shouldShortCircuitUnsupportedSearch(query: string, analysis: ClinicalQueryAnalysis) {
-  if (unavailableDocumentNoisePattern.test(query)) return true;
-  if (clearlyOutsideCorpusMedicalPattern.test(query) && analysis.documentTitleTerms.length === 0) return true;
-  if (!unsupportedSoftTailEligible(analysis)) return false;
-  if (clearlyNonClinicalConsumerPattern.test(query)) return true;
-  return analysis.confidence <= 0.42 && analysis.expandedTerms.length <= 5;
+  const classification = classifyUnsupportedSearch(query, analysis);
+  return classification === "pattern_guard" || classification === "soft_tail_consumer" || classification === "soft_tail_confidence_gate_pass";
 }

 // True only for queries that would short-circuit via the soft tail itself, not a pattern guard.
 export function isUnsupportedSoftTailAnalysis(query: string, analysis: ClinicalQueryAnalysis) {
-  if (unavailableDocumentNoisePattern.test(query)) return false;
-  if (clearlyOutsideCorpusMedicalPattern.test(query) && analysis.documentTitleTerms.length === 0) return false;
-  if (!unsupportedSoftTailEligible(analysis)) return false;
-  if (clearlyNonClinicalConsumerPattern.test(query)) return false;
-  return analysis.confidence <= 0.42 && analysis.expandedTerms.length <= 5;
+  return classifyUnsupportedSearch(query, analysis) === "soft_tail_confidence_gate_pass";
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/rag/rag-query-guard.ts` around lines 19 - 34, Extract the shared
classification logic from shouldShortCircuitUnsupportedSearch and
isUnsupportedSoftTailAnalysis into a single private helper that distinguishes
pattern-guard matches from soft-tail eligibility. Update both exported functions
to reuse that helper while preserving their current behavior: the first should
short-circuit on any guard or soft-tail match, and the second should return true
only for the soft-tail match.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/clinical-hazard-analysis.md`:
- Around line 47-55: Update the applyNumericVerification references in the
hazard analysis to link to its actual definition in src/lib/answer-verification
and its invocation in src/lib/rag/rag.ts around the cited call-site range,
replacing the incorrect rag.ts definition and outcome line links. Keep the
surrounding behavior description unchanged.

In `@docs/rag-injection-threat-model.md`:
- Around line 15-38: Update the documentation citations for buildRagSourceBlock
and its related prompt-field locations to reference the extracted
rag-source-block.ts module, including the sections at 70–86, 99–101, 112–125,
and 135–148. Retain rag.ts citations only for answerInstructions and
buildAnswerInput/answer-input assembly, and ensure all referenced line ranges
match the extracted implementation.

---

Nitpick comments:
In `@src/lib/rag/rag-query-guard.ts`:
- Around line 19-34: Extract the shared classification logic from
shouldShortCircuitUnsupportedSearch and isUnsupportedSoftTailAnalysis into a
single private helper that distinguishes pattern-guard matches from soft-tail
eligibility. Update both exported functions to reuse that helper while
preserving their current behavior: the first should short-circuit on any guard
or soft-tail match, and the second should return true only for the soft-tail
match.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: f344ebf5-2be1-48d1-81e6-9faf55e28cda

📥 Commits

Reviewing files that changed from the base of the PR and between 6d20560 and 6171b2d.

📒 Files selected for processing (114)

• docs/clinical-hazard-analysis.md
• docs/codebase-index.md
• docs/deployment-architecture.md
• docs/maturity-backlog-workorders.md
• docs/observability-slos.md
• docs/openai-cross-border-basis.md
• docs/privacy-impact-assessment.md
• docs/process-hardening.md
• docs/rag-hybrid-findings-and-todo.md
• docs/rag-injection-threat-model.md
• docs/redesign/06-verification.md
• docs/search-rag-master-context.md
• docs/search-rag-master-plan.md
• docs/tenancy-defense-in-depth-review.md
• scripts/check-maintainability-budgets.mjs
• scripts/eval-answer-quality.ts
• scripts/eval-quality.ts
• scripts/eval-rag.ts
• scripts/eval-retrieval.ts
• scripts/eval-search-api.ts
• scripts/eval-search.ts
• scripts/eval-utils.ts
• scripts/warm-retrieval-cache.ts
• src/app/api/answer/route.ts
• src/app/api/answer/stream/route.ts
• src/app/api/documents/[id]/labels/route.ts
• src/app/api/documents/[id]/reviews/route.ts
• src/app/api/documents/[id]/route.ts
• src/app/api/documents/[id]/summarize/route.ts
• src/app/api/documents/[id]/table-facts/route.ts
• src/app/api/documents/bulk/reindex/route.ts
• src/app/api/documents/bulk/route.ts
• src/app/api/search/route.ts
• src/components/clinical-dashboard/display-text.ts
• src/lib/answer-verification.ts
• src/lib/clinical-safety.ts
• src/lib/cross-document-synthesis.ts
• src/lib/rag/rag-answer-support.ts
• src/lib/rag/rag-answer-text.ts
• src/lib/rag/rag-cache-utils.ts
• src/lib/rag/rag-cache.ts
• src/lib/rag/rag-candidate-sources.ts
• src/lib/rag/rag-claim-support.ts
• src/lib/rag/rag-comparison.ts
• src/lib/rag/rag-context-selection.ts
• src/lib/rag/rag-contracts.ts
• src/lib/rag/rag-document-summary-context.ts
• src/lib/rag/rag-eval-cases.ts
• src/lib/rag/rag-eval-diagnostics.ts
• src/lib/rag/rag-extractive-answer.ts
• src/lib/rag/rag-provider.ts
• src/lib/rag/rag-query-guard.ts
• src/lib/rag/rag-quote-verification.ts
• src/lib/rag/rag-retrieval-variants.ts
• src/lib/rag/rag-route-budget.ts
• src/lib/rag/rag-routing.ts
• src/lib/rag/rag-source-block.ts
• src/lib/rag/rag-versioning.ts
• src/lib/rag/rag.ts
• src/lib/semantic-rerank.ts
• src/lib/universal-search.ts
• tests/anonymous-answer-cache-policy.test.ts
• tests/answer-prose-runons.test.ts
• tests/answer-ranking.test.ts
• tests/answer-responsiveness-gate.test.ts
• tests/api-validation-contract.test.ts
• tests/architecture-boundaries.test.ts
• tests/corpus-grounding.test.ts
• tests/cross-tenant-staging-config.test.ts
• tests/document-admin-rate-limit.test.ts
• tests/document-mutation-routes.test.ts
• tests/eval-utils.test.ts
• tests/extractive-answer-formatting.test.ts
• tests/private-access-routes.test.ts
• tests/private-rag-access.test.ts
• tests/public-access-deep.test.ts
• tests/rag-abort-signal.test.ts
• tests/rag-answer-fallback.test.ts
• tests/rag-answer-support.test.ts
• tests/rag-answer-text.test.ts
• tests/rag-cache-invalidation.test.ts
• tests/rag-cache-utils.test.ts
• tests/rag-chunk-load-cache.test.ts
• tests/rag-claim-support.test.ts
• tests/rag-classifier-memo.test.ts
• tests/rag-comparison.test.ts
• tests/rag-content-accuracy.test.ts
• tests/rag-context-budget.test.ts
• tests/rag-document-summary.test.ts
• tests/rag-eval-cases.test.ts
• tests/rag-eval-source-governance.test.ts
• tests/rag-fast-path-ordering.test.ts
• tests/rag-generation-fingerprint.test.ts
• tests/rag-injection.test.ts
• tests/rag-offline-answer.test.ts
• tests/rag-provider.test.ts
• tests/rag-query-concurrency.test.ts
• tests/rag-route-budget.test.ts
• tests/rag-routing.test.ts
• tests/rag-score.test.ts
• tests/rag-second-stage-ranking.test.ts
• tests/rag-shared-cache.test.ts
• tests/rag-tail-latency.test.ts
• tests/rag-trust.test.ts
• tests/rag-variant-early-exit.test.ts
• tests/railway-config.test.ts
• tests/retrieval-access-scope.test.ts
• tests/retrieval-hydration-scope.test.ts
• tests/retrieval-query-variants.test.ts
• tests/semantic-rerank.test.ts
• tests/source-backed-recovery-cross-reference.test.ts
• tests/source-review-route.test.ts
• tests/universal-search.test.ts
• worker/main.ts