Skip to content

feat(review): raise the RAG per-repo chunk cap from 1500 to 4000 - #4398

Merged
JSONbored merged 1 commit into
mainfrom
claude/raise-rag-chunk-cap
Jul 9, 2026
Merged

feat(review): raise the RAG per-repo chunk cap from 1500 to 4000#4398
JSONbored merged 1 commit into
mainfrom
claude/raise-rag-chunk-cap

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • MAX_CHUNKS_PER_REPO (the hard per-repo bound on stored RAG vectors) raised from 1500 to 4000.
  • All 3 currently-gated repos (gittensory, metagraphed, awesome-claude) were sitting exactly at the old cap, confirmed live — meaning real, indexable code was silently never making it into the RAG index for at least one of them.
  • Storage/search cost at this scale is trivial for Qdrant regardless of cap size — the bound exists to protect against a future pathological repo, not because a few thousand vectors is expensive. The real cost of a higher cap is embedding compute, which perf(rag): cache chunk embeddings by content hash, skip redundant re-embeds #4365 (the blob-SHA skip-cache) turns from a recurring per-cron-cycle cost into a one-time cost per file.

Scope

  • Stayed within wantedPaths (src/)
  • No secrets/wallets/trust-score/reward values anywhere
  • No changes to site/, CNAME, **/lovable/**, or CHANGELOG.md

Validation

  • npm run typecheck
  • npx vitest run test/unit/rag.test.ts test/unit/rag-index.test.ts — 127/127 passing (every cap-related test references the exported constant, none hardcode the old value)
  • npm run db:schema-drift:check / npm run selfhost:env-reference:check — clean (no schema/env surface touched)
  • npm audit --audit-level=moderate — 0 vulnerabilities

Safety

  • No secrets/tokens anywhere
  • Single-constant change, no logic touched

All 3 currently-gated repos were sitting at the old cap, and gittensory's own
indexable tree is within range of it before accounting for large files
splitting into multiple chunks -- real code was silently never indexed.
Storage/search cost at this scale is trivial for Qdrant regardless of cap
size; the higher embedding-compute cost is now one-time per file instead of
recurring every cron cycle (#4365's blob-SHA skip-cache).
@JSONbored JSONbored self-assigned this Jul 9, 2026
@JSONbored JSONbored added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 9, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored
JSONbored merged commit b429dee into main Jul 9, 2026
7 checks passed
@JSONbored
JSONbored deleted the claude/raise-rag-chunk-cap branch July 9, 2026 08:46
@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.96%. Comparing base (ca5576f) to head (d08ddc0).
⚠️ Report is 7 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4398   +/-   ##
=======================================
  Coverage   93.96%   93.96%           
=======================================
  Files         401      401           
  Lines       36869    36869           
  Branches    13480    13480           
=======================================
  Hits        34644    34644           
  Misses       1569     1569           
  Partials      656      656           
Files with missing lines Coverage Δ
src/review/rag.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Development

Successfully merging this pull request may close these issues.

1 participant