⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.
Context
src/review/rag-index.ts's fetchRepoTree fetches a repo's full recursive git tree via GitHub's Git Trees
API (?recursive=1). GitHub's own API truncates the response (setting a truncated: true field in the
JSON body) for very large trees. fetchRepoTree's own doc comment claims this is handled:
/**
* Fetch the FULL recursive git tree for a repo at `ref` and return only the blob (file) entries. Uses the
* Git Trees API (`?recursive=1`) — one call yields the whole tree. Returns [] on any non-OK / error response
* (fail-safe: a tree we can't read = nothing to index). `truncated` is honored (GitHub truncates very large
* trees) — we index whatever it returned; the MAX_CHUNKS cap is the real bound anyway.
*/
async function fetchRepoTree(...): Promise<TreeEntry[] | null> {
// ...
const body = (await response.json()) as { tree?: Array<{ path?: string; type?: string; size?: number; sha?: string }> } | null;
// ... builds `entries` from `body?.tree` only; nothing reads `body.truncated`
return entries;
}
In reality, the response type doesn't even declare a truncated field, and nothing in the function ever
reads body.truncated. A truncated (partial) tree response is indistinguishable from a complete one by the
time it reaches the caller.
This matters because the caller unconditionally prunes based on the returned path set:
const rawTree = await fetchRepoTree(env, repoFullName, ref, token, admissionKey);
if (rawTree === null) return empty;
const tree = rawTree.filter(...).sort(...);
await pruneMissingPaths(infra, project, repoName, new Set(tree.map((entry) => entry.path)));
fetchRepoTree only returns null on a network/non-OK error — a 200 OK truncated response returns a
real (but partial) array, so the rawTree === null guard (which deliberately skips pruning "to avoid
deleting good chunks during a transient GitHub/API failure") never fires for this case.
pruneMissingPaths then deletes every previously-indexed RAG chunk whose path is NOT present in the
(incomplete) tree — silently and repeatedly deleting valid, still-existing files' indexed context on every
full reindex of a repository large enough for GitHub to truncate its tree response. No test exercises the
truncated: true case — test/unit/rag-index.test.ts's stubGithub test helper hardcodes
truncated: false.
Requirements
fetchRepoTree must read the response body's truncated field and surface it to its caller (e.g. by
changing its return type to include a truncated: boolean flag alongside the entries, or by returning
null — the existing "skip pruning" signal — when truncated is true).
- The caller (the repo-indexing function that calls
fetchRepoTree and then pruneMissingPaths) must skip
pruneMissingPaths when the tree was truncated, mirroring the existing rawTree === null skip-pruning
guard already in place for a total fetch failure — a truncated tree is exactly as untrustworthy for
pruning decisions as a failed fetch, even though it's still useful for indexing whatever entries it did
return.
- Must not change indexing behavior —
fetchRepoTree's entries (whatever GitHub did return) should still be
indexed normally even when truncated; only the pruning decision changes.
- Must not change behavior for the non-truncated (complete tree) case at all.
Deliverables
All three Deliverables are required in the same PR.
Test Coverage Requirements
This repo's Codecov patch gate requires 99%+ patch coverage on every changed line and branch under
src/**. src/review/rag-index.ts is inside src/**. The new test must set truncated: true in the
stubbed GitHub tree response (extending test/unit/rag-index.test.ts's stubGithub helper, which currently
hardcodes truncated: false) and assert on the real pruning behavior, not just on fetchRepoTree's return
value in isolation.
Expected Outcome
A repository whose git tree is large enough for GitHub to truncate the Trees API response no longer has its
valid, still-existing RAG-indexed files silently pruned away on every full reindex — pruning only happens
when the fetched tree is known to be complete.
Links & Resources
src/review/rag-index.ts — fetchRepoTree (around lines 101-127) and its call site's pruneMissingPaths
invocation (around lines 303-309, the already-correct rawTree === null skip-pruning guard to mirror).
test/unit/rag-index.test.ts — the stubGithub helper to extend with a truncated: true case.
Context
src/review/rag-index.ts'sfetchRepoTreefetches a repo's full recursive git tree via GitHub's Git TreesAPI (
?recursive=1). GitHub's own API truncates the response (setting atruncated: truefield in theJSON body) for very large trees.
fetchRepoTree's own doc comment claims this is handled:In reality, the response type doesn't even declare a
truncatedfield, and nothing in the function everreads
body.truncated. A truncated (partial) tree response is indistinguishable from a complete one by thetime it reaches the caller.
This matters because the caller unconditionally prunes based on the returned path set:
fetchRepoTreeonly returnsnullon a network/non-OK error — a200 OKtruncated response returns areal (but partial) array, so the
rawTree === nullguard (which deliberately skips pruning "to avoiddeleting good chunks during a transient GitHub/API failure") never fires for this case.
pruneMissingPathsthen deletes every previously-indexed RAG chunk whose path is NOT present in the(incomplete) tree — silently and repeatedly deleting valid, still-existing files' indexed context on every
full reindex of a repository large enough for GitHub to truncate its tree response. No test exercises the
truncated: truecase —test/unit/rag-index.test.ts'sstubGithubtest helper hardcodestruncated: false.Requirements
fetchRepoTreemust read the response body'struncatedfield and surface it to its caller (e.g. bychanging its return type to include a
truncated: booleanflag alongside the entries, or by returningnull— the existing "skip pruning" signal — whentruncatedistrue).fetchRepoTreeand thenpruneMissingPaths) must skippruneMissingPathswhen the tree was truncated, mirroring the existingrawTree === nullskip-pruningguard already in place for a total fetch failure — a truncated tree is exactly as untrustworthy for
pruning decisions as a failed fetch, even though it's still useful for indexing whatever entries it did
return.
fetchRepoTree's entries (whatever GitHub did return) should still beindexed normally even when truncated; only the pruning decision changes.
Deliverables
fetchRepoTreereads and surfaces GitHub'struncatedfield from the Git Trees API response.pruneMissingPathswhen the tree came back truncated, verified by anew test: stub the GitHub tree response with
truncated: trueand a partial entry list, run theindexing function, and assert
pruneMissingPaths(or its underlying deletion effect) was NOT invokedfor paths missing from the partial response.
genuinely-removed paths exactly as before.
All three Deliverables are required in the same PR.
Test Coverage Requirements
This repo's Codecov patch gate requires 99%+ patch coverage on every changed line and branch under
src/**.src/review/rag-index.tsis insidesrc/**. The new test must settruncated: truein thestubbed GitHub tree response (extending
test/unit/rag-index.test.ts'sstubGithubhelper, which currentlyhardcodes
truncated: false) and assert on the real pruning behavior, not just onfetchRepoTree's returnvalue in isolation.
Expected Outcome
A repository whose git tree is large enough for GitHub to truncate the Trees API response no longer has its
valid, still-existing RAG-indexed files silently pruned away on every full reindex — pruning only happens
when the fetched tree is known to be complete.
Links & Resources
src/review/rag-index.ts—fetchRepoTree(around lines 101-127) and its call site'spruneMissingPathsinvocation (around lines 303-309, the already-correct
rawTree === nullskip-pruning guard to mirror).test/unit/rag-index.test.ts— thestubGithubhelper to extend with atruncated: truecase.