Skip to content

review(rag-index): honor GitHub's tree-truncation flag before pruning RAG chunks #10328

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

src/review/rag-index.ts's fetchRepoTree fetches a repo's full recursive git tree via GitHub's Git Trees
API (?recursive=1). GitHub's own API truncates the response (setting a truncated: true field in the
JSON body) for very large trees. fetchRepoTree's own doc comment claims this is handled:

/**
 * Fetch the FULL recursive git tree for a repo at `ref` and return only the blob (file) entries. Uses the
 * Git Trees API (`?recursive=1`) — one call yields the whole tree. Returns [] on any non-OK / error response
 * (fail-safe: a tree we can't read = nothing to index). `truncated` is honored (GitHub truncates very large
 * trees) — we index whatever it returned; the MAX_CHUNKS cap is the real bound anyway.
 */
async function fetchRepoTree(...): Promise<TreeEntry[] | null> {
  // ...
  const body = (await response.json()) as { tree?: Array<{ path?: string; type?: string; size?: number; sha?: string }> } | null;
  // ... builds `entries` from `body?.tree` only; nothing reads `body.truncated`
  return entries;
}

In reality, the response type doesn't even declare a truncated field, and nothing in the function ever
reads body.truncated. A truncated (partial) tree response is indistinguishable from a complete one by the
time it reaches the caller.

This matters because the caller unconditionally prunes based on the returned path set:

const rawTree = await fetchRepoTree(env, repoFullName, ref, token, admissionKey);
if (rawTree === null) return empty;
const tree = rawTree.filter(...).sort(...);
await pruneMissingPaths(infra, project, repoName, new Set(tree.map((entry) => entry.path)));

fetchRepoTree only returns null on a network/non-OK error — a 200 OK truncated response returns a
real (but partial) array, so the rawTree === null guard (which deliberately skips pruning "to avoid
deleting good chunks during a transient GitHub/API failure") never fires for this case.
pruneMissingPaths then deletes every previously-indexed RAG chunk whose path is NOT present in the
(incomplete) tree — silently and repeatedly deleting valid, still-existing files' indexed context on every
full reindex of a repository large enough for GitHub to truncate its tree response. No test exercises the
truncated: true case — test/unit/rag-index.test.ts's stubGithub test helper hardcodes
truncated: false.

Requirements

  • fetchRepoTree must read the response body's truncated field and surface it to its caller (e.g. by
    changing its return type to include a truncated: boolean flag alongside the entries, or by returning
    null — the existing "skip pruning" signal — when truncated is true).
  • The caller (the repo-indexing function that calls fetchRepoTree and then pruneMissingPaths) must skip
    pruneMissingPaths when the tree was truncated, mirroring the existing rawTree === null skip-pruning
    guard already in place for a total fetch failure — a truncated tree is exactly as untrustworthy for
    pruning decisions as a failed fetch, even though it's still useful for indexing whatever entries it did
    return.
  • Must not change indexing behavior — fetchRepoTree's entries (whatever GitHub did return) should still be
    indexed normally even when truncated; only the pruning decision changes.
  • Must not change behavior for the non-truncated (complete tree) case at all.

Deliverables

  • fetchRepoTree reads and surfaces GitHub's truncated field from the Git Trees API response.
  • The repo-indexing call site skips pruneMissingPaths when the tree came back truncated, verified by a
    new test: stub the GitHub tree response with truncated: true and a partial entry list, run the
    indexing function, and assert pruneMissingPaths (or its underlying deletion effect) was NOT invoked
    for paths missing from the partial response.
  • An existing-behavior regression test confirms a non-truncated tree response still triggers pruning of
    genuinely-removed paths exactly as before.

All three Deliverables are required in the same PR.

Test Coverage Requirements

This repo's Codecov patch gate requires 99%+ patch coverage on every changed line and branch under
src/**. src/review/rag-index.ts is inside src/**. The new test must set truncated: true in the
stubbed GitHub tree response (extending test/unit/rag-index.test.ts's stubGithub helper, which currently
hardcodes truncated: false) and assert on the real pruning behavior, not just on fetchRepoTree's return
value in isolation.

Expected Outcome

A repository whose git tree is large enough for GitHub to truncate the Trees API response no longer has its
valid, still-existing RAG-indexed files silently pruned away on every full reindex — pruning only happens
when the fetched tree is known to be complete.

Links & Resources

  • src/review/rag-index.tsfetchRepoTree (around lines 101-127) and its call site's pruneMissingPaths
    invocation (around lines 303-309, the already-correct rawTree === null skip-pruning guard to mirror).
  • test/unit/rag-index.test.ts — the stubGithub helper to extend with a truncated: true case.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions