Skip to content

fix(queue): coalesce scheduled RAG full re-index over pending webhook incrementals #1949

Description

@JSONbored

Parent: #1936

Problem

The scheduled six-hourly RAG full re-index (fanOutRagIndexJobs → per-repo rag-index-repo, src/queue/processors.ts:1068-1107) can run for a repo that already has a pending webhook-driven incremental re-index of changed paths for the same repo (runRagIndexJob with paths, :1124-1146). The two are not coalesced against each other (src/selfhost/queue-common.ts:652-657 keys them separately), so the same repo is re-embedded twice — wasting the free-tier vector budget and piling scheduled work on top of live webhook work.

Requirements

  • A scheduled full re-index for a repo should not run while a webhook incremental re-index for that same repo is pending/processing (and vice-versa the full supersedes queued incrementals).
  • Preserve correctness: a full re-index must still eventually run; an incremental must not be silently dropped if no full is pending.

Deliverables

  • Coalesce-scope the two rag-index-repo variants by repo so overlapping full/incremental work collapses.
  • Tests for full-over-incremental supersession and incremental-preserved-when-no-full.

Acceptance criteria

  • No double re-embedding of the same repo in one window; vector-budget spend drops on active repos.
  • RAG retrieval freshness is unchanged.

Expected outcome

Scheduled RAG indexing stops duplicating webhook-driven indexing, cutting redundant embedding cost. (Net-new from the #1936 rate-limit/resource audit, rank 14.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

    Projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions