Skip to content

fix(queue): reconcileLiveDuplicateSiblings fans out unbounded GitHub fetches per duplicate-cluster sibling #5835

Description

@JSONbored

Context

reconcileLiveDuplicateSiblings in src/queue/duplicate-detection.ts (lines 69-113) live-reconciles a PR's duplicate-cluster siblings before the winner is elected. When a PR has overlapping open siblings sharing a linked issue, it fires one live GitHub fetch per sibling via:

await Promise.all(
  overlapping.map(async (sibling) => {
    const liveState = await fetchLivePullRequestState(env, repoFullName, sibling.number, token, admissionKey).catch(() => undefined);
    ...
  }),
);

(lines 94-110). This has no concurrency cap — every open sibling PR sharing the linked issue gets a simultaneous live GitHub REST call in one pass.

This is the same class of "many live per-item GitHub reads" work that src/queue/processors.ts (the file duplicate-detection.ts was extracted from, per its own header comment) explicitly bounds elsewhere:

  • CONTRIBUTOR_CAP_LIVE_CHECK_CONCURRENCY = 10 (processors.ts:4875), used via mapWithConcurrency at processors.ts:2790 and processors.ts:5065.
  • GLOBAL_OPEN_ITEM_LIVE_CHECK_CONCURRENCY = 10 (processors.ts:4853).

mapWithConcurrency itself is defined at processors.ts:4877 and exported for reuse. A popular issue with dozens of duplicate PRs attached would burst that many concurrent REST calls in a single reconciliation pass today, with no bound — risking rate-limit pressure exactly where the module's own doc comment already documents comparable per-repo/per-org bursts.

Requirements

  • reconcileLiveDuplicateSiblings's overlapping.map(...) fan-out must run at a bounded concurrency, not unbounded Promise.all.
  • Reuse the existing mapWithConcurrency helper from src/queue/processors.ts (import it, matching the existing precedent already used by CONTRIBUTOR_CAP_LIVE_CHECK_CONCURRENCY call sites) rather than hand-rolling a new limiter.
  • Pick (or add) a concurrency constant of the same order of magnitude as the existing CONTRIBUTOR_CAP_LIVE_CHECK_CONCURRENCY/GLOBAL_OPEN_ITEM_LIVE_CHECK_CONCURRENCY constants (e.g. reuse one of them, or add a small new named constant local to duplicate-detection.ts if the existing ones aren't a semantic fit) — do not introduce an unbounded default.
  • The existing fail-open behavior (an unresolved/errored fetchLivePullRequestState call keeps the sibling as "still open", per the function's own doc comment) must be preserved exactly — this is a pure concurrency-bounding change, not a behavior change to what counts as stale-closed.
  • No change to reconcileLiveDuplicateSiblings's public signature or the LOOPOVER_DUPLICATE_WINNER flag gating.

Deliverables

  • reconcileLiveDuplicateSiblings fans out over overlapping via mapWithConcurrency (or an equivalent bounded-concurrency helper) instead of unbounded Promise.all.
  • A regression test in test/unit/ (wherever duplicate-detection.ts's existing tests live) asserting that, given a duplicate cluster with more overlapping siblings than the concurrency cap, no more than the cap number of live fetches are in flight simultaneously (e.g. via a fetch stub that tracks concurrent in-flight calls).
  • Existing tests for stale-sibling filtering (dropping a sibling that live-resolves to non-open) continue to pass unmodified in behavior.

Test Coverage Requirements

Aim for 99%+ Codecov patch coverage on the touched lines in src/queue/duplicate-detection.ts (in coverage.includesrc/** is measured). Cover: (1) fewer siblings than the concurrency cap (unchanged behavior), (2) more siblings than the cap (concurrency actually bounded), (3) the existing fail-open path where a fetch errors/resolves undefined. A regression test asserting the concurrency bound itself is required, not just line coverage of the new import/call.

Expected Outcome

A PR/issue with many open duplicate-PR siblings sharing a linked issue no longer bursts one live GitHub REST call per sibling in a single reconciliation pass — the fan-out is capped at the same order of magnitude as the codebase's other per-item live-check fan-outs, closing the one remaining unbounded-concurrency gap in this module family.

Links & Resources

  • src/queue/duplicate-detection.ts (reconcileLiveDuplicateSiblings, lines 69-113)
  • src/queue/processors.ts (mapWithConcurrency, line 4877; CONTRIBUTOR_CAP_LIVE_CHECK_CONCURRENCY, line 4875; GLOBAL_OPEN_ITEM_LIVE_CHECK_CONCURRENCY, line 4853; example call sites at lines 2790 and 5065)
  • Extracted from processors.ts per #4013 (the file's own module-split sequence, documented in duplicate-detection.ts's header comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions