Skip to content

queue(signal-snapshot): time-bound the queue-health history read so the 30-day queue-trend window can resolve a baseline #10020

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

generateSignalSnapshotForRepo loads two histories to build the per-repo queue-trend report. One is
time-bounded, the other is not. src/queue/signal-snapshot.ts:99-123:

  const trendSince = new Date(
    Date.now() - QUEUE_TREND_HISTORY_DAYS * 24 * 60 * 60 * 1000,
  ).toISOString();
  const [ ... totalsHistory, queueHealthHistory ] = await Promise.all([
    ...
    listRepoGithubTotalsSnapshotHistory(env, repo.fullName, {
      sinceIso: trendSince,
      limit: 120,
    }),
    listSignalSnapshots(env, "queue-health", repo.fullName),
  ]);

listSignalSnapshots takes no window and no limit parameter — it hard-caps at the 100 newest rows,
ordered generated_at DESC (src/db/repositories.ts:6075-6095, .orderBy(desc(...)).limit(100)).

queue-health snapshots are written once per repo per generate-signal-snapshots run. That job is enqueued
by the six-hourly full-sync window (src/index.ts:356-357, isFullSyncWindow) — four rows per repo per day —
and additionally by repairDataFidelity's per-repo fan-out (src/queue/processors.ts:6022-6027) and by the
manual /v1/internal/jobs/generate-signal-snapshots route. queue-health is also deliberately excluded
from the dedup prune (src/db/retention.ts:465-467: "queue-health stays EXCLUDED (feeds
buildQueueTrendReport as a real series)"), so the rows accumulate and the 100-row cap is the binding limit.

At four rows/day the newest 100 rows cover ~25 days. buildQueueTrendReport builds 7/14/30-day windows
(src/services/queue-trends.ts:5), and baselineQueuePoint (src/services/queue-trends.ts:154-158) needs a
point at or before latest - windowDays:

  const targetMs = latestMs - windowDays * 24 * 60 * 60 * 1000;
  return [...points].reverse().find((point) => Date.parse(point.generatedAt) <= targetMs) ?? null;

With no point older than ~25 days the 30-day window's baselineQueue is permanently null, so
duplicateTrend (src/services/queue-trends.ts:99) and stalePullRequestRateDelta
(src/services/queue-trends.ts:114) are permanently null for that window, and any repo whose snapshots are
written more often than ~3.3/day (any repo touched by repairDataFidelity or a manual run) loses the 14-day
window too. QUEUE_TREND_HISTORY_DAYS is 35 (src/services/queue-trends.ts:6) — the code intends 35 days of
history and gets ~25.

This is the same defect #9699 already fixed for the sibling reader: src/api/routes.ts:1399-1408 computes an
explicit slopTrendSinceIso and calls listRecentSignalSnapshotsForTargets(..., limit, sinceIso) precisely
because "the row-count cap ... otherwise kept only the most recent few days". The queue-trend reader was left
on the un-bounded listSignalSnapshots.

Requirements

  • src/queue/signal-snapshot.ts must read the queue-health history through
    listRecentSignalSnapshotsForTargets (src/db/repositories.ts:6150-6156) with an explicit sinceIso
    derived from QUEUE_TREND_HISTORY_DAYS, and an explicit per-target row limit, instead of
    listSignalSnapshots.
  • The sinceIso passed must be the SAME trendSince value already computed at
    src/queue/signal-snapshot.ts:99-101 — one window constant for both halves of the trend input, not a second
    literal.
  • The per-target limit must be a named exported constant in src/services/queue-trends.ts sized for at least
    four rows per day across QUEUE_TREND_HISTORY_DAYS (i.e. QUEUE_TREND_HISTORY_DAYS * 4 or larger), so the
    time bound is the primary constraint and the row cap is a backstop. It must be exported so a test can assert
    the relationship rather than re-deriving the number.
  • listRecentSignalSnapshotsForTargets must be called with a single-element target array containing
    repo.fullName, and the result read out with the same exact-casing key convention that helper documents
    (src/db/repositories.ts:6149), falling back to [] when the key is absent.
  • listSignalSnapshots's own signature and its hard limit(100) must NOT change — other callers depend on
    the latest-row-only behaviour.
  • The totals half of the read (listRepoGithubTotalsSnapshotHistory with sinceIso: trendSince, limit: 120)
    must NOT change.
  • buildQueueTrendReport and every function in src/services/queue-trends.ts other than the new exported
    constant must NOT change.

⚠️ Required pattern: mirror src/api/routes.ts:1399-1408 exactly — compute the window ISO from the
feature's own weeks/days constant, pass it as sinceIso alongside a generous row cap, and let the time
bound do the work. What does NOT satisfy this issue: (a) raising listSignalSnapshots' hard limit(100),
which changes every other caller of that function; (b) adding a second, parallel "queue-health history"
query helper in src/db/repositories.ts when listRecentSignalSnapshotsForTargets already selects
payload_json and applies the time bound inside its window function; (c) shrinking
QUEUE_TREND_WINDOWS_DAYS to drop the 30-day window instead of supplying the history it needs.

Deliverables

  • src/queue/signal-snapshot.ts calls listRecentSignalSnapshotsForTargets(env, "queue-health", [repo.fullName], QUEUE_TREND_SNAPSHOT_LIMIT, trendSince) and passes the resulting array as
    queueHealthSnapshots to buildQueueTrendReport.
  • src/services/queue-trends.ts exports QUEUE_TREND_SNAPSHOT_LIMIT, at least
    QUEUE_TREND_HISTORY_DAYS * 4.
  • A test in test/unit/queue-trends.test.ts asserting
    QUEUE_TREND_SNAPSHOT_LIMIT >= QUEUE_TREND_HISTORY_DAYS * 4 and
    QUEUE_TREND_HISTORY_DAYS >= 30, so a future shrink of either constant fails the build.
  • A regression test named for this bug — new file test/unit/signal-snapshot-queue-trend-history.test.ts
    (create it) — that seeds 130 queue-health snapshots for one repo spread across 33 days (four per day)
    plus GitHub-totals snapshots across the same span, runs generateSignalSnapshots(env, repoFullName),
    reads back the queue-trend snapshot persisted by upsertRepoQueueTrendSnapshot
    (src/db/repositories.ts:1699), and asserts the 30-day window has
    status: "ready" with a non-null duplicateTrend and a non-null stalePullRequestRateDelta. Against
    today's code the 30-day window's two delta fields are null.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
adding the exported constant and its arithmetic test without changing the read in
src/queue/signal-snapshot.ts, so the persisted trend is unchanged — does not resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. vitest.config.ts's coverage.include
covers src/**/*.ts, so both src/queue/signal-snapshot.ts and src/services/queue-trends.ts are measured
and gated. The change introduces one nullish branch — the ?? [] fallback when the returned map has no entry
for repo.fullName — and both arms need a test: a repo with snapshots (map hit) and a repo with none (map
miss, empty history, trend windows report status: "unavailable").

Expected Outcome

After this ships, a repo with ≥30 days of queue-health snapshots gets a 30-day queue-trend window that
actually resolves a baseline, so duplicateTrend and stalePullRequestRateDelta carry real numbers instead
of null, and the "duplicate cluster count increased" warning (src/services/queue-trends.ts:125) becomes
reachable on the widest window. The queue-health half of the trend input is bounded by the same 35-day window
the totals half already uses, rather than by an unrelated 100-row cap.

Links & Resources

  • src/queue/signal-snapshot.ts:99-123 — the asymmetric two-history read
  • src/db/repositories.ts:6075-6095listSignalSnapshots, hard limit(100), no time bound
  • src/db/repositories.ts:6150-6199listRecentSignalSnapshotsForTargets, the bounded helper
  • src/services/queue-trends.ts:5-6, :84-118, :154-158 — the windows and the baseline lookup
  • src/db/retention.ts:465-467 — why queue-health is exempt from the dedup prune
  • src/api/routes.ts:1399-1408 — the orb(dashboard): the 8-week slop/duplicate trend can only ever cover ~4 days of history #9699 precedent for the sibling reader
  • src/index.ts:356-357 — the six-hourly enqueue cadence

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions