Skip to content

[Bug]: Maintainer-dashboard "Open PRs cached" metric only counts the first 12 repositories, undercounting the global total (and its sparkline mixes PR-count over repo-count) #398

Description

@joaovictor712

Summary

The /v1/app/maintainer-dashboard handler builds a row of headline metrics. Three of the four are global counts across the full (scoped) data set, but "Open PRs cached" is computed from a list that was deliberately capped at the first 12 repositories, so it under-reports the real number whenever there are more than 12 repos in scope:

// src/api/routes.ts (maintainer-dashboard handler)
const openPullRequests = (
  await Promise.all(repositories.slice(0, 12).map((repo) =>          // <- capped at 12 repos (for the reviewability preview)
    listOpenPullRequests(c.env, repo.fullName).then((rows) => rows.map((pull) => ({ repoFullName: repo.fullName, pull }))),
  ))
).flat();
return c.json({
  // ...
  metrics: [
    { label: "Installations",     value: installations.length, spark: sparklineFromCounts(installations.length, Math.max(installations.length, 1)) },
    { label: "Open PRs cached",   value: openPullRequests.length, spark: sparklineFromCounts(openPullRequests.length, Math.max(repositories.length, 1)) }, // <- BUG
    { label: "Install issues",    value: health.filter((r) => r.status !== "healthy").length, /* ... */ },
    { label: "Rate-limit events", value: rateLimits.length, /* ... */ },
  ],
  reviewability: openPullRequests.slice(0, 20).map(/* ... */),       // the 12-repo cap exists to bound THIS preview list
  // ...
});

openPullRequests is intentionally limited to repositories.slice(0, 12) because it feeds the reviewability preview (sliced again to 20). Reusing that same perf-capped list for the "Open PRs cached" metric value means the metric reflects only the first 12 repositories, not all of them.

Why this is wrong

The metric is presented as a global headline count alongside its siblings, all of which span the full scoped set:

  • Installationsinstallations.length (all installations)
  • Install issueshealth.filter(...) (all health records)
  • Rate-limit eventsrateLimits.length (all rate-limit observations)
  • Open PRs cachedopenPullRequests.lengthonly the first 12 repositories

So for an operator (no scope → all repositories) or any maintainer/owner whose scope includes more than 12 repos, "Open PRs cached" silently undercounts. The repo ordering from listRepositories is arbitrary, so which 12 repos contribute is effectively non-deterministic from the operator's point of view; adding/removing repos can change the number with no actual change in open PRs.

Secondarily, the sparkline for this metric is dimensionally inconsistent:

spark: sparklineFromCounts(openPullRequests.length, Math.max(repositories.length, 1))

The numerator is a PR count (from 12 repos) and the denominator is a repository count (all repos) — two different units. The intended value / max ratio is meaningless here (the numerator can exceed or fall arbitrarily short of the denominator depending on PR density), so the rendered bar does not represent any real proportion.

Reachability

  • GET /v1/app/maintainer-dashboard, gated to maintainer/owner/operator roles.
  • Operators see all repositories (scope === null), so any deployment with >12 registered/installed repos shows an undercounted value. Scoped maintainers with >12 repos in scope are likewise affected.

Failure mode (concrete example)

A deployment has 30 registered repos; the first 12 (arbitrary order) hold 8 open cached PRs, the other 18 hold 25 more.

  • Current: Open PRs cached = 8 (only the first 12 repos), with a sparkline of 8 / 30.
  • Correct: Open PRs cached = 33 (all repos), with a sparkline proportional to a meaningful denominator.

An operator reading the dashboard concludes there are 8 open PRs cached when there are 33 — a ~4x undercount that worsens as repo count grows.

Steps to reproduce

  1. As an operator (or a maintainer scoped to >12 repos), call GET /v1/app/maintainer-dashboard with more than 12 repositories that have cached open PRs.
  2. Compare the Open PRs cached metric value to the actual total number of cached open PRs across all in-scope repos.
  3. Observe the metric equals only the open-PR count of the first 12 repositories.

Expected behavior

"Open PRs cached" reports the total cached open PRs across all in-scope repositories (consistent with the other global metrics), and its sparkline uses a same-unit denominator.

Actual behavior

The metric counts open PRs for only the first 12 repositories (the slice that exists to bound the reviewability preview), and the sparkline divides a 12-repo PR count by the all-repo count.

Suggested fix

Decouple the metric's count from the perf-capped preview fetch. The cached open-PR count per repo is already available cheaply via listRepoSyncStates (RepoSyncStateRecord.openPullRequestsCount, the same source buildRepoDecision uses), so the metric can sum across all in-scope repos in a single query without fetching every repo's PRs:

  • Load listRepoSyncStates(c.env) (already a single query), scope it to repositories (by repoFullName), and compute totalOpenPrs = sum(openPullRequestsCount).
  • Use totalOpenPrs as the Open PRs cached value, and a same-unit denominator for the sparkline (e.g. a sensible max such as the number of in-scope repos times an expected-per-repo bound, or simply Math.max(totalOpenPrs, 1) to render a filled bar like the Installations metric does).
  • Keep the existing repositories.slice(0, 12) fetch solely for the reviewability preview list, where the cap is intended.

Add fail-on-revert coverage: a dashboard built from >12 repos whose later repos hold open PRs must report the full total, not the first-12 subtotal.

Metadata

Metadata

Assignees

No one assigned

    Labels

    slopAI slop and/or attempts to game additional points via manipulation or alt profiles.

    Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions