Skip to content

feat(services): miner-vs-human cohort split for the gate-precision dashboard card - #4623

Merged
JSONbored merged 1 commit into
mainfrom
feat/gate-precision-cohort-split-4520
Jul 10, 2026
Merged

feat(services): miner-vs-human cohort split for the gate-precision dashboard card#4623
JSONbored merged 1 commit into
mainfrom
feat/gate-precision-cohort-split-4520

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Adds an additive, opt-in miner-vs-human split to loadGatePrecisionReport (the feat(ui): gate-precision analytics card (precision + confusion matrix) #2191 gate-precision
    card) — the existing blended perGateType/overall output is completely unchanged; a new cohorts field
    is only present when explicitly requested.
  • No new migration, no identity added to any privacy-scoped table. pull_requests already carries
    authorLogin directly (unlike review_audit, which is deliberately identity-free by design) — the split
    is computed entirely at read time by cross-referencing each gate-block outcome's PR author against a
    confirmed-miner login set.
  • fetchOfficialGittensorMinerLogins (new, src/gittensor/api.ts) fetches the full miner list in ONE call,
    for callers that need to classify many distinct submitters at once — the existing
    fetchOfficialGittensorMiner(login) re-fetches per login (fine for a single webhook-triggered lookup, not
    for a dashboard read touching many PRs' authors). Fail-safe: any fetch failure degrades to an empty set
    (every author reads as human), never throws.
  • GET /v1/repos/:owner/:repo/gate-precision?includeCohorts=true opts a maintainer-dashboard request into
    the extra Gittensor API call; every existing caller (no param) stays byte-identical, including zero extra
    network calls.
  • An unresolvable or unconfirmed PR author always falls to human — never over-classified as a miner.

Scope note: the issue names reversal-rate and findings-by-category as additional targets ("at minimum"
gate-precision). I looked at both:

Rather than guess at unconfirmed wiring, I'm shipping the concretely-defined, explicitly-named floor
("at minimum... gate-precision") as its own focused PR. Happy to pick up the other two as follow-ups once
their actual dashboard surface is confirmed.

Fixes #4520

Test plan

  • test/unit/gate-precision.test.ts: 9 new cases — cohort split absent by default (byte-identical),
    miner/human split with MIN_SAMPLE applied independently per cohort, unresolvable-author fail-safe to
    human, case-insensitive login matching, no-identity-leak privacy check, loadGatePrecisionReport's
    includeCohorts opt-in (fetches the miner list exactly once, zero calls when omitted), and a
    Gittensor-API-failure fail-safe (empty set, not a thrown error)
  • test/unit/gittensor-api.test.ts: 4 new cases for fetchOfficialGittensorMinerLogins — one-call
    batch fetch, missing-githubUsername entries skipped, and fail-safe-to-empty-set on both an HTTP
    error and a network-level rejection
  • test/integration/api.test.ts: extended the existing /gate-precision route coverage with
    ?includeCohorts=true (reuses the test's own already-stubbed /miners endpoint)
  • Verified via lcov branch data across all 3 changed source files (gate-precision.ts, gittensor/api.ts,
    api/routes.ts): zero uncovered lines or branches in this diff
  • npx tsc --noEmit clean; npm run ui:openapi:check clean (this route isn't part of the generated
    OpenAPI surface, confirmed via a clean check with no diff); npm run db:migrations:check clean (no
    migrations touched)
  • Rebased onto latest origin/main at push time; git diff --check clean

@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
13202 1 13201 12
View the full list of 1 ❄️ flaky test(s)
test/unit/queue.test.ts > queue processors > PR-panel generate-tests checkbox (#4589) > respects the repo's configured commit delivery mode via the checkbox (NOT forced comment-only, unlike the auto-trigger)

Flake rate in main: 100.00% (Passed 0 times, Failed 2 times)

Stack Traces | 0.921s run time
AssertionError: expected [] to deeply equal [ 'tree', 'commit', 'ref' ]

- Expected
+ Received

- [
-   "tree",
-   "commit",
-   "ref",
- ]
+ []

 ❯ test/unit/queue.test.ts:26987:25

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 10, 2026
@loopover-orb

loopover-orb Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Caution

🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥

🛑 Gittensory review result - fixes required

Review updated: 2026-07-10 07:56:39 UTC

6 files · 1 AI reviewer · no blockers · readiness 100/100 · CI failing · blocked

🛑 Suggested Action - Manual Review

  • Touches a guarded path — held for manual review: This PR changes guardrail-protected path(s): src/services/gate-precision.ts (matched src/services/**).

Review summary
This adds an additive, opt-in miner-vs-human cohort split to the gate-precision report: buildGatePrecisionReport's fold logic is correctly extracted into foldGateOutcomes and reused for blended/miner/human folds over disjoint partitions of the same scoped outcomes, so miner+human always reconciles to overall. The new fetchOfficialGittensorMinerLogins batches miner classification in one call and fails safe to an empty set (never throws, never over-classifies as miner), and the route/loader wiring keeps every existing caller byte-identical since includeCohorts/minerLogins are only threaded through when explicitly requested. Tests are thorough (partition math, case-insensitivity, unresolvable-author fallback, privacy — no login leaks in cohorts, opt-in-only network call) and trace to the real code paths shown.

Nits — 7 non-blocking
  • src/services/gate-precision.ts: PullRequestRecord.authorLogin is assumed to already exist on the type (per the PR description) but the type definition isn't in the visible diff — worth confirming it's actually an existing field and not something this PR silently relies on without adding.
  • The issue (feat(ui): add a miner-vs-human breakdown to the Wave-3 gate-precision / reversal-rate / findings-by-category dashboard cards #4520) asks for miner-vs-human split across gate-precision, reversal-rate, and findings-by-category 'at minimum'; this PR only ships gate-precision. The description gives a reasoned justification (no confirmed dashboard surface for the other two yet), but as the maintainer I'd want an explicit go-ahead on scoping to gate-precision only rather than accepting the justification unilaterally.
  • src/api/routes.ts: `includeCohorts=true` is a strict string match — confirm this matches the convention used by other boolean query params in this file for consistency.
  • fetchOfficialGittensorMinerLogins fetches the full global miner list on every includeCohorts=true dashboard read with no caching — fine at current scale but worth a comment noting it if the miner list is expected to grow large.
  • Consider naming the MIN_SAMPLE-per-cohort behavior explicitly in the GatePrecisionCohortReport doc comment (it's already covered on foldGateOutcomes, so this is optional).
  • PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
  • Touches a guarded path — held for manual review — A maintainer must review and merge this change.

CI checks failing

  • validate
  • validate-code
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #4520
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 48 registered-repo PR(s), 40 merged, 334 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 48 PR(s), 334 issue(s).
Gate result ⚠️ Not blocking Advisory; not blocking this PR.
Linked issue satisfaction

Partially addressed
The PR adds a working, tested, privacy-safe miner-vs-human cohort split to the gate-precision card (deliverable 1 and partial deliverable 2), with tests for cohort accuracy/privacy/blended-unaffected and doc-comment notes, but it explicitly declines to extend reversal-rate or findings-by-category as the issue requires 'at minimum', and includes no changes to standalone dashboard/analytics document

Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 48 PR(s), 334 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

…shboard card (#4520)

Adds an additive, opt-in miner-vs-human split to loadGatePrecisionReport
(#2191) -- no new migration, no identity added to any privacy-scoped table.
pull_requests already carries authorLogin (unlike the identity-free
review_audit), so the split is computed entirely at read time: fetch the
full confirmed-miner login set once (fetchOfficialGittensorMinerLogins,
new), then classify each gate-block outcome's PR author against it.

?includeCohorts=true opts a maintainer-dashboard request into the extra
Gittensor API call; every existing caller (no param) stays byte-identical.
An unresolvable/unconfirmed author always falls to "human" -- never
over-classified as a miner.

Scope note: the issue also names reversal-rate and findings-by-category as
stretch targets ("at minimum" gate-precision). reversal-rate's dashboard-
card wiring isn't as concretely defined as gate-precision's (its clearest
existing implementation, computeAgentHealth, is consumed by the internal
ops-alerting system, not a confirmed dashboard endpoint), and
findings-by-category has no repo-wide aggregate to extend yet (only
per-PR MCP counts from #4519). Deferred rather than guessed at.

Fixes #4520
@JSONbored
JSONbored force-pushed the feat/gate-precision-cohort-split-4520 branch from a76409c to 2e15a6b Compare July 10, 2026 08:13
@JSONbored
JSONbored merged commit efcd25d into main Jul 10, 2026
6 of 7 checks passed
@JSONbored
JSONbored deleted the feat/gate-precision-cohort-split-4520 branch July 10, 2026 08:21
JSONbored added a commit that referenced this pull request Jul 10, 2026
…igest (#4521)

Extends MaintainerRecapRepoInput/RecapReport with an additive, opt-in
miner-vs-human split, sourced entirely from the upstream GatePrecisionReport
cohorts field (#4520) -- no independent identity read, no new privacy
surface. A repo's recap entry gets a `cohorts` field whenever its injected
GatePrecisionReport carried one; the aggregate totals.cohorts sums only
across repos that did, so a window mixing cohort-aware and legacy call
sites still degrades gracefully instead of half-reporting zeros.

runMaintainerRecapJob now opts loadGatePrecisionReport into includeCohorts
by default -- a periodic digest is exactly the "occasional aggregate view"
that option was designed for, unlike a hot webhook path.

formatMaintainerRecap gains an additive "## Cohorts" section, rendered only
when totals.cohorts is present, sitting between Totals and Per-repo.

Stacks on #4520 (branch feat/gate-precision-cohort-split-4520, PR #4623) --
GatePrecisionReport.cohorts doesn't exist without it. Do not merge before
#4623.

Fixes #4521
JSONbored added a commit that referenced this pull request Jul 10, 2026
…igest (#4521) (#4625)

Extends MaintainerRecapRepoInput/RecapReport with an additive, opt-in
miner-vs-human split, sourced entirely from the upstream GatePrecisionReport
cohorts field (#4520) -- no independent identity read, no new privacy
surface. A repo's recap entry gets a `cohorts` field whenever its injected
GatePrecisionReport carried one; the aggregate totals.cohorts sums only
across repos that did, so a window mixing cohort-aware and legacy call
sites still degrades gracefully instead of half-reporting zeros.

runMaintainerRecapJob now opts loadGatePrecisionReport into includeCohorts
by default -- a periodic digest is exactly the "occasional aggregate view"
that option was designed for, unlike a hot webhook path.

formatMaintainerRecap gains an additive "## Cohorts" section, rendered only
when totals.cohorts is present, sitting between Totals and Per-repo.

Stacks on #4520 (branch feat/gate-precision-cohort-split-4520, PR #4623) --
GatePrecisionReport.cohorts doesn't exist without it. Do not merge before
#4623.

Fixes #4521
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Development

Successfully merging this pull request may close these issues.

feat(ui): add a miner-vs-human breakdown to the Wave-3 gate-precision / reversal-rate / findings-by-category dashboard cards

1 participant