Skip to content

fix(orb): live fleet reversal + reuse telemetry for the homepage hero metrics - #8821

Merged
JSONbored merged 3 commits into
mainfrom
fix/orb-superseded-reversal-signal
Jul 26, 2026
Merged

fix(orb): live fleet reversal + reuse telemetry for the homepage hero metrics#8821
JSONbored merged 3 commits into
mainfrom
fix/orb-superseded-reversal-signal

Conversation

@JSONbored

@JSONbored JSONbored commented Jul 26, 2026

Copy link
Copy Markdown
Owner

What

Both engineering halves of #8820 (the homepage hero's broken metrics). Advances #8820 — post-deploy ops verification (cursor reset + live numbers) remains before it closes.

Half 1 — decision accuracy pegged at a degenerate 100%

Production evidence (2026-07-26): the self-host instance recorded 66 reversal_superseded events (the one-shot culture's dominant real "bot was wrong" shape, #8166), yet all 6,127 exported fleet signals carry reversal_flag='none'. Three stacked drops: the exporter's rev CTE only matched reverted/reopened; the ingest whitelist downgraded the unknown flag to 'none'; and orb_signals' CHECK constraint would have rejected the value anyway — swallowed silently by the best-effort insert. Fleet reversalRate stayed pinned at 0 → published accuracy exactly 100%.

  • orb-collector: rev CTE + flag mapping carry superseded (priority reverted > reopened > superseded); regression test for the reversal-lands-after-first-export re-export path
  • ingest: whitelist superseded
  • analytics: a superseded close disconfirms closePrecision and counts toward reversalRate exactly like a reopen (parity with public-stats.ts, which already counts it)
  • migration 0176: rebuild orb_signals with the widened CHECK (SQLite can't alter one), preserving rows

Half 2 — "AI work reused" renders a dash beside a live-looking sparkline

The trend reads cache hit/miss audit events from the cloud worker's own ledger, frozen since the self-host cutover (last event 2026-06-29) — recent weeks fall under the 5-sample publish floor → null → dash, while the live signal (133k+ cache events) accrues unexported on the self-host.

  • orb-collector: export day-bucketed hit/miss aggregates (counts only — no repos, no PRs, no content) over a 70-day rolling window, riding the same hourly POST; fail-safe when the ledger lacks the table; same event population as the trend's own query (shared AI_REVIEW_REUSE_EVENT_TYPES)
  • ingest: strict validation (YYYY-MM-DD day, clamped non-negative counts, bounded row count) + idempotent upsert per (instance, day)
  • public-reuse-rate-trend: fold counters from REGISTERED instances (the same trust anchor as computeFleetAnalytics) into the same weekly buckets, unconditional on the own-ledger repo allowlist — parity with the fleet-accuracy fold
  • migration 0177: orb_reuse_counters

No UI changes needed — both tiles light up from the same fields once live data flows.

Post-merge ops (tracked on #8820)

  1. Ship in the next selfhost image (rides orb-v3.5.0-beta.2).
  2. Reset the instance's orb_export_cursor row so history re-exports with correct reversal flags (INSERT OR REPLACE dedup makes this safe and idempotent).
  3. Verify the hero: accuracy off the degenerate 100 with real reversal grounding; reuse tile populated.

Verification

  • TSC clean; all affected suites green (collector, ingest, analytics, reuse-trend, public-stats, migration-collisions).
  • Changed-line coverage: 0 uncovered statements/branches across all five touched src files.

…calibration

The fleet pipeline silently dropped reversal_superseded (#8166) at every
hop: the self-host exporter's rev CTE only matched reverted/reopened,
the collector's ingest whitelist downgraded an unknown flag to 'none',
and orb_signals' CHECK constraint would have rejected the value anyway
(swallowed by the best-effort insert). Since supersession is the
one-shot culture's dominant real reversal shape, the fleet's published
reversalRate stayed pinned at 0 and the homepage's reversal-grounded
decision accuracy read a degenerate 100%.

Advances #8820 (the accuracy-number half; the reuse-rate tile is a
separate change).

- orb-collector: rev CTE + flag mapping carry 'superseded' (priority
  reverted > reopened > superseded), regression-tested for the
  reversal-recorded-after-first-export re-export path
- ingest: whitelist 'superseded'
- analytics: a superseded close disconfirms closePrecision and counts
  toward reversalRate exactly like a reopen
- migration 0176: rebuild orb_signals with the widened CHECK
@JSONbored JSONbored self-assigned this Jul 26, 2026
…rk-reused trend

The homepage reuse-rate trend reads cache hit/miss audit events from
the cloud worker's own ledger, which froze at the self-host cutover
(last event 2026-06-29) — recent weekly buckets fall under the publish
floor, so the hero tile renders a dash beside a decaying sparkline
while the live signal (133k+ cache events) accrues unexported on the
self-hosted instances.

Advances #8820 (the reuse-rate half; the accuracy half is the
superseded-reversal export on this same branch's sibling commit).

- orb-collector: export day-bucketed hit/miss aggregates (counts only,
  no repos/PRs) over a 70-day rolling window on the same hourly POST;
  fail-safe when the ledger lacks the table
- ingest: validate (strict day format, clamped non-negative counts)
  and upsert per (instance, day); malformed rows skipped row-by-row
- public-reuse-rate-trend: fold counters from REGISTERED instances
  into the same weekly buckets, unconditional on the own-ledger repo
  allowlist (parity with the fleet-accuracy fold)
- migration 0177: orb_reuse_counters
@JSONbored JSONbored changed the title fix(orb): carry the superseded reversal through the fleet-calibration pipeline fix(orb): live fleet reversal + reuse telemetry for the homepage hero metrics Jul 26, 2026
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@JSONbored
JSONbored merged commit f84ab1d into main Jul 26, 2026
3 of 4 checks passed
@JSONbored
JSONbored deleted the fix/orb-superseded-reversal-signal branch July 26, 2026 09:25
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 26, 2026
@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

❌ 3 Tests Failed:

Tests completed Failed Passed Skipped
21748 3 21745 21
View the top 3 failed test(s) by shortest run time
test/unit/queue-2.test.ts > queue processors > #4603: the SAME sub-floor defect one-shot-closes when aiReviewLowConfidenceDisposition is explicitly one_shot
Stack Traces | 0.393s run time
AssertionError: expected 'error' to be 'completed' // Object.is equality

Expected: "completed"
Received: "error"

 ❯ test/unit/queue-2.test.ts:361:28
test/unit/queue.test.ts > queue processors > linked-issue hard-rule violation persistence (#linked-issue-hard-rule-persistence) > REGRESSION (live-issue-state-change-before-re-evaluation): Pass 1 flags a real violation; Pass 2's live re-parse re-evaluates the SAME issue as clean (assignee removed) but the persisted violation still closes the PR
Stack Traces | 0.396s run time
AssertionError: expected 'error' to be 'completed' // Object.is equality

Expected: "completed"
Received: "error"

 ❯ test/unit/queue.test.ts:2843:32
test/unit/queue.test.ts > queue processors > linked-issue hard-rule violation persistence (#linked-issue-hard-rule-persistence) > REGRESSION (body-edit-during-grace-window): Pass 1 flags a real owner-assigned violation; Pass 2's live re-parse finds NO linked issues (body edited) but the persisted violation still closes the PR
Stack Traces | 0.476s run time
AssertionError: expected 'error' to be 'completed' // Object.is equality

Expected: "completed"
Received: "error"

 ❯ test/unit/queue.test.ts:2798:32

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant