Skip to content

orb(proof): the public proof page's accuracy is 100% by construction, and a waived/pruned ledger still badges "verified" #10012

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

src/review/proof-summary.ts renders the unauthenticated per-repo proof page. Two of its published claims are
structurally unable to be anything but perfect.

1. confirmed can never differ from decided, so the published accuracy is always exactly 1

loadProofSummary computes both counters itself (src/review/proof-summary.ts:304-315):

SELECT COUNT(*) AS decisionCount,
       SUM(CASE WHEN action IN ('merge', 'close') THEN 1 ELSE 0 END) AS decided,
       SUM(CASE WHEN action IN ('merge', 'close') AND reason_code NOT LIKE 'reversal%' THEN 1 ELSE 0 END) AS confirmed
  FROM decision_records WHERE repo_full_name = ?

decision_records.reason_code is TEXT NOT NULL (migrations/0179_decision_records.sql:16) and is written
at decision time from exactly two derivations, neither of which can produce a reversal… value:

  • deriveDecisionReasonCode (src/review/decision-replay.ts:123-125):
    blockerClass !== "none" ? blockerClass : policyCloseKind != null ? \policy_close:${policyCloseKind}` : conclusion`
  • defaultDecisionRecordReasonCode (src/services/agent-action-executor.ts:329-331):
    action.closeKind !== undefined ? \policy_close:${action.closeKind}` : "success"`

Those are the only two values reaching buildDecisionRecord's reasonCode (src/queue/processors.ts:3935
and src/services/agent-action-executor.ts:356), and nothing anywhere UPDATEs an existing
decision_records row's reason_code. Reversals are recorded as separate review_audit /audit_events
rows (reversal_reverted / reversal_reopened / reversal_superseded, src/review/outcomes-wire.ts), which
this query never touches.

So confirmed === decided for every repo, always, and buildProofAccuracy
(src/review/proof-summary.ts:86-96) publishes accuracy: round3(confirmed / decided) = 1 with a Wilson
interval hugging 1 for any repo past the 20-decision floor. This is precisely the failure class the codebase
has already diagnosed once on a different surface: src/review/public-stats.ts:411-413 records that a
mis-chosen denominator "was why the published figure was STILL 100% for all three repos on 2377/602/508
reviewed with 0 reversals".

The module header also states the opposite of what the code does
(src/review/proof-summary.ts:82-83):

 * PURE. `confirmed`/`decided` come from the already-public precision block; nothing new is computed here
 * and no new SQL surface exists for this page.

The already-public precision block is loadPublicRulePrecision
(src/review/public-rule-precision.ts:88-127), which derives confirmed/reversed from
signal.human_override:<ruleId> rows' $.verdict. The page does not read it.

2. A ledger with declared waivers or pruned preimages still badges as clean "verified"

verifyDecisionLedger reports prunedRecords, waivedContentMismatches and waivedUnchainedRecords
alongside ok, and its own doc comment says why (src/review/decision-record.ts:667-677): they are
"counted and published separately, never folded into a clean result's silence, so '231 records were never
chained' can never read as 'nothing to see here'". ok stays true for all three.

The proof page's dependency signature drops them (src/review/proof-summary.ts:285 and :351):

verifyLedger: (env: Env) => Promise<{ ok: boolean; tipSeq: number; totalCount: number; break?: { kind: string; atSeq: number } | undefined }>;

so buildProofLedgerStatus (:117-133) can only ever return { state: "verified", tipSeq, totalCount }, and
buildProofBadgeMessage (:175-186) renders "verified · anchored". On an instance carrying the
LOOPOVER_LEDGER_UNCHAINED_WAIVER / LOOPOVER_LEDGER_CONTENT_WAIVER declarations those parsers exist to
support, the most public surface renders exactly the silence the internal verifier refuses to.

Neither defect is covered: test/unit/proof-summary.test.ts:203-240 seeds 3 decision records (below the
20-decision floor), so the accuracy SQL never produces a published rate in any test, and every
verifyLedger stub in that file returns only { ok, tipSeq, totalCount }.

Requirements

  • loadProofSummary must stop deriving confirmed from reason_code. confirmed must be the count of
    merge/close decisions for this repo that were NOT later reversed, where "reversed" means an
    audit_events row of type reversal_reverted, reversal_reopened or reversal_superseded whose
    target_key is <repo_full_name>#<pull_number> for that decision. decided stays the count of
    action IN ('merge','close') rows.
  • The reversal read must run inside its own section(...) wrapper (src/review/proof-summary.ts:296-303) so a
    failing read degrades that section rather than 503-ing the page, matching the existing per-section contract.
    A failing reversal read must degrade toward confirmed === decided never being asserted: on that failure
    the page must publish { state: "insufficient_data", decided, minimumDecisions }, not a rate.
  • ProofLedgerStatus's verified variant must additionally carry prunedRecords: number,
    waivedContentMismatches: number and waivedUnchainedRecords: number, and buildProofLedgerStatus must
    populate them from the verify result.
  • The verifyLedger dependency signature in BOTH loadProofSummary's deps (:285) and ProofPageDeps
    (:351) must accept those three counts.
  • buildProofBadgeMessage must return "verified · N excluded" (N = the sum of the three counts) instead of
    "verified" / "verified · anchored" whenever that sum is greater than 0, and
    buildProofBadgeColor must return the neutral "#9e9e9e" for that case. A sum of 0 must keep today's
    exact strings and colors byte-identically.
  • Behaviour that must NOT change: PROOF_MIN_DECISIONS (20), PROOF_SAMPLE_RECORDS (5), round3,
    buildProofAccuracy's signature and its insufficient_data shape, buildProofAnchorStatus,
    PROOF_BOUNDARY_STATEMENT, the allowlist-by-name privacy discipline in buildProofSummary, and the
    empty / broken / unavailable ledger states.

⚠️ Required pattern: for the reversal join, mirror loadPublicRulePrecision's treatment of the same event
types (src/review/public-rule-precision.ts:129-136) — the three reversal_* audit_events types are the
canonical reversal signal. For the per-section failure discipline, mirror loadProofSummary's existing
section() helper (src/review/proof-summary.ts:296-303). What does NOT satisfy this issue: widening the
NOT LIKE 'reversal%' pattern (the column never carries that value at all); starting to write a
reversal… reason_code onto decision_records (a new write path, and it would corrupt
deriveDecisionReasonCode's replay contract in src/review/decision-replay.ts:123); adding a second
"verified with caveats" badge route instead of changing the existing one; fixing only the accuracy half or
only the ledger half.

Deliverables

  • src/review/proof-summary.ts: confirmed is computed from the reversal_* audit_events signal, not
    from reason_code. With 25 merge/close decision_records rows for o/r and 2
    reversal_reverted rows whose target_key matches two of them, loadProofSummary(env, "o/r", …)
    returns accuracy: { state: "published", decided: 25, confirmed: 23, accuracy: 0.92, … }.
  • src/review/proof-summary.ts: ProofLedgerStatus's verified variant carries prunedRecords,
    waivedContentMismatches and waivedUnchainedRecords, populated by buildProofLedgerStatus.
  • src/review/proof-summary.ts: buildProofBadgeMessage(summary) returns "verified · 231 excluded"
    for a summary whose ledger is verified with waivedUnchainedRecords: 231 and the other two 0, and
    buildProofBadgeColor returns "#9e9e9e" for it.
  • A test in test/unit/proof-summary.test.ts asserting the 25-decisions/2-reversals case above yields a
    published rate strictly below 1.
  • A test in test/unit/proof-summary.test.ts asserting that 25 merge/close records with ZERO reversal
    rows still publishes accuracy: 1 — the perfect case must remain expressible, it just must no longer
    be the only expressible one.
  • A test in test/unit/proof-summary.test.ts asserting a failing reversal read degrades to
    accuracy: { state: "insufficient_data" } rather than publishing a rate.
  • Tests in test/unit/proof-summary.test.ts for buildProofBadgeMessage / buildProofBadgeColor
    covering both the zero-excluded (unchanged strings/colors) and nonzero-excluded cases.
  • A regression test at test/unit/proof-summary.test.ts named for this bug (e.g.
    "REGRESSION: the published accuracy is derived from real reversals, not from reason_code") that fails
    against the current reason_code NOT LIKE 'reversal%' query.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
fixing the accuracy query but leaving the ledger badge silent about waived/pruned rows, or adding the three
counts to the type without wiring them into the badge — does not resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. vitest.config.ts's coverage.include
covers src/**/*.ts and packages/loopover-engine/src/**/*.ts; src/review/proof-summary.ts is measured and
gated. Every branch the change introduces or touches needs both arms tested: the new reversal-read
section() success and failure arms; buildProofAccuracy's !interval || decided < minimumDecisions
ternary (both disjuncts, plus the passing arm); buildProofLedgerStatus's verified arm with and without a
nonzero exclusion count; buildProofBadgeMessage's new excluded > 0 / excluded === 0 split crossed with
its existing anchor.state === "anchored" split; and buildProofBadgeColor's same split.

Expected Outcome

The public proof page publishes an accuracy figure that can actually go down when a human overturns a bot
decision, and a repo whose ledger carries declared waivers or pruned preimages says so on its badge instead of
reading as unqualified "verified".

Links & Resources

  • src/review/proof-summary.ts:76-96buildProofAccuracy and its "already-public precision block" claim
  • src/review/proof-summary.ts:304-315 — the reason_code NOT LIKE 'reversal%' query
  • src/review/proof-summary.ts:38-48, :117-133, :175-198ProofLedgerStatus, buildProofLedgerStatus,
    the badge message/color
  • src/review/decision-replay.ts:123-125 / src/services/agent-action-executor.ts:329-331 — the only two
    reason_code derivations
  • migrations/0179_decision_records.sql:16reason_code TEXT NOT NULL
  • src/review/decision-record.ts:656-680verifyDecisionLedger's waived/pruned counts and their contract
  • src/review/public-rule-precision.ts:88-136 — the canonical reversal signal
  • src/review/public-stats.ts:405-415 — the same 100%-by-construction failure, already diagnosed elsewhere

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions