You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.
Context
src/review/proof-summary.ts renders the unauthenticated per-repo proof page. Two of its published claims are
structurally unable to be anything but perfect.
1. confirmed can never differ from decided, so the published accuracy is always exactly 1
loadProofSummary computes both counters itself (src/review/proof-summary.ts:304-315):
SELECTCOUNT(*) AS decisionCount,
SUM(CASE WHEN action IN ('merge', 'close') THEN 1 ELSE 0 END) AS decided,
SUM(CASE WHEN action IN ('merge', 'close') AND reason_code NOT LIKE'reversal%' THEN 1 ELSE 0 END) AS confirmed
FROM decision_records WHERE repo_full_name = ?
decision_records.reason_code is TEXT NOT NULL (migrations/0179_decision_records.sql:16) and is written
at decision time from exactly two derivations, neither of which can produce a reversal… value:
Those are the only two values reaching buildDecisionRecord's reasonCode (src/queue/processors.ts:3935
and src/services/agent-action-executor.ts:356), and nothing anywhere UPDATEs an existing decision_records row's reason_code. Reversals are recorded as separate review_audit /audit_events
rows (reversal_reverted / reversal_reopened / reversal_superseded, src/review/outcomes-wire.ts), which
this query never touches.
So confirmed === decided for every repo, always, and buildProofAccuracy
(src/review/proof-summary.ts:86-96) publishes accuracy: round3(confirmed / decided) = 1 with a Wilson
interval hugging 1 for any repo past the 20-decision floor. This is precisely the failure class the codebase
has already diagnosed once on a different surface: src/review/public-stats.ts:411-413 records that a
mis-chosen denominator "was why the published figure was STILL 100% for all three repos on 2377/602/508
reviewed with 0 reversals".
The module header also states the opposite of what the code does
(src/review/proof-summary.ts:82-83):
* PURE. `confirmed`/`decided` come from the already-public precision block; nothing new is computed here
* and no new SQL surface exists for this page.
The already-public precision block is loadPublicRulePrecision
(src/review/public-rule-precision.ts:88-127), which derives confirmed/reversed from signal.human_override:<ruleId> rows' $.verdict. The page does not read it.
2. A ledger with declared waivers or pruned preimages still badges as clean "verified"
verifyDecisionLedger reports prunedRecords, waivedContentMismatches and waivedUnchainedRecords
alongside ok, and its own doc comment says why (src/review/decision-record.ts:667-677): they are
"counted and published separately, never folded into a clean result's silence, so '231 records were never
chained' can never read as 'nothing to see here'". ok stays true for all three.
The proof page's dependency signature drops them (src/review/proof-summary.ts:285 and :351):
so buildProofLedgerStatus (:117-133) can only ever return { state: "verified", tipSeq, totalCount }, and buildProofBadgeMessage (:175-186) renders "verified · anchored". On an instance carrying the LOOPOVER_LEDGER_UNCHAINED_WAIVER / LOOPOVER_LEDGER_CONTENT_WAIVER declarations those parsers exist to
support, the most public surface renders exactly the silence the internal verifier refuses to.
Neither defect is covered: test/unit/proof-summary.test.ts:203-240 seeds 3 decision records (below the
20-decision floor), so the accuracy SQL never produces a published rate in any test, and every verifyLedger stub in that file returns only { ok, tipSeq, totalCount }.
Requirements
loadProofSummary must stop deriving confirmed from reason_code. confirmed must be the count of
merge/close decisions for this repo that were NOT later reversed, where "reversed" means an audit_events row of type reversal_reverted, reversal_reopened or reversal_superseded whose target_key is <repo_full_name>#<pull_number> for that decision. decided stays the count of action IN ('merge','close') rows.
The reversal read must run inside its own section(...) wrapper (src/review/proof-summary.ts:296-303) so a
failing read degrades that section rather than 503-ing the page, matching the existing per-section contract.
A failing reversal read must degrade toward confirmed === decided never being asserted: on that failure
the page must publish { state: "insufficient_data", decided, minimumDecisions }, not a rate.
ProofLedgerStatus's verified variant must additionally carry prunedRecords: number, waivedContentMismatches: number and waivedUnchainedRecords: number, and buildProofLedgerStatus must
populate them from the verify result.
The verifyLedger dependency signature in BOTH loadProofSummary's deps (:285) and ProofPageDeps
(:351) must accept those three counts.
buildProofBadgeMessage must return "verified · N excluded" (N = the sum of the three counts) instead of "verified" / "verified · anchored" whenever that sum is greater than 0, and buildProofBadgeColor must return the neutral "#9e9e9e" for that case. A sum of 0 must keep today's
exact strings and colors byte-identically.
Behaviour that must NOT change: PROOF_MIN_DECISIONS (20), PROOF_SAMPLE_RECORDS (5), round3, buildProofAccuracy's signature and its insufficient_data shape, buildProofAnchorStatus, PROOF_BOUNDARY_STATEMENT, the allowlist-by-name privacy discipline in buildProofSummary, and the empty / broken / unavailable ledger states.
⚠️ Required pattern: for the reversal join, mirror loadPublicRulePrecision's treatment of the same event
types (src/review/public-rule-precision.ts:129-136) — the three reversal_*audit_events types are the
canonical reversal signal. For the per-section failure discipline, mirror loadProofSummary's existing section() helper (src/review/proof-summary.ts:296-303). What does NOT satisfy this issue: widening the NOT LIKE 'reversal%' pattern (the column never carries that value at all); starting to write a reversal…reason_code onto decision_records (a new write path, and it would corrupt deriveDecisionReasonCode's replay contract in src/review/decision-replay.ts:123); adding a second
"verified with caveats" badge route instead of changing the existing one; fixing only the accuracy half or
only the ledger half.
Deliverables
src/review/proof-summary.ts: confirmed is computed from the reversal_*audit_events signal, not
from reason_code. With 25 merge/close decision_records rows for o/r and 2 reversal_reverted rows whose target_key matches two of them, loadProofSummary(env, "o/r", …)
returns accuracy: { state: "published", decided: 25, confirmed: 23, accuracy: 0.92, … }.
src/review/proof-summary.ts: ProofLedgerStatus's verified variant carries prunedRecords, waivedContentMismatches and waivedUnchainedRecords, populated by buildProofLedgerStatus.
src/review/proof-summary.ts: buildProofBadgeMessage(summary) returns "verified · 231 excluded"
for a summary whose ledger is verified with waivedUnchainedRecords: 231 and the other two 0, and buildProofBadgeColor returns "#9e9e9e" for it.
A test in test/unit/proof-summary.test.ts asserting the 25-decisions/2-reversals case above yields a
published rate strictly below 1.
A test in test/unit/proof-summary.test.ts asserting that 25 merge/close records with ZERO reversal
rows still publishes accuracy: 1 — the perfect case must remain expressible, it just must no longer
be the only expressible one.
A test in test/unit/proof-summary.test.ts asserting a failing reversal read degrades to accuracy: { state: "insufficient_data" } rather than publishing a rate.
Tests in test/unit/proof-summary.test.ts for buildProofBadgeMessage / buildProofBadgeColor
covering both the zero-excluded (unchanged strings/colors) and nonzero-excluded cases.
A regression test at test/unit/proof-summary.test.ts named for this bug (e.g. "REGRESSION: the published accuracy is derived from real reversals, not from reason_code") that fails
against the current reason_code NOT LIKE 'reversal%' query.
All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
fixing the accuracy query but leaving the ledger badge silent about waived/pruned rows, or adding the three
counts to the type without wiring them into the badge — does not resolve this issue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted. vitest.config.ts's coverage.include
covers src/**/*.ts and packages/loopover-engine/src/**/*.ts; src/review/proof-summary.ts is measured and
gated. Every branch the change introduces or touches needs both arms tested: the new reversal-read section() success and failure arms; buildProofAccuracy's !interval || decided < minimumDecisions
ternary (both disjuncts, plus the passing arm); buildProofLedgerStatus's verified arm with and without a
nonzero exclusion count; buildProofBadgeMessage's new excluded > 0 / excluded === 0 split crossed with
its existing anchor.state === "anchored" split; and buildProofBadgeColor's same split.
Expected Outcome
The public proof page publishes an accuracy figure that can actually go down when a human overturns a bot
decision, and a repo whose ledger carries declared waivers or pruned preimages says so on its badge instead of
reading as unqualified "verified".
Links & Resources
src/review/proof-summary.ts:76-96 — buildProofAccuracy and its "already-public precision block" claim
src/review/proof-summary.ts:304-315 — the reason_code NOT LIKE 'reversal%' query
src/review/proof-summary.ts:38-48, :117-133, :175-198 — ProofLedgerStatus, buildProofLedgerStatus,
the badge message/color
src/review/decision-replay.ts:123-125 / src/services/agent-action-executor.ts:329-331 — the only two reason_code derivations
migrations/0179_decision_records.sql:16 — reason_code TEXT NOT NULL
src/review/decision-record.ts:656-680 — verifyDecisionLedger's waived/pruned counts and their contract
src/review/public-rule-precision.ts:88-136 — the canonical reversal signal
src/review/public-stats.ts:405-415 — the same 100%-by-construction failure, already diagnosed elsewhere
Context
src/review/proof-summary.tsrenders the unauthenticated per-repo proof page. Two of its published claims arestructurally unable to be anything but perfect.
1.
confirmedcan never differ fromdecided, so the published accuracy is always exactly1loadProofSummarycomputes both counters itself (src/review/proof-summary.ts:304-315):decision_records.reason_codeisTEXT NOT NULL(migrations/0179_decision_records.sql:16) and is writtenat decision time from exactly two derivations, neither of which can produce a
reversal…value:deriveDecisionReasonCode(src/review/decision-replay.ts:123-125):blockerClass !== "none" ? blockerClass : policyCloseKind != null ? \policy_close:${policyCloseKind}` : conclusion`defaultDecisionRecordReasonCode(src/services/agent-action-executor.ts:329-331):action.closeKind !== undefined ? \policy_close:${action.closeKind}` : "success"`Those are the only two values reaching
buildDecisionRecord'sreasonCode(src/queue/processors.ts:3935and
src/services/agent-action-executor.ts:356), and nothing anywhere UPDATEs an existingdecision_recordsrow'sreason_code. Reversals are recorded as separatereview_audit/audit_eventsrows (
reversal_reverted/reversal_reopened/reversal_superseded,src/review/outcomes-wire.ts), whichthis query never touches.
So
confirmed === decidedfor every repo, always, andbuildProofAccuracy(
src/review/proof-summary.ts:86-96) publishesaccuracy: round3(confirmed / decided)=1with a Wilsoninterval hugging 1 for any repo past the 20-decision floor. This is precisely the failure class the codebase
has already diagnosed once on a different surface:
src/review/public-stats.ts:411-413records that amis-chosen denominator "was why the published figure was STILL 100% for all three repos on 2377/602/508
reviewed with 0 reversals".
The module header also states the opposite of what the code does
(
src/review/proof-summary.ts:82-83):The already-public precision block is
loadPublicRulePrecision(
src/review/public-rule-precision.ts:88-127), which derivesconfirmed/reversedfromsignal.human_override:<ruleId>rows'$.verdict. The page does not read it.2. A ledger with declared waivers or pruned preimages still badges as clean "verified"
verifyDecisionLedgerreportsprunedRecords,waivedContentMismatchesandwaivedUnchainedRecordsalongside
ok, and its own doc comment says why (src/review/decision-record.ts:667-677): they are"counted and published separately, never folded into a clean result's silence, so '231 records were never
chained' can never read as 'nothing to see here'".
okstaystruefor all three.The proof page's dependency signature drops them (
src/review/proof-summary.ts:285and:351):so
buildProofLedgerStatus(:117-133) can only ever return{ state: "verified", tipSeq, totalCount }, andbuildProofBadgeMessage(:175-186) renders"verified · anchored". On an instance carrying theLOOPOVER_LEDGER_UNCHAINED_WAIVER/LOOPOVER_LEDGER_CONTENT_WAIVERdeclarations those parsers exist tosupport, the most public surface renders exactly the silence the internal verifier refuses to.
Neither defect is covered:
test/unit/proof-summary.test.ts:203-240seeds 3 decision records (below the20-decision floor), so the accuracy SQL never produces a published rate in any test, and every
verifyLedgerstub in that file returns only{ ok, tipSeq, totalCount }.Requirements
loadProofSummarymust stop derivingconfirmedfromreason_code.confirmedmust be the count ofmerge/close decisions for this repo that were NOT later reversed, where "reversed" means an
audit_eventsrow of typereversal_reverted,reversal_reopenedorreversal_supersededwhosetarget_keyis<repo_full_name>#<pull_number>for that decision.decidedstays the count ofaction IN ('merge','close')rows.section(...)wrapper (src/review/proof-summary.ts:296-303) so afailing read degrades that section rather than 503-ing the page, matching the existing per-section contract.
A failing reversal read must degrade toward
confirmed === decidednever being asserted: on that failurethe page must publish
{ state: "insufficient_data", decided, minimumDecisions }, not a rate.ProofLedgerStatus'sverifiedvariant must additionally carryprunedRecords: number,waivedContentMismatches: numberandwaivedUnchainedRecords: number, andbuildProofLedgerStatusmustpopulate them from the verify result.
verifyLedgerdependency signature in BOTHloadProofSummary'sdeps(:285) andProofPageDeps(
:351) must accept those three counts.buildProofBadgeMessagemust return"verified · N excluded"(N = the sum of the three counts) instead of"verified"/"verified · anchored"whenever that sum is greater than 0, andbuildProofBadgeColormust return the neutral"#9e9e9e"for that case. A sum of 0 must keep today'sexact strings and colors byte-identically.
PROOF_MIN_DECISIONS(20),PROOF_SAMPLE_RECORDS(5),round3,buildProofAccuracy's signature and itsinsufficient_datashape,buildProofAnchorStatus,PROOF_BOUNDARY_STATEMENT, the allowlist-by-name privacy discipline inbuildProofSummary, and theempty/broken/unavailableledger states.Deliverables
src/review/proof-summary.ts:confirmedis computed from thereversal_*audit_eventssignal, notfrom
reason_code. With 25 merge/closedecision_recordsrows foro/rand 2reversal_revertedrows whosetarget_keymatches two of them,loadProofSummary(env, "o/r", …)returns
accuracy: { state: "published", decided: 25, confirmed: 23, accuracy: 0.92, … }.src/review/proof-summary.ts:ProofLedgerStatus'sverifiedvariant carriesprunedRecords,waivedContentMismatchesandwaivedUnchainedRecords, populated bybuildProofLedgerStatus.src/review/proof-summary.ts:buildProofBadgeMessage(summary)returns"verified · 231 excluded"for a summary whose ledger is
verifiedwithwaivedUnchainedRecords: 231and the other two 0, andbuildProofBadgeColorreturns"#9e9e9e"for it.test/unit/proof-summary.test.tsasserting the 25-decisions/2-reversals case above yields apublished rate strictly below 1.
test/unit/proof-summary.test.tsasserting that 25 merge/close records with ZERO reversalrows still publishes
accuracy: 1— the perfect case must remain expressible, it just must no longerbe the only expressible one.
test/unit/proof-summary.test.tsasserting a failing reversal read degrades toaccuracy: { state: "insufficient_data" }rather than publishing a rate.test/unit/proof-summary.test.tsforbuildProofBadgeMessage/buildProofBadgeColorcovering both the zero-excluded (unchanged strings/colors) and nonzero-excluded cases.
test/unit/proof-summary.test.tsnamed for this bug (e.g."REGRESSION: the published accuracy is derived from real reversals, not from reason_code") that failsagainst the current
reason_code NOT LIKE 'reversal%'query.All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
fixing the accuracy query but leaving the ledger badge silent about waived/pruned rows, or adding the three
counts to the type without wiring them into the badge — does not resolve this issue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted.
vitest.config.ts'scoverage.includecovers
src/**/*.tsandpackages/loopover-engine/src/**/*.ts;src/review/proof-summary.tsis measured andgated. Every branch the change introduces or touches needs both arms tested: the new reversal-read
section()success and failure arms;buildProofAccuracy's!interval || decided < minimumDecisionsternary (both disjuncts, plus the passing arm);
buildProofLedgerStatus'sverifiedarm with and without anonzero exclusion count;
buildProofBadgeMessage's newexcluded > 0/excluded === 0split crossed withits existing
anchor.state === "anchored"split; andbuildProofBadgeColor's same split.Expected Outcome
The public proof page publishes an accuracy figure that can actually go down when a human overturns a bot
decision, and a repo whose ledger carries declared waivers or pruned preimages says so on its badge instead of
reading as unqualified "verified".
Links & Resources
src/review/proof-summary.ts:76-96—buildProofAccuracyand its "already-public precision block" claimsrc/review/proof-summary.ts:304-315— thereason_code NOT LIKE 'reversal%'querysrc/review/proof-summary.ts:38-48,:117-133,:175-198—ProofLedgerStatus,buildProofLedgerStatus,the badge message/color
src/review/decision-replay.ts:123-125/src/services/agent-action-executor.ts:329-331— the only tworeason_codederivationsmigrations/0179_decision_records.sql:16—reason_code TEXT NOT NULLsrc/review/decision-record.ts:656-680—verifyDecisionLedger's waived/pruned counts and their contractsrc/review/public-rule-precision.ts:88-136— the canonical reversal signalsrc/review/public-stats.ts:405-415— the same 100%-by-construction failure, already diagnosed elsewhere