⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.
Context
loadPublicRulePrecision (src/review/public-rule-precision.ts:51) exposes a latestBacktestRun block the
module header calls "the reproducibility hook (#8136's conclusion, operationalized)". It reads:
SELECT json_extract(metadata_json, '$.corpusChecksum') AS checksum, created_at FROM audit_events
WHERE event_type IN ('calibration.threshold_backtest_run', 'calibration.logic_backtest_run')
AND json_extract(metadata_json, '$.corpusChecksum') IS NOT NULL
ORDER BY created_at DESC LIMIT 1
Two event types are in the IN list. Only one of them can ever satisfy the IS NOT NULL filter:
calibration.logic_backtest_run is written by buildLogicBacktestAuditInsertSql
(scripts/backtest-logic-check-core.ts:173-190), whose metadata is
{ comparison, headSha, baseSha, corpusChecksum, replayableCount, skippedCount }. This writer runs only from
the CI-side scripts/backtest-logic-check.ts CLI, and only with --persist.
calibration.threshold_backtest_run is written by persistThresholdBacktestRuns
(src/services/threshold-backtest-run.ts:60-86), which runs inside the Worker on every review pass whose
diff touches a KNOWN_THRESHOLDS constant. Its metadata is { comparison, constantName } — no
corpusChecksum, no head/base sha.
So an ORB deployment that runs the in-Worker threshold backtest but has never had the CI logic-backtest CLI
persist a row has latestBacktestRun: null forever. That is not a cosmetic gap:
buildEvalScoreRecordsFromRulePrecision (src/review/eval-score-records.ts:85-86) early-returns [] when
latestBacktestRun is null, so the entire /v1/public/eval-scores surface — the validator-consumable
EvalScoreRecord feed #9266 shipped and #9265 is specified to extend — serves zero records with no
diagnostic.
The two writers are explicitly siblings: buildLogicBacktestAuditInsertSql's own doc calls itself "a sibling
row to #8138's THRESHOLD_BACKTEST_EVENT_TYPE events" and says "corpusChecksum + the two shas are the freeze
point (#8136's reproducibility posture)". The threshold writer simply never gained the field, and the reader
was written as if it had.
persistThresholdBacktestRuns has everything it needs: runThresholdBacktestAdvisory
(src/services/threshold-backtest-run.ts:44-54) builds corpusByRuleId before calling
backtestChangedThreshold, and checksumCases (scripts/backtest-corpus-export-core.ts:23) is the
established canonicalized-sha256 helper the logic-backtest side already uses.
Requirements
checksumCases must become importable from the Worker runtime. Move it (and only it, with its
canonicalization helper) into packages/loopover-engine/src/calibration/ and re-export it from the engine
barrel, updating scripts/backtest-corpus-export-core.ts, scripts/backtest-logic-check.ts and
scripts/attested-backtest-run.ts to import it from @loopover/engine. Its output must be byte-identical
for the same input — an existing corpus manifest checksum must still validate.
runThresholdBacktestAdvisory must compute, per ruleId, the checksum of the corpus slice it actually
backtested, and thread it through ThresholdBacktestRunResult so persistThresholdBacktestRuns can write it.
persistThresholdBacktestRuns must write corpusChecksum into the audit event's metadata for every
persisted comparison, alongside the existing comparison and constantName keys.
- Nothing about the existing
comparison/constantName metadata or the THRESHOLD_BACKTEST_EVENT_TYPE
string may change — src/services/rule-calibration-trend.ts:187-189 reads $.comparison.verdict off these
same rows and must keep working.
⚠️ Required pattern: mirror buildLogicBacktestAuditInsertSql in scripts/backtest-logic-check-core.ts:173-190
— the same field name (corpusChecksum), the same checksumCases producer, the same position in
metadata_json. What does NOT satisfy this issue: inventing a second checksum function or a differently-named
field; removing calibration.threshold_backtest_run from the reader's IN list instead of fixing the writer;
making loadPublicRulePrecision fall back to a placeholder or synthetic checksum when none is found; or
duplicating checksumCases into src/ rather than moving it into the shared engine package.
Deliverables
All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example moving
checksumCases into the engine without wiring it into persistThresholdBacktestRuns — does not resolve this
issue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted. coverage.include covers both src/**/*.ts
and packages/loopover-engine/src/**/*.ts, so src/services/threshold-backtest-run.ts,
src/review/public-rule-precision.ts, src/review/eval-score-records.ts AND the moved checksumCases in
packages/loopover-engine/src/calibration/ are all measured and gated at the same 99%+ branch-counted bar —
the move does not exempt it, and every branch of the moved function needs coverage in its new home. Both arms
of every changed conditional need a test. The byte-stability test in Deliverable 1 and the
loadPublicRulePrecision regression test in Deliverable 3 are both mandatory.
Expected Outcome
An ORB deployment that has run a single in-Worker threshold backtest publishes a real latestBacktestRun
freeze point, and /v1/public/eval-scores emits EvalScoreRecords instead of an empty array — so the
validator-consumable feed reflects the deployment's actual backtest history rather than only CI-side logic runs.
Links & Resources
src/review/public-rule-precision.ts:88-105, src/services/threshold-backtest-run.ts:56-86,
src/review/eval-score-records.ts:85-118, scripts/backtest-logic-check-core.ts:164-200,
scripts/backtest-corpus-export-core.ts:23. Related: #9265, #9266, #9215.
Context
loadPublicRulePrecision(src/review/public-rule-precision.ts:51) exposes alatestBacktestRunblock themodule header calls "the reproducibility hook (#8136's conclusion, operationalized)". It reads:
Two event types are in the
INlist. Only one of them can ever satisfy theIS NOT NULLfilter:calibration.logic_backtest_runis written bybuildLogicBacktestAuditInsertSql(
scripts/backtest-logic-check-core.ts:173-190), whose metadata is{ comparison, headSha, baseSha, corpusChecksum, replayableCount, skippedCount }. This writer runs only fromthe CI-side
scripts/backtest-logic-check.tsCLI, and only with--persist.calibration.threshold_backtest_runis written bypersistThresholdBacktestRuns(
src/services/threshold-backtest-run.ts:60-86), which runs inside the Worker on every review pass whosediff touches a
KNOWN_THRESHOLDSconstant. Its metadata is{ comparison, constantName }— nocorpusChecksum, no head/base sha.So an ORB deployment that runs the in-Worker threshold backtest but has never had the CI logic-backtest CLI
persist a row has
latestBacktestRun: nullforever. That is not a cosmetic gap:buildEvalScoreRecordsFromRulePrecision(src/review/eval-score-records.ts:85-86) early-returns[]whenlatestBacktestRunis null, so the entire/v1/public/eval-scoressurface — the validator-consumableEvalScoreRecordfeed #9266 shipped and #9265 is specified to extend — serves zero records with nodiagnostic.
The two writers are explicitly siblings:
buildLogicBacktestAuditInsertSql's own doc calls itself "a siblingrow to #8138's THRESHOLD_BACKTEST_EVENT_TYPE events" and says "
corpusChecksum+ the two shas are the freezepoint (#8136's reproducibility posture)". The threshold writer simply never gained the field, and the reader
was written as if it had.
persistThresholdBacktestRunshas everything it needs:runThresholdBacktestAdvisory(
src/services/threshold-backtest-run.ts:44-54) buildscorpusByRuleIdbefore callingbacktestChangedThreshold, andchecksumCases(scripts/backtest-corpus-export-core.ts:23) is theestablished canonicalized-sha256 helper the logic-backtest side already uses.
Requirements
checksumCasesmust become importable from the Worker runtime. Move it (and only it, with itscanonicalization helper) into
packages/loopover-engine/src/calibration/and re-export it from the enginebarrel, updating
scripts/backtest-corpus-export-core.ts,scripts/backtest-logic-check.tsandscripts/attested-backtest-run.tsto import it from@loopover/engine. Its output must be byte-identicalfor the same input — an existing corpus manifest checksum must still validate.
runThresholdBacktestAdvisorymust compute, per ruleId, the checksum of the corpus slice it actuallybacktested, and thread it through
ThresholdBacktestRunResultsopersistThresholdBacktestRunscan write it.persistThresholdBacktestRunsmust writecorpusChecksuminto the audit event's metadata for everypersisted comparison, alongside the existing
comparisonandconstantNamekeys.comparison/constantNamemetadata or theTHRESHOLD_BACKTEST_EVENT_TYPEstring may change —
src/services/rule-calibration-trend.ts:187-189reads$.comparison.verdictoff thesesame rows and must keep working.
Deliverables
checksumCaseslives inpackages/loopover-engine/src/calibration/, is exported from the engine barrel,and every existing
scripts/importer is updated; a test asserts its output is unchanged for a fixedfixture corpus.
persistThresholdBacktestRunsinsrc/services/threshold-backtest-run.tswritesmetadata.corpusChecksumfor each comparison, asserted by a new case in the threshold-backtest-run unittest file.
loadPublicRulePrecisionreturns a non-nulllatestBacktestRunwhen the only persistedrun is a
calibration.threshold_backtest_runrow — a test that fails against the current writer.buildEvalScoreRecordsFromRulePrecisionemits one record per rule (not[]) for thatsame fixture, so the
/v1/public/eval-scoresregression is pinned end to end.metadata.comparisonandmetadata.constantNameare still present and unchanged on the persisted event,asserted by an existing or new case.
All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example moving
checksumCasesinto the engine without wiring it intopersistThresholdBacktestRuns— does not resolve thisissue.
Test Coverage Requirements
This repo enforces 99%+ Codecov patch coverage, branch-counted.
coverage.includecovers bothsrc/**/*.tsand
packages/loopover-engine/src/**/*.ts, sosrc/services/threshold-backtest-run.ts,src/review/public-rule-precision.ts,src/review/eval-score-records.tsAND the movedchecksumCasesinpackages/loopover-engine/src/calibration/are all measured and gated at the same 99%+ branch-counted bar —the move does not exempt it, and every branch of the moved function needs coverage in its new home. Both arms
of every changed conditional need a test. The byte-stability test in Deliverable 1 and the
loadPublicRulePrecisionregression test in Deliverable 3 are both mandatory.Expected Outcome
An ORB deployment that has run a single in-Worker threshold backtest publishes a real
latestBacktestRunfreeze point, and
/v1/public/eval-scoresemitsEvalScoreRecords instead of an empty array — so thevalidator-consumable feed reflects the deployment's actual backtest history rather than only CI-side logic runs.
Links & Resources
src/review/public-rule-precision.ts:88-105,src/services/threshold-backtest-run.ts:56-86,src/review/eval-score-records.ts:85-118,scripts/backtest-logic-check-core.ts:164-200,scripts/backtest-corpus-export-core.ts:23. Related: #9265, #9266, #9215.