Skip to content

orb(eval): the threshold-backtest writer never emits `corpusChecksum #9639

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

loadPublicRulePrecision (src/review/public-rule-precision.ts:51) exposes a latestBacktestRun block the
module header calls "the reproducibility hook (#8136's conclusion, operationalized)". It reads:

SELECT json_extract(metadata_json, '$.corpusChecksum') AS checksum, created_at FROM audit_events
 WHERE event_type IN ('calibration.threshold_backtest_run', 'calibration.logic_backtest_run')
   AND json_extract(metadata_json, '$.corpusChecksum') IS NOT NULL
 ORDER BY created_at DESC LIMIT 1

Two event types are in the IN list. Only one of them can ever satisfy the IS NOT NULL filter:

  • calibration.logic_backtest_run is written by buildLogicBacktestAuditInsertSql
    (scripts/backtest-logic-check-core.ts:173-190), whose metadata is
    { comparison, headSha, baseSha, corpusChecksum, replayableCount, skippedCount }. This writer runs only from
    the CI-side scripts/backtest-logic-check.ts CLI, and only with --persist.
  • calibration.threshold_backtest_run is written by persistThresholdBacktestRuns
    (src/services/threshold-backtest-run.ts:60-86), which runs inside the Worker on every review pass whose
    diff touches a KNOWN_THRESHOLDS constant. Its metadata is { comparison, constantName }no
    corpusChecksum, no head/base sha
    .

So an ORB deployment that runs the in-Worker threshold backtest but has never had the CI logic-backtest CLI
persist a row has latestBacktestRun: null forever. That is not a cosmetic gap:
buildEvalScoreRecordsFromRulePrecision (src/review/eval-score-records.ts:85-86) early-returns [] when
latestBacktestRun is null, so the entire /v1/public/eval-scores surface — the validator-consumable
EvalScoreRecord feed #9266 shipped and #9265 is specified to extend — serves zero records with no
diagnostic.

The two writers are explicitly siblings: buildLogicBacktestAuditInsertSql's own doc calls itself "a sibling
row to #8138's THRESHOLD_BACKTEST_EVENT_TYPE events" and says "corpusChecksum + the two shas are the freeze
point (#8136's reproducibility posture)". The threshold writer simply never gained the field, and the reader
was written as if it had.

persistThresholdBacktestRuns has everything it needs: runThresholdBacktestAdvisory
(src/services/threshold-backtest-run.ts:44-54) builds corpusByRuleId before calling
backtestChangedThreshold, and checksumCases (scripts/backtest-corpus-export-core.ts:23) is the
established canonicalized-sha256 helper the logic-backtest side already uses.

Requirements

  • checksumCases must become importable from the Worker runtime. Move it (and only it, with its
    canonicalization helper) into packages/loopover-engine/src/calibration/ and re-export it from the engine
    barrel, updating scripts/backtest-corpus-export-core.ts, scripts/backtest-logic-check.ts and
    scripts/attested-backtest-run.ts to import it from @loopover/engine. Its output must be byte-identical
    for the same input — an existing corpus manifest checksum must still validate.
  • runThresholdBacktestAdvisory must compute, per ruleId, the checksum of the corpus slice it actually
    backtested, and thread it through ThresholdBacktestRunResult so persistThresholdBacktestRuns can write it.
  • persistThresholdBacktestRuns must write corpusChecksum into the audit event's metadata for every
    persisted comparison, alongside the existing comparison and constantName keys.
  • Nothing about the existing comparison/constantName metadata or the THRESHOLD_BACKTEST_EVENT_TYPE
    string may change — src/services/rule-calibration-trend.ts:187-189 reads $.comparison.verdict off these
    same rows and must keep working.

⚠️ Required pattern: mirror buildLogicBacktestAuditInsertSql in scripts/backtest-logic-check-core.ts:173-190
— the same field name (corpusChecksum), the same checksumCases producer, the same position in
metadata_json. What does NOT satisfy this issue: inventing a second checksum function or a differently-named
field; removing calibration.threshold_backtest_run from the reader's IN list instead of fixing the writer;
making loadPublicRulePrecision fall back to a placeholder or synthetic checksum when none is found; or
duplicating checksumCases into src/ rather than moving it into the shared engine package.

Deliverables

  • checksumCases lives in packages/loopover-engine/src/calibration/, is exported from the engine barrel,
    and every existing scripts/ importer is updated; a test asserts its output is unchanged for a fixed
    fixture corpus.
  • persistThresholdBacktestRuns in src/services/threshold-backtest-run.ts writes
    metadata.corpusChecksum for each comparison, asserted by a new case in the threshold-backtest-run unit
    test file.
  • A test asserting loadPublicRulePrecision returns a non-null latestBacktestRun when the only persisted
    run is a calibration.threshold_backtest_run row — a test that fails against the current writer.
  • A test asserting buildEvalScoreRecordsFromRulePrecision emits one record per rule (not []) for that
    same fixture, so the /v1/public/eval-scores regression is pinned end to end.
  • metadata.comparison and metadata.constantName are still present and unchanged on the persisted event,
    asserted by an existing or new case.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example moving
checksumCases into the engine without wiring it into persistThresholdBacktestRuns — does not resolve this
issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. coverage.include covers both src/**/*.ts
and packages/loopover-engine/src/**/*.ts, so src/services/threshold-backtest-run.ts,
src/review/public-rule-precision.ts, src/review/eval-score-records.ts AND the moved checksumCases in
packages/loopover-engine/src/calibration/ are all measured and gated at the same 99%+ branch-counted bar —
the move does not exempt it, and every branch of the moved function needs coverage in its new home. Both arms
of every changed conditional need a test. The byte-stability test in Deliverable 1 and the
loadPublicRulePrecision regression test in Deliverable 3 are both mandatory.

Expected Outcome

An ORB deployment that has run a single in-Worker threshold backtest publishes a real latestBacktestRun
freeze point, and /v1/public/eval-scores emits EvalScoreRecords instead of an empty array — so the
validator-consumable feed reflects the deployment's actual backtest history rather than only CI-side logic runs.

Links & Resources

src/review/public-rule-precision.ts:88-105, src/services/threshold-backtest-run.ts:56-86,
src/review/eval-score-records.ts:85-118, scripts/backtest-logic-check-core.ts:164-200,
scripts/backtest-corpus-export-core.ts:23. Related: #9265, #9266, #9215.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions