Skip to content

fix(engine): three calibration composers silently collapse to 100%-objective-anchor on NaN/negative weight input #8643

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

packages/loopover-engine/src/pairwise-calibration.ts (lines 58-83, normalizeCompositeWeights)
has an isInvalidWeight() guard (line 58) that distinguishes two different caller situations:

  • Caller passed NaN/negative weight(s) by mistake → falls back to the documented default 50/50
    blend.
  • Caller explicitly zeroed every weight on purpose → returns real zeros, which the caller then uses
    for an objective-only fallback.

This distinction is asserted by a dedicated test
(packages/loopover-engine/test/pairwise-calibration.test.ts:146-155, "normalizes invalid weights
without producing NaN" → asserts {objectiveAnchor: 0.5, pairwiseJudge: 0.5}, not objective-only).

Three sibling composer files each have their own normalizeCompositeWeights-shaped function whose
comments literally claim to "converge with reviewer-consensus-calibration.ts / #6170" or "#7443" —
but each only replicates the explicit-all-zero handling, not the invalid-weight distinction that
pairwise-calibration.ts's own comment says was the actual point of #7443:

  • packages/loopover-engine/src/gate-verdict-calibration.ts:305-323 (total <= 0 branch, line 322)
  • packages/loopover-engine/src/reviewer-consensus-calibration.ts:389-406 (same shape, line 405)
  • packages/loopover-engine/src/finding-severity-calibration.ts:352-370 (same shape, line 369)

Concretely: calling computeGateVerdictCompositeCalibrationScore (or the reviewer-consensus /
finding-severity equivalents) with weights: { objectiveAnchor: Number.NaN, pairwiseJudge: -1, structuredGateVerdict: -1 }, while real pairwise/structured scores are available, silently produces
a 100%-objective-anchor composite instead of the documented default blend — exactly the outcome
pairwise-calibration.ts was fixed to avoid. None of the three files' test suites include the
NaN/negative-weight case (only the deliberately-all-zero case, e.g.
gate-verdict-calibration.test.ts:584-603).

Requirements

  • Add the same isInvalidWeight()-style distinction (invalid NaN/negative input → default blend;
    explicit all-zero input → real zeros) to all three files' normalizeCompositeWeights functions,
    mirroring pairwise-calibration.ts's existing implementation exactly.
  • Do not change behavior for any currently-tested case (explicit all-zero must still return real
    zeros in all three files, matching their existing passing tests).

Deliverables

  • gate-verdict-calibration.ts's normalizeCompositeWeights returns each file's documented
    default composite weights (not a 100%-objective-anchor collapse) when called with a NaN or
    negative weight alongside otherwise-valid weights.
  • reviewer-consensus-calibration.ts's equivalent function has the same fix.
  • finding-severity-calibration.ts's equivalent function has the same fix.
  • A new test in each of the three files' test suites, mirroring
    pairwise-calibration.test.ts:146-155 almost verbatim: call the composite function with one
    NaN weight and one negative weight (leaving real dependent scores present), and assert the
    resulting weights match that file's own DEFAULT_COMPOSITE_WEIGHTS, not an objective-only
    collapse.
  • All three files' existing "explicit all-zero" tests continue to pass unchanged.

All four Deliverables above are required in the same PR — a PR that fixes only one or two of the
three files does not resolve this issue.

Test Coverage Requirements

packages/loopover-engine/** is measured by codecov/patch (99%+ target, branch-counted). Each of
the three new tests must exercise the new invalid-weight branch specifically (not just re-run the
existing all-zero case), matching pairwise-calibration.test.ts's existing fixture pattern for the
same distinction.

Expected Outcome

All four calibration composers (pairwise-calibration.ts plus the three fixed here) handle
NaN/negative weight input identically: falling back to their own documented default blend, never
silently collapsing to a 100%-objective-anchor composite. This module is shared by both the miner
(AMS) and review (ORB) sides per each file's own header, so the fix removes a real double-blast-radius
bug.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions