chore(calculators): harden evidence governance contracts - #2601
chore(calculators): harden evidence governance contracts#2601BigSimmo wants to merge 22 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_41157a2c-2436-40f3-95af-1f946ae20d3a) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e92fd8934a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
CI triageCI failed on this PR. Automated classification of the 2 failed job(s):
Compared with main CI run #15862 (cancelled). Classification is evidence routing, not permission to ignore a failure. Exact quarantined Playwright identities remain governed by the flake ledger. |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_014d8f9c-970c-4e4d-88e4-3c40f2e5a758) |
Pull request was converted to draft
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_1c010363-7b19-4583-a6f2-0e7b0a30c93a) |
Address review feedback on PR #2601: the golden-vector registry test previously only counted vectors, so a registry entry with a null or mis-scored vector would still pass. Each vector now runs through deriveCalculator() (the same pure scoring/banding function every calculator mockup uses) and its expected score and band must match the derived result. Still red as intended — golden-vectors.json and the checker script don't exist yet on this branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ENQDEPFuwDZNoNssvV1PTc
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 104870fef8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The regex-only check on responseAnchorSetId validated the ID's shape but never compared it against the fixture's real response options, so an edited option label/point value or an arbitrary syntactically-valid replacement ID would still pass. Add a SHA-256 fingerprint derived from each fixture's item kinds, option labels and point values, and assert responseAnchorSetId matches it — so a modified instrument now fails the contract instead of silently passing. Addresses the "Bind anchor IDs to the response anchors" review thread on PR #2601. Test-only change; the governance checker/registry this suite depends on remain unimplemented by design and stay red. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ENQDEPFuwDZNoNssvV1PTc
…rdening tests tests/calculators-governance-hardening.test.ts existed but the governance data it checked for was never added. This PR was test-driven and unfinished. - Add rights metadata (holder, digital-use/modification/attribution permissions, verifiedAt) for every active calculator (PHQ-9, GAD-7, K10, CAGE, AUDIT-C), sourced from publicly documented rights-holder policies: Pfizer's stated "no permission required" policy for PHQ-9/GAD-7, the public-domain status of K10 and CAGE, and WHO's non-commercial-use policy for the AUDIT-derived AUDIT-C. Every value is something actually checked against a public source, not invented. - Pin a SHA-256 responseAnchorSetId fingerprint of each fixture's response anchors (item kind + option labels/points or checkbox points), matching the algorithm the test recomputes against live fixture data. - Add accessedAt/lastReviewed/nextReview/supersedes to every evidence source in data/calculators/evidence.json. - Add data/calculators/golden-vectors.json: verified answer -> score -> band vectors for every active calculator, checked against the real deriveCalculator scoring logic before being written down. - Add scripts/check-calculator-content.mjs (the fast static sibling of the vitest check) and wire it into check:calculator-content, verify:cheap:internal, and the CI static-pr job; update the two gate-count docs check:gate-manifest flagged. All 5 previously-failing assertions in tests/calculators-governance-hardening.test.ts now pass. Full verify:cheap (all 42 gates, lint, typecheck) and the full offline test suite (14969 passed) are green; the only 2 remaining test failures (clinical-hazard-controls.test.ts, privacy-readiness-contract.test.ts) are pre-existing commit-ancestry issues unrelated to this change, present before it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uRvqm44emrLDMyPF7Pame
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015uRvqm44emrLDMyPF7Pame
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_be33f698-cf13-4776-a47f-122d20b73b0e) |
…t just its wiring The wiring test only confirmed check-calculator-content.mjs is referenced by name in package.json, so an empty no-op script would still pass it. Run the real script against a copy of its content with one required field removed and confirm it actually reports failure. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_f7c73083-6578-4963-a142-9ea0153b3e9d) |
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_40b97162-7ad5-4300-a619-8f5790f12cca) |
Resolves a conflict in .github/workflows/ci.yml: keep both independently-added CI steps (Calculator content guard from this branch, Clinical hazard-controls register from main). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_719012a2-8788-4005-a299-c20f689412db) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_89b1fb8e-59cb-43ce-ae0c-d1f8aa32e46e) |
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_ff40a38d-ef11-4866-b6b2-f45b1e6ac9be) |
# Conflicts: # docs/scripts-index.md
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_c1e62f28-71cd-4f25-988c-d71e8e71074f) |
…ernance-hardening-local # Conflicts: # docs/scripts-index.md
…d test cleanup rmSync - ci-cache-safety.test.ts was red: check:calculator-content runs in CI's static-pr job under static_heavy_changed but was missing from verify:pr-local's local heavy-scope mirror list. - test-runner-safety.test.ts was red: the governance hardening test's temp-dir cleanup used an unbounded recursive fs.rmSync; give it the same maxRetries/retryDelay guard every other fixture cleanup uses. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019KtaqjrGBF8ZC9hKTb9Bri
Summary
Follow-up to merged PR #2491. Implements the calculator evidence-governance hardening
that
tests/calculators-governance-hardening.test.tsspecifies: pinned response-anchorfingerprints, rights metadata for every active calculator, evidence-source access/review
dates, a golden-vector score/band registry, and a
check:calculator-contentgate wiredinto
verify:cheap.check:calculator-contentgate wired intoverify:cheapScope
This PR is deliberately governance/metadata only. It does not change scoring algorithms, score bands, active catalogue membership, clinical recommendations, routes, database state, provider behaviour or deployment configuration. It does not activate 4AT, CIWA-Ar, COWS, BFCRS/BFCSI or EPDS.
Implementation status
The regression test was originally committed RED (it specified fields/files/scripts that did not yet exist on
main). Commits02a63f4band92ed0b59now add the missing rights metadata, evidence dates, golden-vector registry, and checker script;tests/calculators-governance-hardening.test.tsis green.Verification
npx vitest run tests/calculators-governance-hardening.test.ts— 5/5 passednpm run check:calculator-contentnpm run verify:cheap(all 42 gates + lint + typecheck + full offline unit suite — 14969 passed)Two offline test files fail on this branch for reasons unrelated to this change —
tests/clinical-hazard-controls.test.tsandtests/privacy-readiness-contract.test.tsboth fail a
reviewedCommit is not an ancestor of HEADcheck because the commit theirmanifest pins predates this branch's history (a pre-existing squash-merge artifact; that
commit is also not an ancestor of
origin/main, confirming it is unrelated to this PR).Safety
Clinical governance
The existing #2491 safety behavior remains authoritative: SAD PERSONS stays out of active decision support, Y-BOCS and MDQ remain quarantined, incomplete results fail closed, and psychometric interpretation remains separate from management guidance.
Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy)Note
Low Risk
Governance metadata, static checks, and regression tests only; clinical scoring paths and release gating for quarantined instruments are unchanged.
Overview
Adds calculator content governance so evidence, rights, response anchors, and golden scoring vectors cannot drift without failing local and CI gates. This is metadata and enforcement only—no changes to scoring logic, active catalogue, or clinical behaviour.
New
check:calculator-content(scripts/check-calculator-content.mjs) validatesevidence.jsonreview/access dates andsupersedes, the golden-vector registry shape and anchor-ID parity withcalculator-fixtures.ts, and verified rights fields for every active instrument. It is wired intoverify:cheapand CI Calculator content guard on static-heavy PRs.Data and fixture contracts expand: all evidence sources get
accessedAt,lastReviewed,nextReview, andsupersedes; fixtures gainresponseAnchorSetId(SHA-256 fingerprints of item anchors) and explicitCalculatorRights/rightsInfofor released tools; newdata/calculators/golden-vectors.jsonpins answer → score/band for PHQ-9, GAD-7, K10, CAGE, and AUDIT-C.tests/calculators-governance-hardening.test.tsrunsderiveCalculatoragainst every golden vector, binds anchor IDs to live items, and includes a temp-dir test that the checker exits non-zero on corrupted evidence (not just wiring).Docs bump the
verify:cheapstatic-gate count (39 gates inCLAUDE.md; gates skill references check 42 of 42).Reviewed by Cursor Bugbot for commit c27df55. Configure here.
Generated by Claude Code