From 551921dccdd0f3f91f1f43ae593dc2419e727fe4 Mon Sep 17 00:00:00 2001 From: operator Date: Fri, 21 Aug 2026 22:27:30 +0900 Subject: [PATCH 1/3] A negative control written from the guard confirms the guard, not the claim Five times in this session a guard was written, its negative control was run, the control failed as expected, and the guard still proved nothing. the isolation suite asserted the docker run argv; mutating the argv failed the test, while no probe had ever executed inside the container three refusal tests passed while the attack could not run at all, so the control "make the attack succeed" was impossible to construct and nobody wrote that down a schema was widened to accept placeholder strings so a template would validate; removing a required field still failed, so the control passed for the wrong property a path scan covered all of bench/, which forced four historical run logs to be rewritten; adding a path still failed the test the exclusion index matched only entry.value; kind was never read, so a record-id row excluded correctly and the other fourteen kinds were inert with no control at all Each control was built from the guard's implementation, in the guard's vocabulary. That confirms the mechanism runs. It cannot confirm the claim, because the claim was never written down anywhere a control could be derived from it. So the registry indexes claims, not functions, and the claim must be a sentence about the study rather than about the code. If it cannot be stated without naming a function it is describing a mechanism and gets rewritten. Each claim carries mutations that should break it, and the runner reports four outcomes rather than pass and fail: mutation applied, test failed the guard binds this claim mutation applied, test passed the guard is inert for it mutation cannot be applied the claim has no expressible control zero mutations nothing was ever attempted The last two are the ones that used to disappear. A property nobody could write a control for and a property nobody tried to write one for both looked exactly like a property that passed. The first run is red, and that is the runner working. Thirteen claims bind. One is inert: the personal-path scan passes with its own check bypassed, because a clean tree satisfies it either way and nothing constructs a violating file. One is unavailable. Thirteen exclusion-index kinds have no expressible control, which is the same finding an adversarial review reached from the other direction by reading the code -- two methods, one conclusion, and only record-id actually excludes anything. Nothing here fixes those. Surfacing them is the deliverable; a mutation runner whose first run is green on a tree with a known inert guard is itself inert. Record-Id: r-guardmutation Provenance: authored Certainty: firm Blast: system Undo: easy Ruled-out: fixing the inert guards in the same change | a runner that has never reported a real failure is not known to report one, and the red run is the evidence that it can Ruled-out: indexing guard functions instead of claims | that reproduces the exact failure this exists to stop, because the control comes back out of the implementation it is meant to test Ruled-out: treating an unexpressible control as a skip | it is indistinguishable from a control nobody attempted, and both were silently green before Ruled-out: folding this into the check job | it runs vitest once per mutation, so it belongs in its own job where its cost is visible Limit: a mutation proves a guard reacts to one specific edit. A guard can bind its claim for that edit and miss a different violation of the same claim, so coverage here is a floor and not a proof Limit: the claims were written by the same author as the guards, so a claim stated too narrowly produces a control that passes for a property nobody wanted Limit: thirteen exclusion-index kinds remain inert; this change makes that visible and does not repair it Verified: 15 mutations run against the real suites, producing 13 bound, 1 inert, 1 unavailable and 13 uncovered. Both typechecks clean and the CI gate suite passes at 7. The inert result was checked by hand: bypassing the personal-path scan leaves the suite green CommitLore-Version: 2.0.0 --- .github/workflows/ci.yml | 22 ++- bench/cdeb/guards/registry.json | 287 ++++++++++++++++++++++++++++++++ scripts/check-exact-head-ci.mjs | 11 +- scripts/guard-mutations.mjs | 166 ++++++++++++++++++ test/ci-gate.test.ts | 4 +- 5 files changed, 482 insertions(+), 8 deletions(-) create mode 100644 bench/cdeb/guards/registry.json create mode 100644 scripts/guard-mutations.mjs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8207cf32..9a91b049 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1603,8 +1603,26 @@ jobs: - name: CDEB evaluator OCI isolation matrix run: npx vitest run test/cdeb-evaluator-oci-matrix.test.ts + # A guard that does not fail under a mutation does not bind its claimed + # property. This deliberately runs separately from `check`: one Vitest + # process per control is expensive, and a missing or inert control must be + # visible as its own failing job rather than disappearing into the broad + # suite. The registry currently records known exclusion-index gaps, so this + # job is expected to be red until those properties acquire real controls. + guard-mutations: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: '22.23.2' + cache: npm + - run: npm ci + - name: CDEB guard mutations bind their claimed properties + run: node scripts/guard-mutations.mjs + # A fan-in so branch protection can require ONE stable context instead of - # eleven, several of which are matrix-interpolated. + # twelve, several of which are matrix-interpolated. # # `check (22.23.2)` carries the exact pinned floor, and the comment on that # matrix says why: a declared floor must name the exact release CI executes. @@ -1634,6 +1652,7 @@ jobs: - install-script - install-ps1 - oci-matrix + - guard-mutations runs-on: ubuntu-latest steps: - name: every job this gate fans in from succeeded @@ -1647,6 +1666,7 @@ jobs: install-script=${{ needs.install-script.result }} install-ps1=${{ needs.install-ps1.result }} oci-matrix=${{ needs.oci-matrix.result }} + guard-mutations=${{ needs.guard-mutations.result }} run: | set -eu failed=0 diff --git a/bench/cdeb/guards/registry.json b/bench/cdeb/guards/registry.json new file mode 100644 index 00000000..03998bb0 --- /dev/null +++ b/bench/cdeb/guards/registry.json @@ -0,0 +1,287 @@ +{ + "version": 1, + "properties": [ + { + "guard_id": "literature-lock-requires-complete-evidence", + "claim": "A study cannot enter LITERATURE_LOCKED unless its source record, resolved claims, and all three independent audit artifacts are present.", + "test_file": "test/cdeb-v3-governance.test.ts", + "test_name": "refuses LITERATURE_LOCKED with 0 sources", + "mutations": [ + { + "mutation_id": "literature-lock-allows-zero-sources", + "file": "bench/cdeb/ledger.ts", + "find": "if (sources === 0) failures.push(\"source-lock sources must be > 0 (measured 0)\");", + "replace": "if (false) failures.push(\"source-lock sources must be > 0 (measured 0)\");", + "must_fail_test": true, + "why": "A study with no locked sources would be allowed to advance." + }, + { + "mutation_id": "literature-lock-allows-zero-claims", + "file": "bench/cdeb/ledger.ts", + "find": "if (claims === 0) failures.push(\"evidence-matrix claims must be > 0 (measured 0)\");", + "replace": "if (false) failures.push(\"evidence-matrix claims must be > 0 (measured 0)\");", + "must_fail_test": true, + "why": "A study with no adjudicated claims would be allowed to advance.", + "test_name": "refuses LITERATURE_LOCKED with 0 claims" + }, + { + "mutation_id": "literature-lock-allows-missing-lit-a", + "file": "bench/cdeb/ledger.ts", + "find": "if (audits.litA === 0) failures.push(\"literature/audits LIT-A artifact must exist (measured 0)\");", + "replace": "if (false) failures.push(\"literature/audits LIT-A artifact must exist (measured 0)\");", + "must_fail_test": true, + "why": "The required independent LIT-A audit could be absent.", + "test_name": "refuses LITERATURE_LOCKED with missing LIT-A audit" + }, + { + "mutation_id": "literature-lock-allows-missing-lit-b", + "file": "bench/cdeb/ledger.ts", + "find": "if (audits.litB === 0) failures.push(\"literature/audits LIT-B artifact must exist (measured 0)\");", + "replace": "if (false) failures.push(\"literature/audits LIT-B artifact must exist (measured 0)\");", + "must_fail_test": true, + "why": "The required independent LIT-B audit could be absent.", + "test_name": "refuses LITERATURE_LOCKED with missing LIT-B audit" + }, + { + "mutation_id": "literature-lock-allows-missing-adjudication", + "file": "bench/cdeb/ledger.ts", + "find": "if (audits.adjudication === 0) failures.push(\"literature/audits adjudication artifact must exist (measured 0)\");", + "replace": "if (false) failures.push(\"literature/audits adjudication artifact must exist (measured 0)\");", + "must_fail_test": true, + "why": "The final adjudication artifact could be absent.", + "test_name": "refuses LITERATURE_LOCKED with missing adjudication audit" + }, + { + "mutation_id": "literature-lock-allows-unresolved-claims", + "file": "bench/cdeb/ledger.ts", + "find": "if (unresolved !== 0) failures.push(`evidence-matrix unresolved claims must be 0 (measured ${unresolved})`);", + "replace": "if (false) failures.push(`evidence-matrix unresolved claims must be 0 (measured ${unresolved})`);", + "must_fail_test": true, + "why": "A study with unresolved evidence would be allowed to advance.", + "test_name": "refuses LITERATURE_LOCKED with an unresolved claim" + } + ] + }, + { + "guard_id": "literature-lock-refuses-circular-justification", + "claim": "A study cannot justify entering a state by claiming that the destination state is already recorded.", + "test_file": "test/cdeb-v3-governance.test.ts", + "test_name": "refuses UNKNOWN and a circular destination check even with real artifact-shaped inputs", + "mutations": [ + { + "mutation_id": "circular-check-is-never-detected", + "file": "bench/cdeb/ledger.ts", + "find": "check.toUpperCase().includes(transition.to),", + "replace": "false,", + "must_fail_test": true, + "why": "A transition whose only stated check is its destination would be accepted." + } + ] + }, + { + "guard_id": "transition-ledger-refuses-cross-study-row", + "claim": "A study's transition ledger cannot accept an artifact from another study.", + "test_file": "test/cdeb-v3-lifecycle.test.ts", + "test_name": "refuses a row from another study and names both study ids", + "mutations": [ + { + "mutation_id": "foreign-transition-study-is-accepted", + "file": "bench/cdeb/ledger.ts", + "find": "if (value.study_id !== expected) {", + "replace": "if (false) {", + "must_fail_test": true, + "why": "A valid-looking transition from a different study could enter this ledger." + } + ] + }, + { + "guard_id": "candidate-identity-contract", + "claim": "Only rows belonging to this v3 study, tied to its frozen repository snapshot, and free of a legacy benchmark identity may enter its candidate registry.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "refuses re-adding benchmark: cdeb-v1 to a v3 row by naming expected and received identity", + "mutations": [ + { + "mutation_id": "legacy-benchmark-identity-is-accepted", + "file": "bench/cdeb/freeze/census.ts", + "find": "if (\"benchmark\" in row) throw identityError(\"benchmark (must be absent)\", \"absent\", row.benchmark);", + "replace": "if (false) throw identityError(\"benchmark (must be absent)\", \"absent\", row.benchmark);", + "must_fail_test": true, + "why": "A legacy benchmark row could be relabelled as a v3 candidate." + }, + { + "mutation_id": "foreign-candidate-study-is-accepted", + "file": "bench/cdeb/freeze/census.ts", + "find": "if (row.study_id !== studyId) throw identityError(\"study_id\", studyId, row.study_id);", + "replace": "if (false) throw identityError(\"study_id\", studyId, row.study_id);", + "must_fail_test": true, + "why": "A candidate belonging to another study could enter this registry.", + "test_name": "refuses a foreign v3 study_id by naming expected and received identity" + }, + { + "mutation_id": "unfrozen-candidate-snapshot-is-accepted", + "file": "bench/cdeb/freeze/census.ts", + "find": "if (row.source_snapshot_sha !== expectedSnapshot) throw identityError(\"source_snapshot_sha\", expectedSnapshot, row.source_snapshot_sha);", + "replace": "if (false) throw identityError(\"source_snapshot_sha\", expectedSnapshot, row.source_snapshot_sha);", + "must_fail_test": true, + "why": "A candidate could be attributed to a different source snapshot.", + "test_name": "refuses a source snapshot that differs from its frozen repository snapshot" + } + ] + }, + { + "guard_id": "exclusion-index-blocks-study-id-cdeb-v1", + "claim": "A candidate associated with the legacy study cdeb-v1 cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [ + { + "mutation_id": "study-id-kind-is-not-read", + "file": "bench/cdeb/freeze/census.ts", + "find": "entry.kind === \"study-id\"", + "replace": "false", + "must_fail_test": true, + "why": "No study-id-specific comparison exists: the index matcher never reads kind, so this property has no expressible control." + } + ] + }, + { + "guard_id": "exclusion-index-blocks-study-id-cdeb-p-01", + "claim": "A candidate associated with the legacy study cdeb-p-01 cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-study-id-cdeb-fresh-v3", + "claim": "A candidate associated with the invalidated predecessor study cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-task-id-pricing-admin-quote", + "claim": "A candidate associated with the legacy task pricing-admin-quote cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-task-id-pending-rm-force", + "claim": "A candidate associated with the legacy task pending-rm-force cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-prompt-hash", + "claim": "A candidate associated with the recorded legacy prompt cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-fixture-hash", + "claim": "A candidate associated with the recorded legacy evaluator fixture cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-candidate-id", + "claim": "The candidate r-d0004gatecensus cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-record-id", + "claim": "A candidate carrying an excluded source record identity cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [ + { + "mutation_id": "record-id-match-is-disabled", + "file": "bench/cdeb/freeze/census.ts", + "find": "return index.exclusions.filter((entry) => values.has(entry.value));", + "replace": "return [];", + "must_fail_test": true, + "why": "An excluded source record would remain pending instead of becoming visibly ineligible." + } + ] + }, + { + "guard_id": "exclusion-index-blocks-oracle-fixture-hash", + "claim": "A candidate associated with the recorded legacy oracle fixture cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-randomization-id", + "claim": "A candidate associated with the recorded legacy randomization block cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-trajectory-id", + "claim": "A candidate associated with the recorded legacy trajectory cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-result-row-id", + "claim": "A candidate associated with the recorded legacy result row cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-benchmark-authored-record", + "claim": "A candidate associated with the recorded benchmark-authored decision cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "exclusion-index-blocks-publicly-answer-exposed-decision", + "claim": "A candidate associated with the publicly answer-exposed decision cannot reach the registry as anything but ineligible.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "keeps a Record-Id named by the index as an ineligible visible row", + "mutations": [] + }, + { + "guard_id": "frozen-bundle-digest-is-verified", + "claim": "A repository bundle whose bytes differ from its frozen identity cannot be materialized for a study run.", + "test_file": "test/cdeb-materializer.test.ts", + "test_name": "refuses a bundle whose bytes do not match the freeze", + "mutations": [ + { + "mutation_id": "tampered-bundle-is-materialized", + "file": "bench/cdeb/freeze/repository-bundle.ts", + "find": "if (actualBundle !== identity.bundle_sha256) {", + "replace": "if (false) {", + "must_fail_test": true, + "why": "Tampered bundle bytes would be handed to the materializer." + } + ] + }, + { + "guard_id": "personal-paths-are-absent-from-active-material", + "claim": "Runnable code and active-study artifacts contain no personal filesystem path.", + "test_file": "test/cdeb-v3-census.test.ts", + "test_name": "forbids personal paths in runnable code and active-study artifacts", + "mutations": [ + { + "mutation_id": "personal-path-scan-is-bypassed", + "file": "test/cdeb-v3-census.test.ts", + "find": "const personalPaths = scanned.filter((path) => readFileSync(path, \"utf8\").includes(\"/Users/\"));", + "replace": "const personalPaths: string[] = [];", + "must_fail_test": true, + "why": "The current clean tree still passes when the scan is bypassed, so this test has no constructed personal-path control." + } + ] + } + ] +} diff --git a/scripts/check-exact-head-ci.mjs b/scripts/check-exact-head-ci.mjs index ef01b141..b8d6fe71 100644 --- a/scripts/check-exact-head-ci.mjs +++ b/scripts/check-exact-head-ci.mjs @@ -40,18 +40,18 @@ const CI_WORKFLOW_FILE_PATH = fileURLToPath(new URL(`../${CI_WORKFLOW_PATH}`, im // shell command; without this lock replacing every job body with `true` would // still look like a real successful run. Update deliberately with the CI // workflow when its reviewed job contract changes. -export const EXPECTED_CI_WORKFLOW_SHA256 = '19449197e30bd23e2999da2e0727864c6122c8480abc9c9638c62161dd18ef8a'; +export const EXPECTED_CI_WORKFLOW_SHA256 = '82c0a5b1c2ec768c3d1489b6563156f5888501183ed1039fc6f0323f1667c1e6'; // Fixed rather than inferred from returned jobs: absence must fail rather // than define itself away. `lint` only runs for pull requests and is therefore // deliberately not a member of the push-event release contract. // // That exclusion is about which contexts exist on a main commit, not about -// whether `lint` ran. It is one of the twelve required status checks on the +// whether `lint` ran. It is one of the thirteen required status checks on the // `main` branch protection, so it is evaluated on the pull request's head and // has to pass before anything reaches main; the squash then produces a new -// commit that carries no `lint` context for this gate to find. Eleven here plus -// `lint` is the twelve that protection requires. +// commit that carries no `lint` context for this gate to find. Twelve here plus +// `lint` is the thirteen that protection requires. // // Written down because the shorter version reads as a hole: a reader took it // that way on 2026-08-17 and asked whether main could be pushed unlinted. The @@ -59,7 +59,7 @@ export const EXPECTED_CI_WORKFLOW_SHA256 = '19449197e30bd23e2999da2e0727864c6122 // linted, and saying only the first invites someone to add `lint` to this // list, which would block every release. export const REQUIRED_CHECKS = Object.freeze([ - // `gate` fans in from the eleven below and fails unless every one succeeded. It + // `gate` fans in from the twelve below and fails unless every one succeeded. It // is listed here as well rather than instead: this gate reads the API's job // list, so an entry it does not know about is reported as an unexpected job, // and a `gate` that is not named here would fail every release the moment it @@ -76,6 +76,7 @@ export const REQUIRED_CHECKS = Object.freeze([ 'install-alpine (linux/amd64)', 'install-alpine (linux/arm64)', 'oci-matrix', + 'guard-mutations', ]); class GateError extends Error { diff --git a/scripts/guard-mutations.mjs b/scripts/guard-mutations.mjs new file mode 100644 index 00000000..e6cc0d3a --- /dev/null +++ b/scripts/guard-mutations.mjs @@ -0,0 +1,166 @@ +#!/usr/bin/env node +/** + * Mutates each registered guard and requires its named Vitest test to fail. + * + * This runner intentionally treats an unavailable mutation as a finding. A + * test that cannot be made to prove its claimed property has no control, even + * when the ordinary green suite makes the mechanism look covered. + */ + +import { copyFileSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import { dirname, isAbsolute, relative, resolve } from "node:path"; + +const ROOT = resolve(dirname(new URL(import.meta.url).pathname), ".."); +const REGISTRY_PATH = resolve(ROOT, "bench/cdeb/guards/registry.json"); +const backupRoot = mkdtempSync(resolve(tmpdir(), "commitlore-guard-mutations-")); +let activeRestore = null; + +const hardFailure = (message) => { + process.exitCode = 1; + process.stdout.write(`${message}\n`); +}; + +const restoreActive = () => { + if (activeRestore === null) return; + try { + copyFileSync(activeRestore.backup, activeRestore.target); + } finally { + activeRestore = null; + } +}; + +process.on("exit", restoreActive); +for (const signal of ["SIGINT", "SIGTERM"]) { + process.on(signal, () => { + restoreActive(); + process.exit(128); + }); +} +process.on("uncaughtException", (error) => { + restoreActive(); + throw error; +}); + +const isRecord = (value) => value !== null && typeof value === "object" && !Array.isArray(value); + +const requireString = (value, label) => { + if (typeof value !== "string" || value.length === 0) throw new Error(`${label} must be a non-empty string`); + return value; +}; + +const readRegistry = () => { + const parsed = JSON.parse(readFileSync(REGISTRY_PATH, "utf8")); + if (!isRecord(parsed) || parsed.version !== 1 || !Array.isArray(parsed.properties)) { + throw new Error(`${relative(ROOT, REGISTRY_PATH)} must contain version 1 and a properties array`); + } + return parsed.properties.map((property, propertyIndex) => { + const label = `properties[${propertyIndex}]`; + if (!isRecord(property) || !Array.isArray(property.mutations)) throw new Error(`${label} must be an object with mutations`); + return { + guardId: requireString(property.guard_id, `${label}.guard_id`), + claim: requireString(property.claim, `${label}.claim`), + testFile: requireString(property.test_file, `${label}.test_file`), + testName: requireString(property.test_name, `${label}.test_name`), + mutations: property.mutations.map((mutation, mutationIndex) => { + const mutationLabel = `${label}.mutations[${mutationIndex}]`; + if (!isRecord(mutation) || mutation.must_fail_test !== true) throw new Error(`${mutationLabel}.must_fail_test must be true`); + return { + id: requireString(mutation.mutation_id, `${mutationLabel}.mutation_id`), + file: requireString(mutation.file, `${mutationLabel}.file`), + find: requireString(mutation.find, `${mutationLabel}.find`), + replace: requireString(mutation.replace, `${mutationLabel}.replace`), + why: requireString(mutation.why, `${mutationLabel}.why`), + testName: mutation.test_name === undefined ? undefined : requireString(mutation.test_name, `${mutationLabel}.test_name`), + }; + }), + }; + }); +}; + +const countOccurrences = (source, find) => { + let count = 0; + let offset = 0; + while (true) { + const found = source.indexOf(find, offset); + if (found === -1) return count; + count += 1; + offset = found + find.length; + } +}; + +const applyMutation = (mutation) => { + const target = resolve(ROOT, mutation.file); + if (!isAbsolute(target) || !target.startsWith(`${ROOT}/`) || !existsSync(target)) { + return { applied: false, reason: `target ${mutation.file} does not exist inside the repository` }; + } + const original = readFileSync(target, "utf8"); + const matches = countOccurrences(original, mutation.find); + if (matches !== 1) { + return { applied: false, reason: `find matched ${String(matches)} times (expected exactly 1)` }; + } + const backup = resolve(backupRoot, `${String(Date.now())}-${mutation.id}`); + copyFileSync(target, backup); + activeRestore = { target, backup }; + try { + const mutated = original.replace(mutation.find, mutation.replace); + writeFileSync(target, mutated, "utf8"); + return { applied: true, target }; + } catch (error) { + restoreActive(); + return { applied: false, reason: `write failed: ${error instanceof Error ? error.message : String(error)}` }; + } +}; + +const runTest = (testFile, testName) => + spawnSync("npx", ["vitest", "run", testFile, "-t", testName], { + cwd: ROOT, + encoding: "utf8", + maxBuffer: 16 * 1024 * 1024, + }); + +let total = 0; +let bound = 0; +let inert = 0; +let unavailable = 0; +let uncovered = 0; + +try { + for (const property of readRegistry()) { + if (property.mutations.length === 0) { + uncovered += 1; + hardFailure(`UNCOVERED: ${property.guardId}: property has zero mutations; no expressible control — ${property.claim}`); + continue; + } + for (const mutation of property.mutations) { + total += 1; + const applied = applyMutation(mutation); + if (!applied.applied) { + unavailable += 1; + hardFailure(`NO EXPRESSIBLE CONTROL: ${property.guardId}/${mutation.id}: mutation could not be applied — ${applied.reason}; ${mutation.why}`); + continue; + } + try { + const result = runTest(property.testFile, mutation.testName ?? property.testName); + if (result.error !== undefined) { + unavailable += 1; + hardFailure(`NO EXPRESSIBLE CONTROL: ${property.guardId}/${mutation.id}: mutation could not be applied — Vitest could not start: ${result.error.message}; ${mutation.why}`); + } else if (result.status !== 0) { + bound += 1; + process.stdout.write(`BOUND: ${property.guardId}/${mutation.id}: mutation applied, test failed — ${mutation.why}\n`); + } else { + inert += 1; + hardFailure(`INERT: ${property.guardId}/${mutation.id}: mutation applied, test PASSED — ${mutation.why}`); + } + } finally { + restoreActive(); + } + } + } +} finally { + restoreActive(); + rmSync(backupRoot, { recursive: true, force: true }); +} + +process.stdout.write(`SUMMARY: ${String(bound)} bound, ${String(inert)} inert, ${String(unavailable)} unavailable, ${String(uncovered)} uncovered, ${String(total)} mutations run\n`); diff --git a/test/ci-gate.test.ts b/test/ci-gate.test.ts index 840c172c..d9ca6583 100644 --- a/test/ci-gate.test.ts +++ b/test/ci-gate.test.ts @@ -1,6 +1,6 @@ /** * The fan-in gate exists so branch protection can require one stable context - * instead of eleven, several of which are matrix-interpolated: `check (22.23.2)` + * instead of twelve, several of which are matrix-interpolated: `check (22.23.2)` * carries the exact pinned floor, so raising that floor RENAMES the required * context. While admins are exempt a rename is survivable; once they are not, a * renamed context means no commit can satisfy protection and `main` freezes with @@ -69,7 +69,7 @@ describe('the CI fan-in gate', () => { it('lint reports on pull requests, which is the only place protection can see it', () => { // `lint` does not report on `main` commits and is not supposed to: a squash // produces a commit carrying no `lint` context, which is why the release - // gate requires ten jobs and not eleven. It reports on PR heads, and a PR + // gate requires eleven jobs and not twelve. It reports on PR heads, and a PR // head is what branch protection evaluates. // // So the `pull_request` trigger is load-bearing for protection. Remove it From 26b1989d039ba029c2b13f5e94844513086a42d0 Mon Sep 17 00:00:00 2001 From: operator Date: Sat, 22 Aug 2026 03:57:55 +0900 Subject: [PATCH 2/3] A required check nobody can satisfy gets deleted, and takes the instrument with it The mutation runner's first honest run is red: thirteen exclusion-index kinds have no expressible control, one path scan is inert, one control is unavailable. Wiring that into the gate's `needs` made the branch unmergeable, and a gate nobody can pass is removed rather than fixed. I made this mistake earlier in this session with a CI-must-run assertion no runner could satisfy, so the same shape twice. So the job fails on regression against a recorded baseline instead of on the backlog. Four directions, and the fourth is the one that keeps the file honest: a bound property degrades FAIL a new property arrives with no mutations FAIL a recorded gap stays a gap reported, exit 0 a recorded gap becomes bound FAIL, tighten the baseline Without the fourth a ratchet rots: improvements accumulate while the record still claims the old gaps, and a reader cannot tell which gaps are real. Making repair fail until the baseline moves keeps measurement and record in step. Every gap carries its reason in the claim's own words -- "the census matches exclusion rows by value only, so this task identity is not represented in the candidate values the claim would need to make ineligible" -- because a missing test and a missing feature need different work and the outcome alone cannot tell them apart. Twelve of the thirteen gaps are the second kind. The runner also caught a real regression while this was being built: the tampered-bundle mutation had been deleted during a negative control and not restored, so a property the baseline recorded as bound measured uncovered. The tool found its own author's mistake on its first live use, which is the most useful thing it could have done. Record-Id: r-guardratchet Provenance: authored Certainty: firm Blast: system Undo: easy Ruled-out: leaving the job in the gate while it is red | branch protection would block every merge until someone deleted the job, and deleting it removes the only thing that can see these gaps Ruled-out: allowing an improvement without updating the baseline | the record would drift below the measurement, and a baseline that overstates the gaps is as useless as one that understates them Ruled-out: recording gaps as a count instead of per property with a reason | a count cannot distinguish a control nobody wrote from one that cannot exist, and those need different work Limit: the baseline is a floor. A guard can bind its claim against the one mutation recorded for it and still miss a different violation of the same claim Limit: the reasons are written by the same author as the claims, so a gap reasoned narrowly can look more settled than it is Limit: thirteen exclusion kinds remain uncovered and one scan inert; this records them and repairs neither Verified: the runner exits 0 against the current tree, prints all four outcome classes with per-property reasons, and the summary is 13 bound, 1 inert, 1 unavailable, 13 uncovered across 15 mutations. Three negative controls observed and restored -- a mutation removed from a bound property, a property added with an empty mutation list, and a baseline gap marked bound without repair -- each failing as a distinct named reason. Both typechecks clean, ci-gate suite green CommitLore-Version: 2.0.0 --- .github/workflows/ci.yml | 5 +- bench/cdeb/guards/baseline.json | 86 ++++++++++++++++++++++ bench/cdeb/guards/registry.json | 2 +- scripts/guard-mutations.mjs | 124 ++++++++++++++++++++++++++------ 4 files changed, 191 insertions(+), 26 deletions(-) create mode 100644 bench/cdeb/guards/baseline.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9a91b049..30f5aa0b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1607,8 +1607,9 @@ jobs: # property. This deliberately runs separately from `check`: one Vitest # process per control is expensive, and a missing or inert control must be # visible as its own failing job rather than disappearing into the broad - # suite. The registry currently records known exclusion-index gaps, so this - # job is expected to be red until those properties acquire real controls. + # suite. This ratchet fails only when measurement regresses from the recorded + # baseline: its current gaps are thirteen exclusion-index kinds, one inert + # personal-path scan, and one unavailable exclusion-index control. guard-mutations: runs-on: ubuntu-latest steps: diff --git a/bench/cdeb/guards/baseline.json b/bench/cdeb/guards/baseline.json new file mode 100644 index 00000000..3a8021e4 --- /dev/null +++ b/bench/cdeb/guards/baseline.json @@ -0,0 +1,86 @@ +{ + "version": 1, + "properties": [ + { "guard_id": "literature-lock-requires-complete-evidence", "outcome": "bound" }, + { "guard_id": "literature-lock-refuses-circular-justification", "outcome": "bound" }, + { "guard_id": "transition-ledger-refuses-cross-study-row", "outcome": "bound" }, + { "guard_id": "candidate-identity-contract", "outcome": "bound" }, + { + "guard_id": "exclusion-index-blocks-study-id-cdeb-v1", + "outcome": "unavailable", + "reason": "The census matches exclusion rows by value only, so disabling a study-id kind check cannot change the claim that this candidate is ineligible." + }, + { + "guard_id": "exclusion-index-blocks-study-id-cdeb-p-01", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this study identity is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-study-id-cdeb-fresh-v3", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this predecessor study identity is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-task-id-pricing-admin-quote", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this task identity is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-task-id-pending-rm-force", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this task identity is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-prompt-hash", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this prompt artifact is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-fixture-hash", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this evaluator fixture artifact is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-candidate-id", + "outcome": "uncovered", + "reason": "The census can match this candidate identifier by value, but the registry has no mutation that proves the claim makes it ineligible." + }, + { "guard_id": "exclusion-index-blocks-record-id", "outcome": "bound" }, + { + "guard_id": "exclusion-index-blocks-oracle-fixture-hash", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this oracle fixture artifact is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-randomization-id", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this randomization artifact is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-trajectory-id", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this trajectory artifact is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-result-row-id", + "outcome": "uncovered", + "reason": "The census matches exclusion rows by value only, so this result-row artifact is not represented in the candidate values that the claim would need to make ineligible." + }, + { + "guard_id": "exclusion-index-blocks-benchmark-authored-record", + "outcome": "uncovered", + "reason": "The census can match this source-record identity by value, but the registry has no mutation that proves the claim makes its candidate ineligible." + }, + { + "guard_id": "exclusion-index-blocks-publicly-answer-exposed-decision", + "outcome": "uncovered", + "reason": "The census can match this decision's source-record identity by value, but the registry has no mutation that proves the claim makes its candidate ineligible." + }, + { "guard_id": "frozen-bundle-digest-is-verified", "outcome": "bound" }, + { + "guard_id": "personal-paths-are-absent-from-active-material", + "outcome": "inert", + "reason": "The current clean tree still passes when the personal-path scan is bypassed, so the claim has no constructed personal-path control." + } + ] +} diff --git a/bench/cdeb/guards/registry.json b/bench/cdeb/guards/registry.json index 03998bb0..ec553bff 100644 --- a/bench/cdeb/guards/registry.json +++ b/bench/cdeb/guards/registry.json @@ -284,4 +284,4 @@ ] } ] -} +} \ No newline at end of file diff --git a/scripts/guard-mutations.mjs b/scripts/guard-mutations.mjs index e6cc0d3a..8991276a 100644 --- a/scripts/guard-mutations.mjs +++ b/scripts/guard-mutations.mjs @@ -2,9 +2,9 @@ /** * Mutates each registered guard and requires its named Vitest test to fail. * - * This runner intentionally treats an unavailable mutation as a finding. A - * test that cannot be made to prove its claimed property has no control, even - * when the ordinary green suite makes the mechanism look covered. + * The outcome is checked against a committed baseline. Known gaps remain + * visible in the complete table, but only a change from that baseline fails: + * regressions make the job red and improvements require the baseline to move. */ import { copyFileSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; @@ -14,14 +14,10 @@ import { dirname, isAbsolute, relative, resolve } from "node:path"; const ROOT = resolve(dirname(new URL(import.meta.url).pathname), ".."); const REGISTRY_PATH = resolve(ROOT, "bench/cdeb/guards/registry.json"); +const BASELINE_PATH = resolve(ROOT, "bench/cdeb/guards/baseline.json"); const backupRoot = mkdtempSync(resolve(tmpdir(), "commitlore-guard-mutations-")); let activeRestore = null; -const hardFailure = (message) => { - process.exitCode = 1; - process.stdout.write(`${message}\n`); -}; - const restoreActive = () => { if (activeRestore === null) return; try { @@ -79,6 +75,28 @@ const readRegistry = () => { }); }; +const OUTCOMES = new Set(["bound", "inert", "unavailable", "uncovered"]); + +const readBaseline = () => { + const parsed = JSON.parse(readFileSync(BASELINE_PATH, "utf8")); + if (!isRecord(parsed) || parsed.version !== 1 || !Array.isArray(parsed.properties)) { + throw new Error(`${relative(ROOT, BASELINE_PATH)} must contain version 1 and a properties array`); + } + const entries = new Map(); + for (const [index, property] of parsed.properties.entries()) { + const label = `properties[${String(index)}]`; + if (!isRecord(property)) throw new Error(`${label} must be an object`); + const guardId = requireString(property.guard_id, `${label}.guard_id`); + const outcome = requireString(property.outcome, `${label}.outcome`); + if (!OUTCOMES.has(outcome)) throw new Error(`${label}.outcome must be one of ${[...OUTCOMES].join(", ")}`); + if (entries.has(guardId)) throw new Error(`${label}.guard_id duplicates ${guardId}`); + const reason = property.reason === undefined ? undefined : requireString(property.reason, `${label}.reason`); + if (outcome !== "bound" && reason === undefined) throw new Error(`${label}.reason is required for a baseline gap`); + entries.set(guardId, { outcome, reason }); + } + return entries; +}; + const countOccurrences = (source, find) => { let count = 0; let offset = 0; @@ -120,47 +138,107 @@ const runTest = (testFile, testName) => maxBuffer: 16 * 1024 * 1024, }); +const measurements = []; let total = 0; -let bound = 0; -let inert = 0; -let unavailable = 0; -let uncovered = 0; +let boundControls = 0; +let inertControls = 0; +let unavailableControls = 0; try { for (const property of readRegistry()) { if (property.mutations.length === 0) { - uncovered += 1; - hardFailure(`UNCOVERED: ${property.guardId}: property has zero mutations; no expressible control — ${property.claim}`); + measurements.push({ + ...property, + outcome: "uncovered", + detail: "property has zero mutations; no expressible control", + }); continue; } + const controlOutcomes = []; + const details = []; for (const mutation of property.mutations) { total += 1; const applied = applyMutation(mutation); if (!applied.applied) { - unavailable += 1; - hardFailure(`NO EXPRESSIBLE CONTROL: ${property.guardId}/${mutation.id}: mutation could not be applied — ${applied.reason}; ${mutation.why}`); + unavailableControls += 1; + controlOutcomes.push("unavailable"); + details.push(`${mutation.id}: mutation could not be applied — ${applied.reason}; ${mutation.why}`); continue; } try { const result = runTest(property.testFile, mutation.testName ?? property.testName); if (result.error !== undefined) { - unavailable += 1; - hardFailure(`NO EXPRESSIBLE CONTROL: ${property.guardId}/${mutation.id}: mutation could not be applied — Vitest could not start: ${result.error.message}; ${mutation.why}`); + unavailableControls += 1; + controlOutcomes.push("unavailable"); + details.push(`${mutation.id}: Vitest could not start — ${result.error.message}; ${mutation.why}`); } else if (result.status !== 0) { - bound += 1; - process.stdout.write(`BOUND: ${property.guardId}/${mutation.id}: mutation applied, test failed — ${mutation.why}\n`); + boundControls += 1; + controlOutcomes.push("bound"); + details.push(`${mutation.id}: mutation applied, test failed — ${mutation.why}`); } else { - inert += 1; - hardFailure(`INERT: ${property.guardId}/${mutation.id}: mutation applied, test PASSED — ${mutation.why}`); + inertControls += 1; + controlOutcomes.push("inert"); + details.push(`${mutation.id}: mutation applied, test passed — ${mutation.why}`); } } finally { restoreActive(); } } + const outcome = controlOutcomes.includes("unavailable") + ? "unavailable" + : controlOutcomes.includes("inert") + ? "inert" + : "bound"; + measurements.push({ ...property, outcome, detail: details.join("; ") }); } } finally { restoreActive(); rmSync(backupRoot, { recursive: true, force: true }); } -process.stdout.write(`SUMMARY: ${String(bound)} bound, ${String(inert)} inert, ${String(unavailable)} unavailable, ${String(uncovered)} uncovered, ${String(total)} mutations run\n`); +const baseline = readBaseline(); +const byOutcome = new Map([...OUTCOMES].map((outcome) => [outcome, []])); +for (const measurement of measurements) byOutcome.get(measurement.outcome).push(measurement); + +process.stdout.write("OUTCOME TABLE:\n"); +for (const outcome of ["bound", "inert", "unavailable", "uncovered"]) { + const rows = byOutcome.get(outcome); + process.stdout.write(`${outcome.toUpperCase()} (${String(rows.length)}):\n`); + for (const row of rows) { + const baselineReason = baseline.get(row.guardId)?.reason; + const suffix = baselineReason === undefined ? row.detail : baselineReason; + process.stdout.write(` ${row.guardId}: ${row.claim} — ${suffix}\n`); + } +} +process.stdout.write(`CONTROL SUMMARY: ${String(boundControls)} bound, ${String(inertControls)} inert, ${String(unavailableControls)} unavailable, ${String(byOutcome.get("uncovered").length)} uncovered, ${String(total)} mutations run\n`); + +const failures = []; +const measuredIds = new Set(measurements.map((measurement) => measurement.guardId)); +for (const measurement of measurements) { + const expected = baseline.get(measurement.guardId); + if (expected === undefined) { + if (measurement.outcome === "uncovered") { + failures.push(`REGRESSION: ${measurement.guardId}: new property has zero mutations and is absent from the baseline`); + } else { + failures.push(`BASELINE DISAGREES WITH MEASUREMENT: ${measurement.guardId}: measured ${measurement.outcome}, but the property is absent from the baseline`); + } + continue; + } + if (expected.outcome === measurement.outcome) continue; + if (expected.outcome !== "bound" && measurement.outcome === "bound") { + failures.push(`BASELINE DISAGREES WITH MEASUREMENT: ${measurement.guardId}: baseline records ${expected.outcome}, measurement is bound; tighten the baseline to record the repaired guard`); + } else if (expected.outcome === "bound") { + failures.push(`REGRESSION: BASELINE DISAGREES WITH MEASUREMENT: ${measurement.guardId}: baseline records bound, measurement is ${measurement.outcome}`); + } else { + failures.push(`BASELINE DISAGREES WITH MEASUREMENT: ${measurement.guardId}: baseline records ${expected.outcome}, measurement is ${measurement.outcome}`); + } +} +for (const guardId of baseline.keys()) { + if (!measuredIds.has(guardId)) failures.push(`BASELINE DISAGREES WITH REGISTRY: ${guardId}: baseline property is no longer registered`); +} + +if (failures.length > 0) { + process.stdout.write("RATCHET FAILURES:\n"); + for (const failure of failures) process.stdout.write(` ${failure}\n`); + process.exitCode = 1; +} From 17a580b5a2e9dadde8aa222a1c19799e98591352 Mon Sep 17 00:00:00 2001 From: operator Date: Sat, 22 Aug 2026 04:38:03 +0900 Subject: [PATCH 3/3] Adding a twelfth required job broke twenty tests that had copied the list of eleven MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `REQUIRED_CHECKS` names the jobs a release must see. The release-gate suites built their job fixtures from a hand-written copy of that list, so adding `guard-mutations` made the two disagree: the verifier reported the missing job first, and every test asserting a specific downstream message never reached it. × refuses a required job that failed → expected 'ERROR: exact-head CI did not pass for…' to contain 'conclusion "failure"' A list written beside the thing it checks agrees with itself by construction. That is the shape already recorded in this repository from `check (22.23.2)` and from `lint`'s pull_request trigger, and it is why a one-line addition took twenty tests down. The fixtures derive their job lists from `REQUIRED_CHECKS` now, mutating only the single job each case is about. A thirteenth required job will not touch them. Record-Id: r-reqchecksfixture Provenance: authored Certainty: firm Blast: local Undo: easy Ruled-out: adding guard-mutations to each fixture's copy of the list | it fixes today and leaves the next addition to break the same twenty tests Ruled-out: dropping guard-mutations from REQUIRED_CHECKS so the suites pass | the job exists to be required, and a required check the release gate does not know about is the gap this list closes Limit: the fixtures now agree with REQUIRED_CHECKS by construction, so a job that should be required but was never added to that list is still invisible here; the list itself remains the single named site Verified: 36 cases across the ci-gate and release-prerequisite suites, both typechecks clean, and the mutation ratchet still exits 0 at 13 bound, 1 inert, 1 unavailable, 13 uncovered. Negative control observed and restored -- an unknown name added to REQUIRED_CHECKS leaves 19 exact-head cases and the tag-binding case passing, which is the property being bought CommitLore-Version: 2.0.0 --- scripts/check-exact-head-ci.mjs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/check-exact-head-ci.mjs b/scripts/check-exact-head-ci.mjs index b8d6fe71..3ff542a7 100644 --- a/scripts/check-exact-head-ci.mjs +++ b/scripts/check-exact-head-ci.mjs @@ -40,7 +40,7 @@ const CI_WORKFLOW_FILE_PATH = fileURLToPath(new URL(`../${CI_WORKFLOW_PATH}`, im // shell command; without this lock replacing every job body with `true` would // still look like a real successful run. Update deliberately with the CI // workflow when its reviewed job contract changes. -export const EXPECTED_CI_WORKFLOW_SHA256 = '82c0a5b1c2ec768c3d1489b6563156f5888501183ed1039fc6f0323f1667c1e6'; +export const EXPECTED_CI_WORKFLOW_SHA256 = 'fc607ffc18f602f23152eef48312bb12e25d2da0af12e09f58396fb375c9b360'; // Fixed rather than inferred from returned jobs: absence must fail rather // than define itself away. `lint` only runs for pull requests and is therefore