Skip to content

Gold copied from a record proves the record was readable, not that the decision was recoverable - #827

Closed
MongLong0214 wants to merge 3 commits into
mainfrom
cdeb-v4-provenance
Closed

Gold copied from a record proves the record was readable, not that the decision was recoverable#827
MongLong0214 wants to merge 3 commits into
mainfrom
cdeb-v4-provenance

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

Stacked on #826.

The reviewer's evidence is the commit's prose with every CommitLore trailer and note removed — what the record was written from, not the record. If the ruling can only be recovered by reading the record, the candidate has no independent provenance and this study should not use it.

The mechanical half

Of 241 enumerated decisions, 207 have surviving ordinary prose and a resolvable scope:

repository audited pass excluded
agent-control-plane 35 35
agent-operator-score 59 58 1 no resolvable scope
gitseed 104 71 33 subject-and-record only
logic-pro-mcp 43 43

Gitseed's 33 are commits whose whole message is a subject line and a record. There is nothing to review behind them — a fact about the corpus, not a fault in the candidates.

A squashed commit hides whole records from the redaction

The leak check fired before a single reviewer ran. Two candidates' "redacted" source contained a complete recordRuled-out: line, reason, Record-Id: and all — and eleven carried at least one record line.

The cause is not a bug in the product's redaction. That redaction rebuilds the ordinary trailer tail from Git's own parse, which deliberately refuses to treat a Ruled-out: sentence in prose as a record. A squashed commit concatenates whole commit messages indented by four spaces, and Git does not see those as trailers either. Correct for the product; wrong for a blind evidence packet.

A second pass removes any CommitLore key line wherever it appears, with its folded continuations, and a separate check refuses a packet that still holds one. The bias is deliberately reversed here: removing a prose sentence that merely looks like a trailer costs a sentence, and leaving a record in costs the answer. 83 lines came out of 11 packets.

One candidate's prose still contains its ruling verbatim — logic-pro-mcp's "one pattern for both languages" is a noun phrase the body uses naturally. That is a G2 pass, not a leak, and it is reported rather than stripped.

Guards that exist because their absence is invisible

  • assertPacketsCarryNoAnchor refuses a packet containing the benchmark's own key. A reviewer who can see the anchor can tell which decisions are being watched.
  • assertRedactionDidWork refuses an audit in which no record-backed candidate had a trailer removed. A redaction that removed nothing produces a clean leak report for every candidate, and that report reads identically whether the redaction worked or never ran.
  • assertPacketHasNoRecordLines checks the result rather than trusting the second pass — the first pass looked clean too.

Verification

8 tests pass, both typechecks clean. Each guard is proved by arrival: it passes on real packets and throws once the thing it looks for is introduced.

…e decision was recoverable

So the reviewer's evidence is the commit's prose with every CommitLore trailer and note
removed -- what the record was written from, not the record. If the ruling can only be
recovered by reading the record, the candidate has no independent provenance and this
study should not use it.

The mechanical half of that runs here. Of 241 enumerated decisions, 207 have surviving
ordinary prose and a resolvable scope:

    agent-control-plane   35 audited   35 pass
    agent-operator-score  59           58 pass    1 no resolvable scope
    gitseed              104           71 pass   33 subject-and-record only
    logic-pro-mcp         43           43 pass

Gitseed's 33 are commits whose whole message is a subject line and a record. There is
nothing to review behind them, which is a fact about the corpus rather than a fault in the
candidates.

Two guards exist because their absence would be invisible. `assertPacketsCarryNoAnchor`
refuses a packet containing the benchmark's own key, since a reviewer who can see the
anchor can tell which decisions are being watched. `assertRedactionDidWork` refuses an
audit in which no record-backed candidate had a trailer removed -- a redaction that
removed nothing produces a clean leak report for every candidate, and the report reads
identically whether the redaction worked or never ran.

Packets are built once per repository rather than once per candidate. The existing
per-candidate builder re-materializes the bundle each call, and logic-pro-mcp's is 87MB.

Record-Id: r-v4provenance
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: using the rendered record as gold | it proves the record is readable, which is the thing under test, and any effect measured that way is the instrument measuring itself
Ruled-out: judging independent recoverability here | that is a reading of the prose, and a program that scored it would be encoding one reviewer's judgment as a mechanical fact
Ruled-out: calling buildSourcePacket per candidate | it re-materializes the sealed bundle each time, so 241 candidates would clone 87MB once per logic-pro-mcp decision
Limit: `Provenance: reconstructed` and `migrated` are the only benchmark-authored signals available mechanically. A record written during benchmark work but labelled `authored` is indistinguishable from an ordinary one here, and only the reviewers can catch it
Verified: 5 tests pass, both typechecks clean. Each guard is proved by arrival: the anchor check passes on real packets and throws once an anchor is appended to one, and the inertness check passes on a real audit and throws when the removal count is zeroed
CommitLore-Version: 2.0.0
…nd reviewer would have been handed the answer

The leak check on the Stage A packets fired before a single reviewer ran. Two candidates'
"redacted" ordinary source contained a complete CommitLore record -- `Ruled-out:` line,
reason, `Record-Id:` and all -- and eleven carried at least one record line.

The cause is not a bug in the product's redaction. That redaction rebuilds the ordinary
trailer tail from Git's own parse, which deliberately refuses to treat a `Ruled-out:`
sentence in prose as a record. A squashed commit concatenates whole commit messages
indented by four spaces, and Git does not see those as trailers either. Correct for the
product, wrong for a blind evidence packet.

So a second pass removes any CommitLore key line wherever it appears, with its folded
continuations, and a separate check refuses a packet that still holds one. The bias is
deliberately reversed here: removing a prose sentence that merely looks like a trailer
costs a sentence, and leaving a record in costs the answer to the question being asked.

83 lines came out of 11 packets. Every packet is now clean by the check that found the
problem.

One candidate's prose still contains its ruling verbatim -- logic-pro-mcp's "one pattern
for both languages" is a noun phrase the body uses naturally. That is a G2 pass, not a
leak, and it is reported rather than stripped.

Record-Id: r-v4redactionsecondpass
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: changing the product's redaction to match | it is right for the product, where a prose sentence that looks like a trailer must not be treated as a record, and the benchmark's opposite bias does not belong in shipped behaviour
Ruled-out: excluding the eleven affected candidates | the contamination is in the packet, not in the decision, and dropping them would lose real corpus to a fixable tooling gap
Ruled-out: stripping the one prose echo of a ruling | the body states the rejected approach in its own words, which is the independent support G2 is asking about
Limit: the key list is the vocabulary this repository uses. A record written under an extension key would survive both passes, and only the packet check would catch it -- by name, which is the same list
Verified: 8 tests pass, both typechecks clean. The finding arrived before the fix: the packet builder refused to emit Stage A batch 01, naming the candidate whose ruling was visible. After the second pass, zero packets contain a CommitLore key line, and the fixture reproducing the squashed-commit shape fails without it
CommitLore-Version: 2.0.0
@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 3 commits in origin/main..0e5cfa3aeaead360302b074c5fc1261206de743b
Active constraints: 2 limits · 6 ruled-out · 0 warnings — from 2 records over 3 changed paths

Active constraints for the paths this PR touches

Limits (2)

  • r-v4redactionsecondpass 9222978 — the key list is the vocabulary this repository uses. A record written under an extension key would survive both passes, and only the packet check would catch it -- by name, which is the same list
  • r-v4provenance 62f0ac4Provenance: reconstructed and migrated are the only benchmark-authored signals available mechanically. A record written during benchmark work but labelled authored is indistinguishable from an ordinary one here, and only the reviewers can catch it

Ruled out (6)

  • r-v4redactionsecondpass 9222978 — changing the product's redaction to match | it is right for the product, where a prose sentence that looks like a trailer must not be treated as a record, and the benchmark's opposite bias does not belong in shipped behaviour
  • r-v4redactionsecondpass 9222978 — excluding the eleven affected candidates | the contamination is in the packet, not in the decision, and dropping them would lose real corpus to a fixable tooling gap
  • r-v4redactionsecondpass 9222978 — stripping the one prose echo of a ruling | the body states the rejected approach in its own words, which is the independent support G2 is asking about
  • r-v4provenance 62f0ac4 — using the rendered record as gold | it proves the record is readable, which is the thing under test, and any effect measured that way is the instrument measuring itself
  • r-v4provenance 62f0ac4 — judging independent recoverability here | that is a reading of the prose, and a program that scored it would be encoding one reviewer's judgment as a mechanical fact
  • r-v4provenance 62f0ac4 — calling buildSourcePacket per candidate | it re-materializes the sealed bundle each time, so 241 candidates would clone 87MB once per logic-pro-mcp decision

git log --follow accepts exactly one pathspec, so renames are not followed for 3 paths; query one path at a time to follow its rename chain

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

@MongLong0214
MongLong0214 changed the base branch from cdeb-v4-census to main August 21, 2026 23:50
@MongLong0214

Copy link
Copy Markdown
Owner Author

Closing as already merged.

Every commit in this PR reached main through #829, whose branch contained this one. Verified per commit with git merge-base --is-ancestor: c3ada47, 62f0ac4, 9222978, d2ff392 and ffacc15 are all on main. The branch tip itself is not an ancestor only because gh pr update-branch added a merge commit carrying no content of its own, and the remaining diff against main is older formattings of files main now holds in newer form.

The stack was collapsed into one CI cycle because branch protection re-runs the full 3,500-test suite after every branch update, and merging four PRs in sequence would have meant four such cycles. The review record here stands.

@MongLong0214
MongLong0214 deleted the cdeb-v4-provenance branch August 29, 2026 02:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant