Gold copied from a record proves the record was readable, not that the decision was recoverable - #827
Gold copied from a record proves the record was readable, not that the decision was recoverable#827MongLong0214 wants to merge 3 commits into
Conversation
…e decision was recoverable
So the reviewer's evidence is the commit's prose with every CommitLore trailer and note
removed -- what the record was written from, not the record. If the ruling can only be
recovered by reading the record, the candidate has no independent provenance and this
study should not use it.
The mechanical half of that runs here. Of 241 enumerated decisions, 207 have surviving
ordinary prose and a resolvable scope:
agent-control-plane 35 audited 35 pass
agent-operator-score 59 58 pass 1 no resolvable scope
gitseed 104 71 pass 33 subject-and-record only
logic-pro-mcp 43 43 pass
Gitseed's 33 are commits whose whole message is a subject line and a record. There is
nothing to review behind them, which is a fact about the corpus rather than a fault in the
candidates.
Two guards exist because their absence would be invisible. `assertPacketsCarryNoAnchor`
refuses a packet containing the benchmark's own key, since a reviewer who can see the
anchor can tell which decisions are being watched. `assertRedactionDidWork` refuses an
audit in which no record-backed candidate had a trailer removed -- a redaction that
removed nothing produces a clean leak report for every candidate, and the report reads
identically whether the redaction worked or never ran.
Packets are built once per repository rather than once per candidate. The existing
per-candidate builder re-materializes the bundle each call, and logic-pro-mcp's is 87MB.
Record-Id: r-v4provenance
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: using the rendered record as gold | it proves the record is readable, which is the thing under test, and any effect measured that way is the instrument measuring itself
Ruled-out: judging independent recoverability here | that is a reading of the prose, and a program that scored it would be encoding one reviewer's judgment as a mechanical fact
Ruled-out: calling buildSourcePacket per candidate | it re-materializes the sealed bundle each time, so 241 candidates would clone 87MB once per logic-pro-mcp decision
Limit: `Provenance: reconstructed` and `migrated` are the only benchmark-authored signals available mechanically. A record written during benchmark work but labelled `authored` is indistinguishable from an ordinary one here, and only the reviewers can catch it
Verified: 5 tests pass, both typechecks clean. Each guard is proved by arrival: the anchor check passes on real packets and throws once an anchor is appended to one, and the inertness check passes on a real audit and throws when the removal count is zeroed
CommitLore-Version: 2.0.0
…nd reviewer would have been handed the answer The leak check on the Stage A packets fired before a single reviewer ran. Two candidates' "redacted" ordinary source contained a complete CommitLore record -- `Ruled-out:` line, reason, `Record-Id:` and all -- and eleven carried at least one record line. The cause is not a bug in the product's redaction. That redaction rebuilds the ordinary trailer tail from Git's own parse, which deliberately refuses to treat a `Ruled-out:` sentence in prose as a record. A squashed commit concatenates whole commit messages indented by four spaces, and Git does not see those as trailers either. Correct for the product, wrong for a blind evidence packet. So a second pass removes any CommitLore key line wherever it appears, with its folded continuations, and a separate check refuses a packet that still holds one. The bias is deliberately reversed here: removing a prose sentence that merely looks like a trailer costs a sentence, and leaving a record in costs the answer to the question being asked. 83 lines came out of 11 packets. Every packet is now clean by the check that found the problem. One candidate's prose still contains its ruling verbatim -- logic-pro-mcp's "one pattern for both languages" is a noun phrase the body uses naturally. That is a G2 pass, not a leak, and it is reported rather than stripped. Record-Id: r-v4redactionsecondpass Provenance: authored Certainty: firm Blast: system Undo: easy Ruled-out: changing the product's redaction to match | it is right for the product, where a prose sentence that looks like a trailer must not be treated as a record, and the benchmark's opposite bias does not belong in shipped behaviour Ruled-out: excluding the eleven affected candidates | the contamination is in the packet, not in the decision, and dropping them would lose real corpus to a fixable tooling gap Ruled-out: stripping the one prose echo of a ruling | the body states the rejected approach in its own words, which is the independent support G2 is asking about Limit: the key list is the vocabulary this repository uses. A record written under an extension key would survive both passes, and only the packet check would catch it -- by name, which is the same list Verified: 8 tests pass, both typechecks clean. The finding arrived before the fix: the packet builder refused to emit Stage A batch 01, naming the candidate whose ruling was visible. After the second pass, zero packets contain a CommitLore key line, and the fixture reproducing the squashed-commit shape fails without it CommitLore-Version: 2.0.0
CommitLore — record lintTrailers: clean — 3 commits in Active constraints for the paths this PR touchesLimits (2)
Ruled out (6)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
|
Closing as already merged. Every commit in this PR reached The stack was collapsed into one CI cycle because branch protection re-runs the full 3,500-test suite after every branch update, and merging four PRs in sequence would have meant four such cycles. The review record here stands. |
Stacked on #826.
The reviewer's evidence is the commit's prose with every CommitLore trailer and note removed — what the record was written from, not the record. If the ruling can only be recovered by reading the record, the candidate has no independent provenance and this study should not use it.
The mechanical half
Of 241 enumerated decisions, 207 have surviving ordinary prose and a resolvable scope:
Gitseed's 33 are commits whose whole message is a subject line and a record. There is nothing to review behind them — a fact about the corpus, not a fault in the candidates.
A squashed commit hides whole records from the redaction
The leak check fired before a single reviewer ran. Two candidates' "redacted" source contained a complete record —
Ruled-out:line, reason,Record-Id:and all — and eleven carried at least one record line.The cause is not a bug in the product's redaction. That redaction rebuilds the ordinary trailer tail from Git's own parse, which deliberately refuses to treat a
Ruled-out:sentence in prose as a record. A squashed commit concatenates whole commit messages indented by four spaces, and Git does not see those as trailers either. Correct for the product; wrong for a blind evidence packet.A second pass removes any CommitLore key line wherever it appears, with its folded continuations, and a separate check refuses a packet that still holds one. The bias is deliberately reversed here: removing a prose sentence that merely looks like a trailer costs a sentence, and leaving a record in costs the answer. 83 lines came out of 11 packets.
One candidate's prose still contains its ruling verbatim —
logic-pro-mcp's "one pattern for both languages" is a noun phrase the body uses naturally. That is a G2 pass, not a leak, and it is reported rather than stripped.Guards that exist because their absence is invisible
assertPacketsCarryNoAnchorrefuses a packet containing the benchmark's own key. A reviewer who can see the anchor can tell which decisions are being watched.assertRedactionDidWorkrefuses an audit in which no record-backed candidate had a trailer removed. A redaction that removed nothing produces a clean leak report for every candidate, and that report reads identically whether the redaction worked or never ran.assertPacketHasNoRecordLineschecks the result rather than trusting the second pass — the first pass looked clean too.Verification
8 tests pass, both typechecks clean. Each guard is proved by arrival: it passes on real packets and throws once the thing it looks for is introduced.