Shipping delivers a decision that has no identifier, which is the whole reason this study exists - #828
Merged
Merged
Conversation
…e decision was recoverable
So the reviewer's evidence is the commit's prose with every CommitLore trailer and note
removed -- what the record was written from, not the record. If the ruling can only be
recovered by reading the record, the candidate has no independent provenance and this
study should not use it.
The mechanical half of that runs here. Of 241 enumerated decisions, 207 have surviving
ordinary prose and a resolvable scope:
agent-control-plane 35 audited 35 pass
agent-operator-score 59 58 pass 1 no resolvable scope
gitseed 104 71 pass 33 subject-and-record only
logic-pro-mcp 43 43 pass
Gitseed's 33 are commits whose whole message is a subject line and a record. There is
nothing to review behind them, which is a fact about the corpus rather than a fault in the
candidates.
Two guards exist because their absence would be invisible. `assertPacketsCarryNoAnchor`
refuses a packet containing the benchmark's own key, since a reviewer who can see the
anchor can tell which decisions are being watched. `assertRedactionDidWork` refuses an
audit in which no record-backed candidate had a trailer removed -- a redaction that
removed nothing produces a clean leak report for every candidate, and the report reads
identically whether the redaction worked or never ran.
Packets are built once per repository rather than once per candidate. The existing
per-candidate builder re-materializes the bundle each call, and logic-pro-mcp's is 87MB.
Record-Id: r-v4provenance
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: using the rendered record as gold | it proves the record is readable, which is the thing under test, and any effect measured that way is the instrument measuring itself
Ruled-out: judging independent recoverability here | that is a reading of the prose, and a program that scored it would be encoding one reviewer's judgment as a mechanical fact
Ruled-out: calling buildSourcePacket per candidate | it re-materializes the sealed bundle each time, so 241 candidates would clone 87MB once per logic-pro-mcp decision
Limit: `Provenance: reconstructed` and `migrated` are the only benchmark-authored signals available mechanically. A record written during benchmark work but labelled `authored` is indistinguishable from an ordinary one here, and only the reviewers can catch it
Verified: 5 tests pass, both typechecks clean. Each guard is proved by arrival: the anchor check passes on real packets and throws once an anchor is appended to one, and the inertness check passes on a real audit and throws when the removal count is zeroed
CommitLore-Version: 2.0.0
…nd reviewer would have been handed the answer The leak check on the Stage A packets fired before a single reviewer ran. Two candidates' "redacted" ordinary source contained a complete CommitLore record -- `Ruled-out:` line, reason, `Record-Id:` and all -- and eleven carried at least one record line. The cause is not a bug in the product's redaction. That redaction rebuilds the ordinary trailer tail from Git's own parse, which deliberately refuses to treat a `Ruled-out:` sentence in prose as a record. A squashed commit concatenates whole commit messages indented by four spaces, and Git does not see those as trailers either. Correct for the product, wrong for a blind evidence packet. So a second pass removes any CommitLore key line wherever it appears, with its folded continuations, and a separate check refuses a packet that still holds one. The bias is deliberately reversed here: removing a prose sentence that merely looks like a trailer costs a sentence, and leaving a record in costs the answer to the question being asked. 83 lines came out of 11 packets. Every packet is now clean by the check that found the problem. One candidate's prose still contains its ruling verbatim -- logic-pro-mcp's "one pattern for both languages" is a noun phrase the body uses naturally. That is a G2 pass, not a leak, and it is reported rather than stripped. Record-Id: r-v4redactionsecondpass Provenance: authored Certainty: firm Blast: system Undo: easy Ruled-out: changing the product's redaction to match | it is right for the product, where a prose sentence that looks like a trailer must not be treated as a record, and the benchmark's opposite bias does not belong in shipped behaviour Ruled-out: excluding the eleven affected candidates | the contamination is in the packet, not in the decision, and dropping them would lose real corpus to a fixable tooling gap Ruled-out: stripping the one prose echo of a ruling | the body states the rejected approach in its own words, which is the independent support G2 is asking about Limit: the key list is the vocabulary this repository uses. A record written under an extension key would survive both passes, and only the packet check would catch it -- by name, which is the same list Verified: 8 tests pass, both typechecks clean. The finding arrived before the fix: the packet builder refused to emit Stage A batch 01, naming the candidate whose ruling was visible. After the second pass, zero packets contain a CommitLore key line, and the fixture reproducing the squashed-commit shape fails without it CommitLore-Version: 2.0.0
…le reason this study exists
The predecessor's delivery gate asked whether an expected `Record-Id` appeared in the bytes
the hook forwarded. Under the owner's estimand that is the wrong question, and it is the
question that emptied two of four repositories before any measurement.
So the probe drives the same surface -- `commitlore inject --hook-input` against a real
`PreToolUse` edit payload, at the frozen v1.2.0 dist, at the shipping default budget of 800
tokens -- and reads the forwarded bytes for content: the ruling, the reason, the path
scope, the lifecycle. Identity is recorded beside the result and never conditions it.
Across the 207 candidates with independent ordinary source:
delivered 154
carrying a Record-Id 69
carrying none 85
ruling visible 157
reason visible 157
scope correct 157
stale delivered as current 0
85 id-less decisions delivered is the finding. `logic-pro-mcp` has no `Record-Id` anywhere
in its history and its decisions still reach an agent about to edit the paths they cover.
Two things about the numbers are worth stating. The scope result is not vacuous: 71 of the
out-of-scope probes returned a non-empty payload, so the injector does forward other
records for an unrelated path -- it just never forwarded *this* candidate's ruling there.
And the first run of this probe reported 0 delivered of 207, which read exactly like a
finding. The extracted release tree had no installed dependencies and the CLI exited 1
before reading a record. Every content field is false in both cases, so the distinction is
now made by a check rather than by whoever reads the summary.
Record-Id: r-v4delivery
Provenance: authored
Certainty: firm
Blast: system
Undo: easy
Ruled-out: keeping the Record-Id visibility gate | it measures the instrument's own restriction, and under it 43 of logic-pro-mcp's 43 decisions fail for a field that did not exist when they were written
Ruled-out: a single substring match as the gate | a record sharing one phrase would pass, so the gate needs ruling, reason, scope and lifecycle together, with the out-of-scope probe as the part that can actually fail
Ruled-out: running the working-tree dist | it has drifted from v1.2.0, so it would measure behaviour nobody shipped
Limit: this shows the content can reach an agent that edits one of the decision's own paths at the shipping budget. It says nothing about whether an agent will edit those paths, use a tool the matcher covers, or act on what it reads -- those are the questions a measured run would ask
Verified: 6 tests pass including an end-to-end probe in which a record with no `Record-Id` at all has its ruling and reason delivered through the shipping hook. All 207 real probes exit 0. The injector guard is proved by arrival: it throws on the exact stderr the broken first run produced
CommitLore-Version: 2.0.0
CommitLore — record lintTrailers: clean — 1 commit in Active constraints for the paths this PR touchesLimits (2)
Ruled out (6)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
Owner
Author
|
Closing as already merged. Every commit in this PR reached The stack was collapsed into one CI cycle because branch protection re-runs the full 3,500-test suite after every branch update, and merging four PRs in sequence would have meant four such cycles. The review record here stands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #827.
The predecessor's delivery gate asked whether an expected
Record-Idappeared in the bytes the hook forwarded. Under the owner's estimand that is the wrong question — and it is the question that emptied two of four repositories before any measurement.This probe drives the same surface (
commitlore inject --hook-input, a realPreToolUseedit payload, the frozen v1.2.0 dist, the shipping default budget of 800 tokens) and reads the forwarded bytes for content: the ruling, the reason, the path scope, the lifecycle. Identity is recorded beside the result and never conditions it.Result, across the 207 candidates with independent ordinary source
85 id-less decisions delivered is the finding.
logic-pro-mcphas noRecord-Idanywhere in its history, and its decisions still reach an agent about to edit the paths they cover.Two things about the numbers
The scope result is not vacuous. 71 of the out-of-scope probes returned a non-empty payload, so the injector does forward other records for an unrelated path — it just never forwarded this candidate's ruling there.
And the first run reported 0 delivered of 207, which read exactly like a finding. The extracted release tree had no installed dependencies and the CLI exited 1 before reading a record. Every content field is false in both cases, so the distinction is now made by a check (
assertInjectorRan) rather than by whoever reads the summary.Verification
6 tests pass, including an end-to-end probe in which a record with no
Record-Idat all has its ruling and reason delivered through the shipping hook. All 207 real probes exit 0. The injector guard is proved by arrival — it throws on the exact stderr the broken first run produced.