Measure how much of the active decision set reaches a fresh agent: 81.7% at the shipped budget - #374
Merged
Conversation
CommitLore — record lintTrailers: clean — 6 commits in Active constraints for the paths this PR touchesLimits (21)
Ruled out (34)
Warnings (12)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
Issue #343 asks how much of a repository's active decision set a fresh agent recovers before its first edit, and says why the question is answerable: the repository holds the records, so recall against them is countable, and a record that was retired and still surfaced is countable as an error. A metric written once the numbers are on screen is not a measurement, so this commit carries the method and nothing else -- no harness, no result, no figure. The next commit implements it and the one after that runs it. If a definition here turns out to be inconvenient when the numbers exist, the history says which came first. Two decisions are the ones worth arguing with. The primary denominator is the active records attached to the path being edited, not every active record in the repository: the question is what an agent has before *this* edit, and a repository-wide denominator would make every route's score a function of how large the repository has grown rather than of what bears on the change. The repository-wide figure is reported next to it anyway, so the choice can be checked instead of taken on trust. The second is that this is not ADR-0017's registered study, and must not be read as it. That protocol scores an agent's written brief against gold annotated from ordinary project evidence, and it rules out CommitLore records as gold for exactly the right reason: a brief scored against them would measure whether the tool can read its own encoding. This measures bytes delivered, which is a ceiling on what any agent could recover -- and for a ceiling the record set is the only correct denominator. The registered pilot stays unrun. Record-Id: r-delivmethod Limit: what a route hands over bounds recovery from above and is not recovery; nothing measured under this method can say an agent read or used a delivered record Ruled-out: Running ADR-0017's registered pilot instead | it needs two independent annotators and forty agent runs, while #343 asks a narrower question that this repository can already decide Ruled-out: Every active record in the repository as the primary denominator | it scores a path-scoped route against a quantity that grows with the repository rather than with the edit, so it is reported beside the primary rather than as it Ruled-out: Writing the method after the first run | a definition chosen once the numbers are visible cannot be told apart from one fitted to them Certainty: firm Blast: module Undo: easy Verified: this commit adds one document and no code; nothing is measured by it Unverified: whether this corpus exercises the expiry half of the error term, which only the run can report and this registration cannot promise
The measurement registered in bench/DECISION-DELIVERY.md needed a harness, and the shape it had to fit was already in the repository. The deterministic suite writes rows carrying harness_commit, harness_digest and dist_digest, refuses a dirty checkout, refuses mixed provenance, re-checks that neither HEAD nor dist moved during the run, and renders its own markdown beside the JSONL. So decision_delivery is a metric of that suite rather than a script beside it, and COMMITLORE_DETERMINISTIC_RECOVERY_ONLY=1 follows the DENSITY_ONLY escape hatch that already exists for running one metric against this repository's own history. It is also a required metric, so a complete suite run cannot quietly omit it. The answer key lives in its own module for one reason: it must not be able to reach the product. bench/deterministic/census.ts walks git log, parses trailers with Git's own interpret-trailers, reads changed paths from git show, and re-implements the SPEC 5 lifecycle fold; a test fails the build if an import from src/ or dist/ ever appears in it. Scoring a projection against a key that projection computed would be worth nothing. Where the two folds could disagree the row says so rather than hiding it: it carries the Supersedes: trailers the block walk resolved beside the count a raw line scan of the same messages found, and the number of records sitting on commits that changed no path. The arms reuse the ablation flags inject.ts already exposes -- noScope and noLifecycle together are the every-record arm ADR-0017 registered -- rather than a second projection written for the bench, which would measure the bench's idea of an unfiltered dump instead of the product's. One consequence outside the metric: a single-metric report no longer appends the economic case. That section cites another run's file, and a one-metric dataset carrying it reads as though it measured costs it never touched. Record-Id: r-delivharness Limit: gold attaches a record to the paths its commit changed and follows renames, which is also how the shipped projection decides scope, so a high commitlore recall is partly definitional and only the budget, grading and index resolution can move it below one Ruled-out: A standalone script beside bench/, in the shape of bench/retrieval/compare.ts | it would carry a source hash instead of the harness_commit and harness_digest ADR-0018 requires, and nothing would stop a full suite from omitting the metric Ruled-out: Deriving the answer key from runQuery or the record index | that scores the product's retrieval against the product's own census, which ADR-0017 rules out for the same reason Ruled-out: Resolving the rename chain only for paths with no direct attachment | the saving is one git call per path and the cost is dropping exactly the pre-rename records the measurement exists to count Ruled-out: Exempting the delivery run from the concurrent-bench refusal, as density is | r-densitymerge135 exempted seconds of git log, while this run spends minutes projecting every path and is exactly what would corrupt a wall-clock arm's percentiles Certainty: firm Blast: module Undo: easy Verified: test/decision-delivery.test.ts passes at 18, including the import ban that fails the build if census.ts ever reaches into src/ or dist/, and a fixture whose superseded record ordinary git log surfaces and the shipped route does not; deterministic-bench, density-denominator, economics and bench-provenance pass beside it at 66 total; src and bench typecheck clean and npm run build leaves dist byte-identical Unverified: whether the two lifecycle folds agree on this repository's corpus, which the run reports and this commit cannot
…te does The run is in. Of the 2,149 active (path, record) pairs this repository holds for files an agent could edit, the shipped path-scoped projection delivers 1,764 -- 82.1% -- and not one of the 1,900 records it hands over has been retired. That is the answer to #343, and it is not the best recall in the table. Ordinary `git log -- <path>` delivers 94.2%, twelve points more, and leaves no path empty-handed where the scoped route delivers nothing on 117 of 1,037. It pays for that with 75 retired records and 2.6 times the tokens. Recall is the side of that trade the product loses, and the comparator is reported first because a table showing only 82.1% would read as a win. The unfiltered dump is not a third option. At the shipped budget it recovers 2.3%: eight hundred tokens buys ten records for the entire repository and they are almost never the path's. Unbudgeted it reaches 92.1% for 83 million tokens and 7,259 retired records surfaced. The rows locate most of the missing 17.9 points without another run. The unbudgeted dump misses exactly 170 pairs, which is exactly the shipped route's withheld_records count: ten records in this repository grade `blocked`, so 92.1% is the ceiling for any injection route here and the trust grader sets it, not path scoping. The remaining ten points are the difference between a path-scoped projection and a repository-wide one, and no committed row splits them further, so the write-up does not guess. The last finding is about the question rather than the tool. Two independent implementations of "which records belong to this file" disagree on 215 pairs one way and 136 the other. Any recall figure inherits whichever definition its author picked, and this one says which. Record-Id: r-deliv821 Limit: one corpus, one repository, one query strategy per route; delivery bounds recovery from above and no agent was run Warn: zero retired records delivered is evidence about the supersede filter on seven records and no evidence at all about expiry, because this corpus contains none Ruled-out: Publishing only the path-scoped route's figure | the comparator is what makes 82.1% mean anything, and the route that beats it is the one a reader would otherwise assume was worse Ruled-out: Splitting the remaining ten points into budget and path scope | no committed row separates them, and a number derived from a second uncommitted script is exactly what this project refuses to publish Ruled-out: Rerunning with the report tightened first | the artifact and the renderer that produced it stay in step, which is worth more than removing one repeated sentence Certainty: firm Blast: local Undo: easy Verified: the run is bench/results/decision-delivery-20260801T051755Z.jsonl and its generated .md, ten rows sharing one harness commit, one dist digest and one machine descriptor; every figure in section 9 was read back out of that jsonl rather than transcribed; node scripts/check-readme-numbers.mjs reports the README block byte-identical at 39 lines and 2262 bytes; src and bench typecheck clean, dist rebuilds identical, and decision-delivery, deterministic-bench, readme-numbers and dogfood pass at 62 Unverified: how many of the ten blocked records are false positives, which this run counts but cannot adjudicate; and whether any agent would use what it was handed
The first run's headline read "ordinary git log recovers twelve points more than the shipped route". That comparison put an unbounded `git log` against an 800-token projection, so it moved two things at once and licensed a statement about neither. Section 9.3 of that run already said the residual could be the budget; the table was still published as though it were the mechanism. Understating the product on a measurement that does not support the claim is the same class of error as overstating it. Two arms fix it, and they are registered here before the run that reports them, for the same reason the metric was: an arm chosen after seeing what it does to the number is not a measurement. `commitlore-unbudgeted` is the path-scoped projection with the cap removed. Whatever it still misses is scoping or grading; the rest of the shortfall is the cap. `git-log-path-budgeted` is `git log` for the path cut to the same 800 tokens -- the comparison a real agent faces, because context is finite and an unbounded route is not one anybody can use. Which end the Git arm is cut from decides its number, so the section says. `git log` prints newest first and `buildInjection` drops its tail, so keeping a prefix cuts the same end the product cuts. The trailing partial line goes with it: a Record-Id line severed halfway was not delivered, and an anchored line scan would otherwise match it against the end of the string. Both original arms stay and are re-measured, so the two runs are readable against each other. The first run keeps its own section rather than being deleted; it measured the five arms it covered. Record-Id: r-delivpaired Limit: an arm added after a run cannot be presented as pre-registered, so it is dated and reasoned in the section it changes rather than folded into the original text Ruled-out: Leaving the confounded table and explaining it in prose | the explanation sat under a table whose top line already said the opposite, and a reader who stops at the table has been told the wrong thing Ruled-out: Dropping the unbudgeted arms and reporting only budgeted ones | the ceiling a mechanism reaches with unlimited context is the other half of the attribution, and one row cannot carry both Ruled-out: Cutting the budgeted git arm from the oldest end | git log prints newest first and the projection drops its tail, so cutting the head would score a route the product does not implement Certainty: firm Blast: module Undo: easy Verified: this commit changes one document and no code; the harness that implements these arms lands next, and the run after that Unverified: whether the shortfall is the cap or the scoping, which is the question the arms exist to answer and which no run has yet answered
Implements the arms registered in bench/DECISION-DELIVERY.md section 5. Nothing about the metric, the denominator, the error term or the scoring moves; the route list grows from five entries to seven and the budget column stops being decoration. The two git arms read one `git log`. Running the command twice would double the run for two views of the same bytes, and more importantly the budgeted arm has to be the same text its unbudgeted twin saw, or the pair is no longer a controlled comparison of one variable. `truncateToBudget` keeps a prefix, which cuts the oldest end because `git log` prints newest first and `buildInjection` drops its tail. It also discards the line the cut severed. Without that the anchored `Record-Id:` scan matches a half-line against the end of the string and scores a record that was never delivered -- a test pins both halves of that. The truncation assertion that guarded the unbudgeted every-record arm now guards both unbudgeted arms, because a silent cut on either would read as a property of the route rather than of a constant this file chose. Record-Id: r-delivarms Limit: the paired rows separate the cap from the mechanism only while both members read identical inputs, which is why the git pair shares one command's output rather than issuing two Ruled-out: A separate git invocation per arm | two invocations can disagree under a concurrent write, and the pair would then differ by more than the budget it exists to isolate Ruled-out: Counting a Record-Id line the budget cut in half | the anchored scan matches it against the end of the string, which would credit a record the route never handed over Ruled-out: Widening the truncation assertion into a warning | an unbudgeted arm that truncated is a wrong number, and a warning in a log nobody reads is how it would ship Certainty: firm Blast: module Undo: easy Verified: test/decision-delivery.test.ts passes at 21, including a truncation case that pins which end is cut and that the severed line is not scored, and a monotonicity check that no budgeted arm reaches past its unbudgeted twin in recovered records or tokens; deterministic-bench, density-denominator, economics, bench-provenance and dogfood pass beside it at 57; bench and src typecheck clean and dist rebuilds identical Unverified: what the added arms report on this repository, which the next commit measures
…ehind The paired arms are in, and they reverse the first run's conclusion. Against `git log` for the same path cut to the same 800 tokens, the shipped projection delivers 81.7% of this repository's 2,217 active (path, record) pairs where ordinary Git delivers 42.0%. It leaves 117 paths empty where Git leaves 183, it surfaces 0 retired records where Git surfaces 7, and it spends fewer tokens doing it -- 511,412 against 673,134 -- because raw commit prose carries subject lines and bodies the projection does not render. The first run reported the opposite because it compared an unbounded `git log` against a capped projection. That number is real and it is still in section 10; it is a statement about the cap, and it was published as though it were a statement about path scoping. Section 10 now says so at the top, because a reader who stops at its table would otherwise carry away the wrong conclusion. The two new arms also settle what the first run could only bound. Removing the cap takes the projection from 1,811 pairs to 2,047, so the cap costs 10.6 points. The repository-wide dump with its cap removed recovers 2,047 pairs as well -- the same number -- so path scoping costs nothing at all, and it reaches that with 741,429 tokens against 92,175,612 and 0 retired records against 7,322. Whatever holds the shipped route below 100%, the path filter is not it. What does hold it there is the trust grader. The unbudgeted projection misses exactly 170 pairs, which is exactly the withheld_records count on both CommitLore rows: ten records here grade `blocked`, so 92.3% is the ceiling for any injection route on this corpus. Unbudgeted against unbudgeted, ordinary Git is 46 pairs ahead precisely because it renders those, and pays 75 retired records for the privilege. docs/evidence.md exists on dev now, so the summary goes there beside the other measured results, with the method and the full tables staying in bench/ next to the run that produced them. Record-Id: r-delivpairedresult Limit: one corpus, one repository, one query strategy per route; this is delivery, which bounds recovery from above, and no agent was run Warn: the corpus holds zero expired records, so 0 retired records delivered is evidence about the supersede filter on seven records and none at all about expiry Ruled-out: Deleting the first run once it was superseded | it is a correct measurement of the five arms it covered, and a project that removes the run that embarrassed it cannot be checked Ruled-out: Correcting the first run's numbers in place | its rows were produced by a harness that had no budgeted git arm, and editing a conclusion under unchanged rows is how a result stops matching its evidence Ruled-out: Leading with the unbudgeted comparison, where git log is still ahead | no agent has unbounded context, so that row is a mechanism's ceiling rather than a route anyone can take Certainty: firm Blast: local Undo: easy Verified: the run is bench/results/decision-delivery-20260801T060225Z.jsonl and its generated .md, 14 rows sharing one harness commit, one dist digest and one machine descriptor; every figure in section 9 and in docs/evidence.md was read back out of that jsonl rather than transcribed; node scripts/check-readme-numbers.mjs reports the README block byte-identical at 39 lines and 2262 bytes; decision-delivery, deterministic-bench, readme, readme-numbers, readme-order, readme-positioning, help-text-honesty, economics and dogfood all pass; src and bench typecheck clean and dist rebuilds identical Unverified: how many of the ten blocked records are false positives, which this run counts but cannot adjudicate; and whether any agent uses what it was handed
MongLong0214
force-pushed
the
bench-issue-343
branch
from
August 1, 2026 06:30
2fb3151 to
e9332a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #343.
A measurement, not a feature. One question, one corpus, one number.
The number
81.7%. The shipped path-scoped projection delivers 1,811 of the 2,217 active
(path, record) pairs this repository holds for files an agent could edit, and
not one of the 1,947 records it hands over has been retired.
Every delivering family is measured twice — once at the shipped 800-token budget
and once with none — because a capped route compared against an uncapped one
measures the cap and the mechanism at the same time and cannot separate them
afterwards.
code-onlygit-log-path-budgetedgit-log-pathevery-record-budgetedevery-record-unbudgetedcommitlorecommitlore-unbudgetedAt the budget an agent actually has, the scoped route is 39.7 points ahead of
ordinary Git.
git-log-path-budgetedis the samegit logoutput cut to thesame 800 tokens: it recovers 42.0%, surfaces 7 retired records, leaves 183 of
1,046 paths with nothing against 117, and spends more tokens doing it, because
raw commit prose carries subjects and bodies the projection does not render.
An unbounded
git logrecovers 94.4%. That is real and it is not a routeanybody can take — 1.35M tokens across the population, and no agent has that
context. The unbudgeted rows are each mechanism's ceiling.
Budget or scope: the cap costs 10.6 points, scoping costs nothing
commitlore1,811 →commitlore-unbudgeted2,047. The 800-token cap costs236 pairs, 10.6 points.
commitlore-unbudgeted2,047 andevery-record-unbudgeted2,047 — the samenumber. Path scoping costs nothing. The path-scoped projection reaches
every pair the repository-wide dump reaches, with 741,429 tokens against
92,175,612 and 0 retired records against 7,322.
The grader sets the ceiling.
commitlore-unbudgetedmisses exactly 170pairs, which is exactly
withheld_recordson both CommitLore rows: ten recordshere grade
blocked, so 92.3% is the ceiling for any injection route on thiscorpus. Unbudgeted against unbudgeted, Git is ahead by 46 pairs precisely
because it renders those, and pays 75 retired records for it.
Corpus, so the run is reproducible
This repository at harness commit
b3f5692f7d1bd0af2f27b3f31b8ebac1e2dcb0e0,dist digest
37ffd480ee14…: 345 records over 549 commits — 338 active, 7superseded, 0 expired at HEAD's committer instant. 1,229 tracked paths, 160
declared generated by the repository's own
.gitattributes. Primary population:1,046 authored paths, 2,217 (path, active record) pairs, 164 reachable only
through a rename.
Raw output:
bench/results/decision-delivery-20260801T060225Z.{jsonl,md},14 rows.
Method, registered before each run
bench/DECISION-DELIVERY.mdwas committed on its own, with no harness and nofigures, in
b97f3a6— one commit before the code and two before the numbers.It fixes what counts as recovered, the denominator, what counts as an error, the
arms and the limits.
The two paired arms added after the first run are recorded as a dated
deviation in §5, committed in
a0ecca3before the harness that implementsthem and two commits before the run that reports them. The metric, denominator,
error term and scoring are unchanged; only arms were added.
The answer key is
bench/deterministic/census.ts:git log, Git's owninterpret-trailers,git show --name-only, and a second implementation of theSPEC §5 lifecycle fold. It imports nothing from
src/ordist/, and a testfails the build if it ever does. Its self-checks ship on the row: 9
Supersedes:trailers resolved by the block walk against 9 found by a raw linescan of the same messages.
Which end the budgeted Git arm is cut from decides its number, so §5 says:
git logprints newest first andbuildInjectiondrops its tail, so a prefix cutsthe same end the product cuts. The line severed by the cut is discarded — an
anchored
Record-Id:scan would otherwise match a half-line against the end ofthe string.
It fits the existing harness rather than sitting beside it
decision_deliveryis a metric of the deterministic suite, not a new script. Itinherits
harness_commit+harness_digest(ADR-0018),dist_digest, theclean-checkout refusal, the mixed-provenance refusal and the mid-run
product-changed check.
COMMITLORE_DETERMINISTIC_RECOVERY_ONLY=1follows theexisting
DENSITY_ONLYescape hatch, and the metric is inREQUIRED_METRICS, soa full suite run cannot omit it. Unlike density it keeps the concurrent-bench
refusal:
r-densitymerge135exempted seconds ofgit log, while this run spendsminutes projecting every path and would corrupt a wall-clock arm's percentiles.
The first run is kept, and labelled
bench/results/decision-delivery-20260801T051755Z.*stays committed. It is acorrect measurement of the five arms it covered, and its headline conclusion was
wrong: it compared an unbounded
git logagainst a capped projection and readthe difference as a cost of path scoping. §10 now says so above its own table,
and records that the rebase plus the harness change leave those rows
un-re-derivable under ADR-0018 — kept as a record of what was measured, not as a
re-derivable result.
Read it with its limits
repository's figures, and the corpus is the one whose maintainer wrote the tool.
recover; it says nothing about what one would. It is not ADR-0017's
registered study, does not discharge it, and that pilot remains unrun.
Zero retired records delivered is evidence about the supersede filter on seven
records and no evidence at all about expiry.
"which records belong to this file" differ, which is what holds the scoped
route's precision to 93.0%.
commitlore81.7% against 81.7%,
git-log-path-budgeted37.8% against 42.0%. Nothingreverses; the equal-budget gap widens.
Verification
npx vitest run— 79 files, 1,994 passed, 1 skipped (three shards on acontended machine: 610+1, 436, 948).
npm run typecheckandnpx tsc -p bench/tsconfig.json --noEmit— clean.npm run build—dist/rebuilds byte-identical.node scripts/check-readme-numbers.mjs— exit 0, README block byte-identicalat 39 lines / 2,262 bytes.
No README or translation is touched. The summary entry goes in
docs/evidence.md, which exists ondevafter the rebase; the method and fulltables stay in
bench/beside the run that produced them.