Release 0.7.0 — the behaviour claim is measured - #482
Merged
Merged
Conversation
The README carried two numbers in 380 lines: the guard's 44.8% precision and 22.0% recall, and the 7.4 seconds to index 768 commits. The first of those is a disclosure about a secondary, experimental feature -- so the only performance figure a first-time reader met was the one saying the advisory mostly misses. The delivery measurement is the answer to the question this product exists for, and it lived only in bench/ and docs/evidence.md. Moving it up is not a new claim; every figure in the table is already in docs/evidence.md and derives from the committed run. The paragraph under it is the point of the change. A delivery number is a recall metric, and recall metrics have a known failure mode in exactly this setting: SWE-bench measured BM25 recall climbing from 29.58 to 51.06 across its context budgets and reported that performance drops anyway, because models are ineffective at localizing in a larger context. Publishing 81.7% without that sentence beside it would be doing what this market does. Publishing it with the sentence is the difference between a measurement and a marketing figure, and the citation is the field's founding benchmark rather than our own argument. The quote is taken from the v3 PDF, not from a summary of it. An earlier draft of this change carried "performance can still drop, as shown in Table 4"; the paper says "performance drops, as shown in Table 2". Record-Id: r-surfacedeliv Limit: every figure in the table is measured on this repository measuring itself, which is the weakest part of the evidence and is stated in the paragraph rather than left for a reader to discover Ruled-out: Leading with the token figures | 511,412 against 673,134 is the smaller half of the result, and a token-reduction headline is the claim this market makes without a harness behind it Ruled-out: Citing the resolve-rate numbers from the same table | they were not extracted from the PDF in this session, and the paper's own sentence carries the argument without them Certainty: firm Blast: local Undo: easy Verified: readme, readme-order, readme-numbers, readme-positioning and compatibility-matrix pass at 83 across all four files; check-readme-numbers.mjs exit 0 with the BENCH block byte-identical at 39 lines and 2262 bytes; spec/verify.sh OK at 26 fixtures; the recall figures 29.58 and 51.06 and the quoted sentence were read out of arXiv:2310.06770v3 directly Unverified: how the table reads to someone who has not already accepted that delivery and effect are different things
Put the delivery result where a reader meets it
M1 ran the right experiment at the wrong size: 5/30 against 7/30, p = 0.7480, and 5.1% power to detect the difference it observed. Its own verdict says what that means -- an experiment with a 5% chance of seeing its own effect has not tested the hypothesis. M5 changes n and nothing else. The size is registered here rather than chosen later, and so is the basis for it. A pilot run today measured a 33.3% control rate over three tasks, and this document deliberately does not use it: that sample included the strongest task in the set, and planning from it would size the run too small. The planning rate is M1's 23.3%, measured over all ten tasks. Where a convenient number and a conservative one disagree before a run, the run takes the conservative one. The two conditions that invalidated earlier matrices are written in as preconditions rather than hoped for. M4 recorded no per-run treatment exposure and was withdrawn entirely; M5 requires accepted_records > 0 on every treatment row and 0 on every control row, states what happens to a row that fails, and declares the run compromised rather than analysable past a 5% exclusion rate. M4 also records its model as "not recorded"; the harness now refuses a non-simulated run without --model, so that cannot recur. The stopping rule is the part worth reading. 1,160 runs is roughly 77 hours, and a long run invites a look partway through. All rows complete before any table is computed -- no interim analysis, no early stop, no extension after seeing a result. Sharding is for restartability and carries no analytical meaning. Record-Id: r-m5prereg Limit: ten synthetic tasks, one model and one harness; this bounds an effect on this instrument and is not evidence about coding agents in general Ruled-out: Planning from the pilot's 33.3% control rate | it covered three tasks including the strongest one, so it would have sized the run below what the full task set supports Ruled-out: Powering only a halving of the rate | 166 per arm fits in a fraction of the time, but a null at that size would not bound the difference M1 actually observed Ruled-out: Interim analysis with an early stop | the run is long enough to make peeking tempting, which is exactly why the rule is fixed before it starts Certainty: firm Blast: local Undo: easy Verified: n computed from the two-proportion normal approximation at alpha 0.05 two-sided and power 0.80, cross-checked against the power table in bench/VERDICT-M1.md; the pilot recorded guard_exposure complete on 12 of 12 rows with accepted_records non-zero on all six treatment rows and zero on all six control rows, so the exposure precondition is known to be satisfiable Unverified: whether the harness sustains a 77-hour run without drift; the pilot ran 48 minutes
Fix the M5 design before there is a number to fit it to
docs/evidence.md has carried break-even under "not measured" with a one-line reason: it would need a per-turn ledger of provider-reported usage and an observed cost for work spent on a rejected alternative. That sentence is about the break-even #138 withdrew, whose denominator was the value of a prevented re-proposal -- a behavioural quantity this project has never observed and had estimated from unrelated runs. The denominator does not have to be that. decision_delivery already measured the delivered-token difference between two information routes, on the same corpus, in the same unit. A ratio built on that has a measured numerator and a measured denominator, which is exactly what the withdrawn one did not. So this registers the method and the refusals first, the way bench/DECISION-DELIVERY.md was done, and section 9 is empty. What is registered matters more than what it will report: the write side counts only the two terms obtainable with no model call -- the generated harvest prompt's scaffold and the staged diff -- so it is a floor, and every break-even from it is a lower bound rather than an estimate. The two terms it omits are named on the result row itself, so a consumer of the JSONL cannot mistake the floor for a total. The five reduction pairs are fixed here rather than after the run. Two of them are unfavourable by construction and the fifth has no number at all: a reduction against a route that reads no history is a division by zero, and the honest report of that is "undefined", not a large percentage in either direction. The harness reads the read side from a committed delivery run instead of remeasuring it. Remeasuring would move the corpus -- the record count grows with the commits that add this file -- and leave one ratio citing two corpora for its two halves. Record-Id: r-tokenledger Limit: both sides are byte-derived token proxies under CHARS_PER_TOKEN=4, so the ratio cancels a uniform tokenizer error and does not cancel a differential one between prose and diffs Ruled-out: Remeasuring decision_delivery inside this harness | it would price the read side on a corpus that already includes the commits adding this measurement, and docs/evidence.md would cite two corpora for one ratio Ruled-out: Reporting a single write-side accounting | issue #138 established that a break-even which does not say what it counted is underspecified, so the diff term is reported both charged and free Ruled-out: Estimating the drafting turn from the 112 recorded runs | those rows carry a session total from --output-format json with no per-turn attribution, so any split across turns would be invented Certainty: firm Blast: module Undo: easy Verified: npx vitest run passes 2041 tests across 80 files; tsc -p tsconfig.json --noEmit and tsc -p bench/tsconfig.json --noEmit both exit 0; check-readme-numbers.mjs exit 0 with the BENCH block byte-identical at 39 lines and 2262 bytes; the 26 token-ledger tests cover the merge-borne and root-borne exclusions on a synthetic history and the network scan against a planted node:https import Unverified: whether the reconstructed diff matches the bytes `git diff --cached` printed on the day each record was written; the reconstruction rule is stated in the method document rather than checked
A trial run at HEAD was thrown away before it was committed. It priced 348 captures at HEAD while its read side came from a delivery run three records earlier, so the numerator and the denominator of every ratio described different repositories. The method document objects to precisely that on the other side of the same argument -- section 4 declines to remeasure the read side because doing so would leave one ratio citing two corpora -- and then section 3 walked into it from the other direction. The history is now the harness_commit the delivery run recorded, read off the same file the read side comes from. A commit that no longer resolves stops the run rather than falling back to HEAD: ADR-0018's digest fallback establishes that harness code is identical, which is not the same as having the history back, and a ledger measured over a different history would still populate every field and still look plausible. The deviation is recorded in the section it changes, dated, with its reason, before the run that reports the figures -- not folded into the original text. Record-Id: r-ledgerpin Limit: the ancestry check proves this checkout continues the history the read side was measured on; it does not prove nothing between the two commits changed how a diff reconstructs Ruled-out: Falling back to HEAD when the recorded commit is unresolvable | it turns a loud provenance failure into a quiet corpus mismatch, which is the class of defect ADR-0018 exists to keep loud Ruled-out: Keeping the HEAD run beside a pinned one as a sensitivity arm | it was never committed or quoted, so keeping it adds a second table a reader has to reconcile for a 3-record difference Certainty: firm Blast: module Undo: easy Verified: 28 token-ledger tests pass, including one asserting history_ref equals read_source.harness_commit and one asserting an unresolvable commit is refused; tsc -p bench/tsconfig.json --noEmit exits 0 Unverified: behaviour when the recorded commit resolves but is on an unrelated history -- the ancestry check refuses it, and no test constructs that repository
docs/evidence.md has carried break-even under "not measured" since the figure #138 withdrew. What was missing there was a denominator: the old ratio divided a measured cost by the value of a prevented re-proposal, and this project has never observed one. The measurement runs because decision_delivery supplied a different denominator -- a measured difference in delivered tokens between two routes, on a corpus this repository already published. The write side is a floor and is quoted as one. Two of its four terms are obtainable with no model call: the generated harvest prompt's scaffold at 1,197 tokens, and each commit's staged diff, which takes a median capture to 3,537 tokens and the mean to 10,064. The mean is 3.8x the median because a few commits staged generated files, so the mean is the wrong number to carry away and the distribution is printed instead of it. Verification's zero stops being an assertion: 14 built modules reachable from the verify entry points were scanned for a network client and none carried one. The corpus check is the part worth trusting. 343 single-parent captures plus 3 merge-borne records is 346, which is exactly the record_bearing_commits the delivery run's own census counted at the same commit -- two independent walks of one history agreeing. It also falsified something this document asserted: three merges do carry a record, all of them "Merge dev into <branch>" integrations documenting a conflict resolution. The count was reported so the assumption would be checked, and it was wrong. The finding is not the one a product would want. Against a naive whole-repository dump the records pay for themselves after 39 reads, but nobody has 92 million tokens of context, so that is a real number about an unreal baseline. Against the comparator an agent actually faces -- git log on the path, truncated to the same 800 tokens -- break-even is at least 22,326 path-scoped reads, twenty-one full passes over every evaluated file. That is not a plausible amount of editing. So on this corpus the token argument is the weak one and recall is what carries the product: 81.7% against 42.0% at the same budget. And against an agent that reads no history at all there is no break-even at any read count, because a comparator that spends nothing offers nothing to amortize against. That row is in the table rather than omitted from it, and its reduction column reads undefined rather than a large negative percentage. The reduction figures carry their denominators on the same line for the same reason. The one worth quoting is 99.2%: the same projection with the budget removed on both sides, recovering the same count of gold pairs, at 708.8 tokens per read against 88,122.0. Equal in count -- the delivery row records counts and not sets, and that caveat travels with the number. What is still missing is named instead of estimated. The tokens a model spends drafting a record need a model call, and the driver could not attribute one to that turn anyway: it reads a single session-total usage object out of --output-format json. A per-turn ledger plus a bench arm that runs capture against each run's own transcript is what would close it. Record-Id: r-ledgerresult Limit: both sides are byte-derived proxies under CHARS_PER_TOKEN=4 rather than a provider tokenizer, so the ratio cancels a uniform error and not a differential one between diff text and prose Limit: break-even in reads assumes reads land on the evaluation set the way the delivery run's per-path average describes, and real editing concentrates on a few files Ruled-out: Leading with the 99.4% reduction against the whole-repository dump | it is the largest number in the table and its denominator is a route no agent can run, which is the figure this market publishes Ruled-out: Omitting the code-only row because it has no percentage | a reduction table that drops the case where the product is the expensive option is the thing the method document was written to avoid Ruled-out: Publishing a single write-side accounting | #138 established that a break-even which does not say what it counted is underspecified, so the diff term is reported both charged in full and charged at nothing Certainty: firm Blast: local Undo: easy Verified: the run is bench/results/token-ledger-20260801T122953Z.jsonl at harness commit 8665be3, harness digest 9c91253e4cb06a0c07076620ecdad01ebf7c13f9, dist digest f54cda4795ccc1083e00aa38d8637a2e6f22466ef20213fa7d127c7fd301d1d2, over history b3f5692; 343+3=346 agrees with the delivery census; npx vitest run passes; tsc -p tsconfig.json --noEmit and tsc -p bench/tsconfig.json --noEmit exit 0; spec/verify.sh OK at 26 fixtures; check-readme-numbers.mjs exit 0 with the BENCH block byte-identical at 39 lines and 2262 bytes; validate --range over the adoption range exits 0 Unverified: whether the reconstructed per-commit diff matches the bytes git diff --cached printed on the day each record was written, and whether the two routes recovering 2,047 pairs each recovered the same 2,047 -- the delivery row records counts, not sets
Measure the token ledger: what a record costs to write, what the projection saves to read
The first M5 launch omitted --task, whose default is every fixture in bench/tasks/. That directory holds the ten reproposal-* cases this measurement registers and ten qualification-gitseed-* cases that qualify the harness and test no hypothesis. The run was producing all twenty, so it did not match §3's "over all ten tasks" and would have taken twice as long to say so. It was caught 3.6 hours in at 80 rows, during the §6 precondition check, and restarted with the ten fixtures named explicitly. The shard script now passes them and says in a comment why, because the failure here was a default asserting itself through an omission rather than anything anyone decided. The 80 rows are committed rather than deleted. A restart with no artifact behind it is indistinguishable from a run that always looked this way, and this project has spent a lot of credibility on the difference. They are marked not citable and excluded from the analysis set. Worth stating explicitly because it is the thing that makes a restart legitimate or not: no outcome was examined first. What was read is what §6 and §7 require to be monitored -- exposure completeness, arm-correct accepted_records, stopped_by balance, model recorded. The 2x2 table was not computed, and the cause of the restart has nothing to do with results. Those rows do carry one useful fact, which is why the deviation note quotes it: the preconditions that invalidated M4 held perfectly at 0.0% exclusion against a 5% threshold, so the design is satisfiable and the restart risks nothing but time. Record-Id: r-m5dev1 Limit: the deviation note reports integrity figures from the discarded rows, which is defensible only because none of them is the registered outcome; a reader who disagrees should discount them entirely Ruled-out: Letting the 20-task run finish | it produces the registered rows correctly but takes 4.3 days instead of 2.2, and leaves a permanent question about whether the extra fixtures mattered Ruled-out: Deleting the off-design rows | the artifact is the evidence that the restart happened for the stated reason Certainty: firm Blast: local Undo: easy Verified: the restarted shard script passes the ten reproposal fixtures explicitly and its syntax was checked before launch; readme-numbers passes and check-readme-numbers.mjs exits 0 with the BENCH block byte-identical Unverified: whether any other runner default is silently in force in this measurement -- only --task was audited
The deviation note admitted only --task had been audited, which is not a state to leave a measurement in. The rest are checked now: --tasks resolves to bench/tasks/ correctly, --max-turns is deliberately unset so each fixture's own budget applies as it did in M1, and --timeout-ms and --permission-mode take the same defaults M1 ran under. No second default is silently in force. The audit did surface one number. On the registered fixtures, over-turns ran at 22.5% in the off-design sample against 15.0% in M1. A run cut off at its turn budget had less chance to re-propose, so a false on such a row is weaker evidence than a false on a completed one. The temptation is to exclude those rows. That is exactly the choice this document exists to stop: the rule is visible now only because the rate is, and M1 kept them. §7 stands unchanged. What is added is an obligation on the verdict -- report the over-turns share per arm beside the table -- registered while the outcome is still unknown, which is the only time such a requirement is worth anything. Record-Id: r-m5overturns Limit: the 22.5% comes from 40 discarded rows, so it is an early indication of the rate rather than a measurement of it Ruled-out: Excluding over-turns rows from the analysis set | the rule would be changed after seeing the rate, and M1 kept them, so the two runs would stop being comparable on the one axis M5 was built to vary Ruled-out: Raising the turn budget mid-run | it changes the instrument between shards and makes the earlier rows a different measurement Certainty: firm Blast: local Undo: easy Verified: every --option in bench/runner.ts compared against the shard script's command line; over-turns share computed from t702-m1-final.jsonl (15.0% of 60 reproposal rows) and the off-design sample (22.5% of 40) Unverified: whether the over-turns share on the registered run matches the off-design sample, which is what the verdict will report
The first screen carried three different taglines. The image said "Git remembers what changed. CommitLore remembers why.", the heading under it said "Your agents inherit the code. Make them inherit the judgment.", and the alt text said a coding agent must not revive a decision the repository already reversed. The image was left behind by a positioning change that moved the heading twice and never reopened the SVG. The rebuild keeps what was already right: the split composition, the warm paper palette, and real material from this repository rather than decoration. What changes is the headline, and what the card on the right shows. It used to show one ruled-out alternative. It now shows an active record beside a superseded one that is withheld, because the lifecycle filter is the part of this product no comparable tool has, and the old card demonstrated the part several of them do. The demo SVG gained a `<title>` and a `<desc>`, and the generator gained the reason. `aria-label` on the root serves the README's `img` embed and nothing else; a reader who opens the file directly had no name for it. Regenerated rather than hand-edited, since the file says not to edit it by hand. The bottom caption was clipped on the first attempt -- it read "hook con" at the canvas edge -- which a rendered preview caught and a well-formedness check would not have. Shortened until it fits. Record-Id: r-heroconsist Limit: the card text is 19-20 units, so at a 360-pixel mobile render it falls below the legible threshold; the headline and the alt text carry the message there, and the same content is in the Markdown below, but the card itself is decoration at that width Ruled-out: Converting the animated demo to a GIF | the skill that prompted this review states GitHub does not play animation inside SVG, and GitHub was observed serving this file byte-identical with all five SMIL elements intact, so a GIF would cost the text layer and the file size for nothing Ruled-out: Turning the delivery table into a chart | tables belong in Markdown where they stay selectable and diffable, and the numbers would have left the text layer Certainty: firm Blast: local Undo: easy Verified: rendered at 1200 and inspected -- nothing clipped, active and superseded rows both legible; both SVGs parse; readme, readme-order, readme-numbers, readme-positioning, compatibility-matrix, demo and manifest pass at 114; check-readme-numbers.mjs exit 0 with the BENCH block byte-identical; spec/verify.sh OK at 26 fixtures; the skill's own audit_readme.py reports no issues Unverified: how the hero reads in GitHub's dark theme -- the palette is a fixed light paper and was not re-checked against a dark background
Record the M5 deviation instead of quietly restarting
Make the hero image say what the heading below it says
…s number Every figure this project has is CommitLore measuring CommitLore. A competitive review named that as the largest evidence weakness and it is a real one, not a presentational one. bench/DECISION-DELIVERY.md §7 already says so in its first bullet, but declaring a weakness is not measuring past it. This registers what can and cannot be carried to a repository nobody here wrote, and it does so before the harness exists, so the commit order is checkable the same way DECISION-DELIVERY.md's was. The split it fixes is the whole design. The `git log` baseline needs no records at all — it is a fact about how deep a file's history is and how long its commit messages are — so whether 42.0% is a property of `git log` or a property of our repository is answerable outright. The delivery figure needs records, and no external repository has any, so records have to be manufactured from ground truth the project declared itself: revert commits, selected by the machine- written `This reverts commit <sha>` line rather than by a `Revert "…"` subject, because that line names the commit and makes a record derivable instead of guessed. Six filters and a return check are fixed here rather than after seeing which ones flatter the number, including the 0.5 threshold and the two alternative thresholds the run has to report beside it. The document also states, in the words the result section will have to repeat, that a backfilled figure is an upper bound on a process no user has run. Every generated record is correct by construction because the filter kept only reverts whose change is provably still gone. That is an oracle, and an oracle is not evidence anyone would record anything. Limit: reverts are the one decision class an arbitrary repository declares mechanically — constraints, warnings and quietly-abandoned alternatives leave no revert and are invisible to this corpus Limit: four repositories, all Python, all large and long-lived; nothing here will speak to a small or young repository or to another ecosystem's commit habits Ruled-out: rewriting the external histories to plant trailers in commit messages | it changes every downstream sha, destroys the pinned-SHA corpus statement, and src/core/backfill.ts already refuses it for the same reason Ruled-out: reporting only the plain `git log` arms on the backfilled corpus | they score 0% there by construction because the records are in notes, and a 0 presented as a fact about Git is the most dishonest number this could produce Ruled-out: sampling paths on the large repositories | the sweep is one `git log` per path and a deterministic sample still invites the question the full sweep closes Warn: do not read §4's coverage figure as a recall figure — it equals one only if records are one per commit and evenly spread, which is false in detail here, which is why the calibration row at b3f5692 exists Verified: the four pinned SHAs were read from `git rev-parse HEAD` on full clones taken today, and the harness will refuse to measure any other tree Unverified: no number exists yet — the harness this document specifies has not been written, and §9 is empty on purpose Evidence: bench/EXTERNAL-CORPUS.md Blast: local Undo: easy Certainty: firm Record-Id: r-e1a7c4 Provenance: authored CommitLore-Version: 2.0.0
Implements bench/EXTERNAL-CORPUS.md, committed one commit earlier and unchanged since. Three measurements: the record-free part of the `git log` baseline over every tracked path of five corpora, the deterministic revert-to-record backfill, and the registered delivery metric run on what the backfill produced. No number yet — §9 of the method document is still empty. The delivery metric is not reimplemented. `measureDecisionDelivery` scores the external corpus, so a figure here is comparable to the 81.7% row because it came out of the same scorer, the same denominator and the same answer key. That forced two changes to a registered harness, and both are additive with the registered behaviour as the default, so this repository's own rows do not move: `buildCensus` gains an opt-in that folds `refs/notes/commitlore` into the answer key, and `measureDecisionDelivery` gains a route list that defaults to the seven arms §5 of DECISION-DELIVERY.md registered. The notes-aware Git arms are the awkward part and they are not optional. Backfill cannot rewrite an external history — it would change every sha and delete the pinned-SHA corpus statement the whole measurement rests on — so the records land in the notes mirror, where `git log --format=%B` cannot see them. The plain Git arms therefore score 0% on that corpus. That 0 is a fact about the record channel, not about Git, so both plain arms are kept and reported and `git-log-path-notes-budgeted` is added as the comparator that reads the same records from the same place at the same budget. Two things the harness refuses rather than works around. A clone that is not at the pinned SHA, or is not clean, stops the run: the delivery metric reads the working tree through `git ls-files`, so a dirty clone would score files the pinned tree does not contain. And the coverage sweep asserts, per path, that `git log --format=%B` really is each body followed by one LF — the offsets that decide which commits survive a prefix cut are wrong otherwise, and a coverage figure computed from wrong offsets still looks like a result. Limit: the sweep is two `git log` calls per path and the delivery phase runs `git log --follow` on every tracked path, so a full run over the four externals is hours rather than minutes on one machine Ruled-out: reimplementing the scorer for the external corpus | two scorers that drift by a line produce two numbers that look comparable and are not, which is the failure #163 already recorded here Ruled-out: replacing the per-path `--follow` chain with one forward pass over a rename graph | it is much faster and it is a different attachment predicate from the one DECISION-DELIVERY.md §3 registered, so the two corpora would stop being comparable Ruled-out: turning the census notes option on by default | every measurement registered before it existed read commit messages only, and a denominator that silently grew would invalidate them without saying so Warn: `budgeted_log_coverage` is not recall — it counts commits, not records, and only the calibration row at b3f5692 says how far apart the two are on a repository where both are known Verified: 15 new tests over a fixture whose reverts undo modifications rather than additions, so the reverted path is still tracked at HEAD and the denominator is not empty Verified: a notes-only record is invisible to `git-log-path` and delivered by `git-log-path-notes`, measured rather than asserted Verified: `buildCensus` finds 0 records without the option and 1 with it, on the same fixture Verified: typecheck of bench/tsconfig.json, npm run typecheck, and scripts/check-readme-numbers.mjs all clean Unverified: the four externals have not been measured yet — the harness has run end to end only on psf/requests and on the test fixture Evidence: bench/external/run.ts Evidence: bench/external/backfill.ts Evidence: test/external-corpus.test.ts Follows: r-e1a7c4 Blast: module Undo: easy Certainty: firm Record-Id: r-3c9d52 Provenance: authored CommitLore-Version: 2.0.0
First measurement this project has on a corpus it did not write: 137,706 commits of Django, SymPy, scikit-learn and requests against the 549 of its own. The method was registered two commits ago and nothing in it moved. The answer to the question that mattered is clean. The record-free skeleton of the `git-log-path-budgeted` arm reads 55.6%, 49.3%, 44.9% and 37.4% on the four externals. On this repository, at the exact commit that produced the published rows, it reads 39.1% against a published record recall of 42.0% — and 66.8% macro against 65.2%. So the proxy tracks the thing it stands in for to within three points where both are known, and 42.0% sits inside the external band, nearer its floor than its ceiling. The mechanism does not transfer even though the number does, and that is the more useful finding. Django loses coverage because its paths carry a median of 8 commits at 213 tokens for the whole log; this repository loses it because its paths carry a median of 1 commit at 687 tokens, so one message exhausts the budget and 258 of 1,069 paths get nothing at all against Django's 1 of 7,079. Writing longer commit messages makes ordinary Git worse at a fixed budget. That is a cost of the practice this tool encourages and it was not visible anywhere in bench/ before now. The delivery half is an oracle and is labelled as one in §9.0, above the number rather than below it. 169 records generated from revert commits, each correct by construction because the filter kept only reverts whose change is provably still gone. `commitlore` reads 100.0% on all four, which §9.5 says plainly is what the attachment predicate forces once records are small and the budget never binds — not a retrieval result. The informative quantity is the gap to ordinary Git reading the same records from the same place at the same budget, and §9.6 decomposes it into three different stories rather than pooling them: Django is a budget story, scikit-learn is a rename story, and SymPy leaves 35 pairs that neither explains. That residue is the one new thing about the baseline. Four of the 35 were opened by hand and in each the commit carrying the record changed the path and does not appear in `git log -- <path>` at all. Git's default history simplification does not enumerate every commit that touched a path — which, if it generalises, costs the ordinary-Git comparator records that no budget and no `--follow` would recover, on this repository as much as on SymPy. Limit: no maintainer of these four repositories wrote a record, so §9.4 bounds a process nobody performed and is not evidence anybody would perform it Limit: zero superseded and zero expired records in the whole external corpus, so `stale_delivered = 0` on every arm is arithmetic and not evidence about the lifecycle filter Limit: one template, one record shape, one decision class — reverts are the decisions loud enough to reach the history as an undo, and constraints, warnings and quietly-abandoned alternatives leave none Ruled-out: quoting the 0.0% on the two plain Git arms as a result | the records are in the notes mirror because backfill must not rewrite these histories, so that 0 is a fact about the record channel and not about Git Ruled-out: pooling the four repositories into one figure | the three-way split in §9.6 is invisible in any average, and the question asked was whether a figure from one repository describes another Warn: `commitlore` at 100.0% is near-definitional on this corpus — gold and the shipped route decide which records belong to a file by the same predicate, and with a budget that never binds there is nothing left for it to falsify Verified: 39.1% record-free coverage against 42.0% published recall, and 66.8% macro against 65.2%, at the same commit b3f5692 Verified: 41 of 418 revert candidates name a change that is materially back in the tree at the pinned SHA and were refused; Django accepts 33 at a 0.25 threshold, 43 at 0.5 and 49 at 0.75 Verified: 102 of the 169 accepted records carry no reason at all, because `git revert`'s default message is a subject and a boilerplate line Verified: every clone was asserted to be at its pinned SHA and clean before it was measured, and the byte layout of `git log --format=%B` was asserted per path before any commit was scored as delivered Unverified: whether history simplification explains SymPy's 35-pair residue at population scale — four cases were opened, the count was not measured Unverified: nothing here says the 81.7% delivery figure transfers; the corpus that would answer it is one where somebody other than this tool's author wrote the records, and no such repository exists yet Evidence: bench/EXTERNAL-CORPUS.md Evidence: bench/results/external-corpus-20260801T133722Z.jsonl Follows: r-3c9d52 Blast: local Undo: easy Certainty: firm Record-Id: r-7f2b18 Provenance: authored CommitLore-Version: 2.0.0
Measure CommitLore on four repositories it did not write
Every figure this project publishes was measured on this repository measuring itself, and the sharpest reading of that is not that the numbers are wrong but that nobody can tell. The `git log` comparator was the part most exposed: 42.0% could as easily have been a fact about how this repository writes commits as a fact about `git log`. It is not. On four repositories nobody here wrote -- Django, SymPy, scikit-learn and Requests, at pinned commits -- the share of a path's history surviving an 800-token cut runs 37.4% to 55.6%, and 42.0% sits inside that band. The more useful half is what did not transfer, and it costs us rather than flatters us. Our paths carry a median of one commit at 687 tokens where Django's carry eight at 213, so 258 of our 1,069 paths get nothing at all at that budget against Django's 1 of 7,079. Long commit messages make the ordinary-Git baseline worse at a fixed budget, and that is this project's own practice showing up in its own comparator. It was not visible anywhere before this run. The delivery figure from that corpus is deliberately not quoted here. It reads 100.0%, and it reads that way because a backfilled record is one line, the budget never binds, and gold and the shipped route decide what belongs to a file by the same predicate. The pointer says to read the two sections that explain it before the number, because the number alone would be the most misleading thing on this page. Record-Id: r-extbaseline Limit: the band is four Python repositories chosen for having enough revert history to backfill from, so it is evidence about large long-lived Python projects rather than about repositories in general Ruled-out: Quoting the 100.0% delivery figure | it is the value the attachment predicate forces on one-line backfilled records, not a retrieval result, and on a README it would read as the opposite Ruled-out: Leading with the external band instead of our own 81.7% | the external corpus has no real records in it, so it cannot carry the claim the delivery number makes Certainty: firm Blast: local Undo: easy Verified: readme, readme-order, readme-numbers, readme-positioning and compatibility-matrix pass at 83 across all four language files; check-readme-numbers.mjs exit 0 with the BENCH block byte-identical; spec/verify.sh OK at 26 fixtures; the four pinned corpus SHAs were resolved against the GitHub API and their dates match the corpus statement Unverified: whether the band holds outside Python, and whether a repository whose records were written by someone other than this tool's author would reproduce 81.7%
Answer the strongest objection to the delivery number, in the README
The README's only performance figure was 7.4 seconds to index 768 commits, from a field report. That answers "is this cheap to try" and leaves the question anyone with a real repository asks next: what happens at a hundred times that size. The answer is measured and it was sitting in docs/evidence.md. An indexed context query at 100,000 commits is 496 ms p50; the commit-msg hook is 185.85 ms p50 and the injection hook 102.40 ms. Those are the numbers that decide whether this stays installed rather than whether it gets tried once, and the second question is the one that costs a user something to answer for themselves. The paragraph also carries the figure from that run that looks bad: the same query without an index takes 86,673 ms at 100,000 commits. Publishing the good number while leaving that one in a linked document would be choosing which half of a single measurement a reader sees. It also explains a design choice rather than excusing it -- the index is not a speedup on a query that already worked, it is what makes the query possible at that size, which is why init builds one and doctor checks it. Record-Id: r-scaleproof Limit: the 100,000-commit figures come from a synthetic repository built by the deterministic harness, not from a real codebase of that size, so they describe the index's shape rather than any particular project Ruled-out: Quoting only the 496 ms | it is one arm of a run whose other arm is 86,673 ms, and a reader who found that later would be right to distrust the first number Ruled-out: A latency table | three figures do not need one, and the README just had its reference material moved into docs/ Certainty: firm Blast: local Undo: easy Verified: readme, readme-order, readme-numbers, readme-positioning and compatibility-matrix pass at 83 across all four language files; check-readme-numbers.mjs exit 0 with the BENCH block byte-identical; spec/verify.sh OK at 26 fixtures; the beautify skill's audit_readme.py reports no issues; every figure quoted was read out of docs/evidence.md rather than recomputed Unverified: how the indexed query behaves on a real repository at that size, which no run in this project has measured
Put the scaling numbers where the objection is raised
A competitive survey turned up arXiv 2603.15566 -- "Lore: Repurposing Git Commit Messages as a Structured Knowledge Protocol for AI Coding Agents", March 2026, four months before this repository existed. Its abstract describes this product: git trailers carrying constraints, rejected alternatives, agent directives and verification metadata, no infrastructure beyond git, a standalone CLI, readable by any agent that can run a shell command. Compared against its README, the vocabularies map almost one-to-one -- Constraint to Limit, Rejected to Ruled-out, Confidence to Certainty, Scope-risk to Blast, Reversibility to Undo. Both worked examples are an auth-token-expiry commit. So the differentiator table had a row missing, and it is the row a reader would have found on their own. Finding it that way teaches them something about this project rather than about Lore. What is actually different was checked rather than assumed. Lore has no Supersedes:, no Expires: and no lifecycle concept -- grepping its README for supersede, expire, lifecycle, retire or reversed returns only matches inside its own worked example, which is about expired auth tokens. It has no trust grading. And the paper says it "outlines an empirical validation path", which is a precise way of saying it ran no experiment. That leaves a claim this project can defend: not that it invented the protocol, but that it added the lifecycle and the grading, and ran the validation the paper outlined -- including VERDICT-M1 at p = 0.7480, VERDICT-M4 withdrawn for unverifiable exposure, and ROUTE-GAP, where the injected context was read and ignored. Being second is a fact about a date. Being second and quiet about it would have been a fact about the project. Record-Id: r-priorart Limit: the comparison is against Lore's README and its abstract; the full paper was not read, so a lifecycle described only in the PDF would have been missed Ruled-out: Saying nothing | the survey found it in an afternoon and anyone evaluating this project will Ruled-out: Framing it as convergent design | it may well be, but "we arrived independently" is unfalsifiable and asks for exactly the trust this project argues should not be extended to unverifiable claims Ruled-out: A feature-by-feature comparison table | Lore is a design paper with a small implementation, and scoring it against a shipped tool would be unfair and would read as defensive Certainty: firm Blast: local Undo: easy Verified: arXiv 2603.15566 resolved through the arXiv API -- title and abstract read directly, published 2026-03-16; the trailer vocabulary and the absence of any lifecycle term were checked against tmdgusya/lora's README; readme, readme-order, readme-numbers, readme-positioning and compatibility-matrix pass at 83 across all four language files; spec/verify.sh OK at 26 fixtures Unverified: whether the resemblance is convergent or derivative, which is not decidable from either artifact and is not claimed either way
Name the paper that published this protocol first
`hooks install` writes commitlore.bin/commitlore.root into one repository's config. An upgrade installs the new release somewhere else and never visits the repositories that pinned the old one, so a repo can sit two releases behind on the path that actually validates its commits while `commitlore --version` reports the newest one. Reproduced against a v0.5.0 install directory alongside 0.6.0: both doctor and hooks status called it healthy, and doctor printed the stale v0.5.0 path inside its own `ok` line. It had the value in hand and did not act on it. That is not cosmetic, because the hook is the enforcement point. Everything 0.5.1 and 0.6.0 shipped that touches validate, prepare-commit-msg or post-commit is inert in such a repository -- this release's Ruled-out: narrowing, the shallow-clone gate, duplicate-id dedup, the fail-open capture hooks. It is also latent: deleting the old install directory turns every commit there into a hook failure. The comparison goes in core/hook-target.ts, the mirror both readers already share, so doctor and hooks status cannot disagree about it and the remedy they already name -- `commitlore hooks install` -- is the one that fixes it. A pin whose version cannot be established is reported too, not passed over. The pinned build still decides what runs, and "could not find out" is a different answer from "it matches"; a false green there costs a repository every fix shipped since the pin was written, which is the trade this project has made the same way everywhere else. Record-Id: r-pinskew Limit: the comparison reads the package.json above the recorded path rather than running it with --version, so a pin whose manifest and bundle disagree is reported by its manifest Ruled-out: Re-pointing the pin from doctor --fix | doctor's --fix boundary is reversible local config only, and the module comment states the commit-msg hook is reported and never written -- rewriting commitlore.bin would also change which binary validates commits, silently, from a diagnostic Ruled-out: Having the upgrade re-point every repository | install.sh cannot enumerate the repositories that ran hooks install, which is why the issue itself proposes reporting as the self-correcting half Ruled-out: Spawning the pinned bundle for --version | the recorded path is by design a value a .git/config edit can change, and hooks status spawns nothing today -- executing it on every status and every doctor is a larger door than the question is worth Ruled-out: Treating a manifest with no version as a match | that is the false green the check exists to remove Certainty: firm Blast: module Undo: easy Verified: reproduced with a v0.5.0 install pinned beside 0.6.0 -- doctor and hooks status both reported ok before, both name the skew and `commitlore hooks install` after, and running that command clears it; 6 tests added across doctor, hooks and hook-target failed first and pass now; doctor, hooks, init, cli, hooks-entry-resolution and hook-target pass at 180; npm run typecheck and npm run build clean Unverified: whether any repository other than the reporter's carries a stale pin, which nothing here can enumerate
Compare the version the hook is pinned to, instead of printing it
TOKEN-LEDGER.md blocker B gave two reasons the drafting turn (W4) is missing from the write side. One was that it needs a model call. The other was sharper: the driver read a single session-total `usage` object, so the harness could not have attributed an answer to the turn that produced it even if a call were made. The second reason is now false. This is the instrument, not the measurement. The CLI was run rather than read about. `--output-format stream-json` emits an assistant event per content block, each carrying the same `message.usage` and the same `message.id`, so summing them double-counts every turn. Deduplicated, `input_tokens` and both cache fields are final and reconcile exactly with the session total — and `output_tokens` does not. It is the `message_start` snapshot: three turns reported 4, 1 and 1 against real outputs of 157, 193 and 36, a session total of 6 against 403. Nothing on the event marks it provisional while its neighbours are not. The term the write side is missing is precisely the one term this format reports wrongly, and the correct figure arrives only on `message_delta`, only under `--include-partial-messages`. Both raw captures are committed as fixtures, so that paragraph is checkable rather than asserted. Nothing an earlier run measured moves. `--per-turn-usage` is opt-in and off is the shape every row in bench/results/ was produced under. Both output formats end in the same `result` object, and `transcript`, `turns` and `tokens` are still read from it under the same field selection — verified against a live run: session total 26 + 649 + 752 = the 1,427 the row reported. The ledger is built from events the JSON format never printed and nothing already measured is re-derived from them. Every ledger carries its own audit rather than asking to be trusted: `turn_total`, `session_total`, and a `reconciled` boolean that is true only when the turns sum to the total the CLI states for itself, field for field. A parser that missed or duplicated a turn says so on the row. The schema gate was already failing before this touched it. `reproposal_matches` and the four `rejected_path_*` counts reached the rows without reaching `result.schema.json`, so `verify.mjs` rejected all 80 rows of the most recent matrix on `additionalProperties`. Adding a sixth field to that list without fixing it would have shipped a gate that gates nothing. Limit: `turn_usage` carries the provider's tokenizer; every figure in bench/TOKEN-LEDGER.md is in the product's own ceil(chars / 4) proxy. A future W4 must be reported beside the floor, never added into it, until both sides are on one tokenizer Limit: `content_blocks` says a turn called a tool, not which tool or on what — the tool_use block's name and input are discarded, so #141's rejected-path tool actions stay uncounted Ruled-out: turning per-turn capture on by default | it changes the CLI invocation, and a harness flag that quietly rewrote how a run is observed would make old and new rows two different things under one name Ruled-out: summing the `usage` on assistant events | they repeat once per content block and their output_tokens is a pre-generation snapshot; that reading is wrong by 397 of 403 on the one probe that checked it Ruled-out: writing `turn_usage` on every row with nulls when uninstrumented | an always-null field is worse than no field, and absent already reads as "not instrumented" everywhere else on this row Ruled-out: retaining the whole stream to parse at the end | partial messages emit an event per chunk, so a long run would hold tens of megabytes to produce a few hundred integers Warn: this closes the attribution half of blocker B only. W4 is still unmeasured, W3 is still unrecoverable for this corpus, and no figure in TOKEN-LEDGER.md section 9 changes Warn: section 5 of TOKEN-LEDGER.md is pre-registration and was appended to, not edited — the floor has to stay visibly declared in advance Verified: the raw event stream of two real invocations, kept and committed under test/fixtures/claude-stream/ with local paths redacted Verified: the per-turn ledger reconciles with the CLI's own session total on all four usage fields — 26 / 386 / 307 / 72,845 on the fixture, and again on a live run through the wired driver Verified: the same driver with the option off and on returns the same transcript, turn count and token field selection Verified: 12 new tests; bench/verify.mjs now validates 252 rows across three current-shape result files that it previously rejected Verified: npm run typecheck, tsc on bench, npm run build, and scripts/check-readme-numbers.mjs all clean Unverified: no drafting turn has been priced. No bench arm runs `capture` against its own transcript and diff, so the number blocker B is about still does not exist Unverified: subagent attribution is covered by a constructed stream, not an observed one — no probe here produced a nested agent Evidence: bench/TOKEN-LEDGER.md Evidence: bench/drivers/stream-json.ts Evidence: test/fixtures/claude-stream/partial-messages.jsonl Follows: r-ledgerresult Blast: module Undo: easy Certainty: firm Record-Id: r-4e7c19 Provenance: authored CommitLore-Version: 2.0.0
test/hooks.test.ts carried one literal NUL at offset 7059, line 184, inside
`.join('\n\0\n')` -- where the `\0` was the raw byte sitting in the source, not
the two-character escape it reads as. Git calls a blob binary when it finds a
NUL in the first 8000 bytes, and 7059 is inside that window, so every diff of
this file printed `Binary files ... differ` on GitHub and locally. Scanning all
1,251 tracked files finds exactly one hit, this one.
The cost was paid on every review. #388 changed this file, and reviewing that
change meant stripping the byte into a scratch copy and diffing that by hand.
The file is 1,200-plus lines and covers the commit-msg hook, so the file hardest
to review was the one covering the enforcement point. Search was worse than the
diff: `grep -n OWN_HOOKS_DIR test/hooks.test.ts` exited 1 and reported nothing
for a symbol on line 49, well before the byte -- not truncation at the NUL but a
refusal to report the file at all, and `git grep` answers only `Binary file
test/hooks.test.ts matches`.
The question is whether the NUL was buying something. The separator joins
per-file entries of the form `${name} ${mode} ${contents}`, so it has to be a
sequence that will not turn up inside a hook body; otherwise two different states
of .git/hooks could serialise to the same string and a change would pass unseen.
A NUL looks like the one sequence a text file cannot hold. In this position it is
not. The entries are read with `readFileSync(path, 'utf8')`, which decodes a 0x00
byte in a hook to U+0000 in the string -- measured directly against a hook file
written with a NUL in it. The old separator was therefore exactly as forgeable as
a text one, and worse for the only hooks that plausibly carry NULs at all: a
shell script holds none, a compiled hook holds many.
What actually makes either spelling safe is that the joined string is never
parsed. It is compared for equality against a snapshot of the same directory
taken earlier in the same process, so an ambiguous separator cannot mis-split
anything. The only failure available to it is a collision, and a collision needs
a hook body carrying the sentinel on a line of its own, placed so the whole
concatenation is reproduced byte for byte. None of that reasoning was written
anywhere, which is how the byte survived; it is now recorded at the definition,
which is the part that keeps.
The assertion is untouched. `expect(snapshotOwnHooks()).toBe(ownHooksBefore)`
still compares names, modes and bytes, and the helper still builds its entries
the same way. Only the spelling of the separator moved.
Record-Id: r-nulsep
Limit: the sentinel is safe because nothing that writes into .git/hooks contains it, not because it is unrepresentable -- a hook body carrying the sentinel on its own line is the one input that could in principle collide, and the helper does not reject it
Ruled-out: Keeping the NUL and recording why in a comment | the reason would have to be that a NUL cannot occur in a snapshotted body, and it can -- readFileSync utf8 decodes 0x00 to U+0000 -- so the comment would have documented a guarantee the code does not provide
Ruled-out: Marking the file text in .gitattributes | that hides the symptom from git diff while leaving the byte in the source, so grep, editors and everything else that reads the tree as text still skip or truncate it
Ruled-out: Length-prefixing each entry to make the encoding injective | genuinely collision-proof, but it changes what the snapshot serialises, and #389 is about how the separator is spelled rather than what the comparison holds
Ruled-out: Joining on a plain newline | hook bodies end in a newline, so entries would run together and a change that moved bytes across an entry boundary could vanish, which is the collision the separator exists to prevent
Certainty: firm
Blast: local
Undo: easy
Verified: one NUL at offset 7059 confirmed by byte scan, and the same scan over all 1,251 tracked files reports none anywhere after the change; readFileSync utf8 measured to yield U+0000 from a hook written with a 0x00 byte, which is what makes the NUL non-load-bearing; the new blob holds no NUL, so a further edit to the file diffs as text and git diff --numstat reports line counts instead of a dash; the helper re-run with the new separator against a scratch repository's real .git/hooks still detects a one-byte content change, a 755 to 644 mode change, a hook added, a hook removed, and a body that embeds the sentinel itself; hooks, init and doctor pass at 130; npm run typecheck and npm run build are clean and dist is byte-unchanged
Unverified: whether any hook anywhere carries the sentinel in its body, which nothing here can enumerate; the diff of this commit itself still reads binary because its pre-image is, which no change to the post-image can alter
Make per-turn token usage observable, and say what it still cannot price
Spell the hook snapshot separator as text, so the file can be reviewed
A stop reason this document never named turned up in the registered run: over-tokens, from the fixture's own per-task budget rather than the invocation cap, which has spent about 3 million of 200 million. It is the same shape as over-turns -- a run cut off before finishing, where a false carries less than a false on a completed run. At 71 of 1,160 rows the combined truncation rate is 16.7% in the treatment arm and 31.4% in the control. The off-design sample was balanced at 15 and 15; this is not. Two readings fit and the honest thing is to write both down now rather than pick whichever the final table makes convenient. It could be noise that pushes the control's rate down, which works against the hypothesis rather than for it. Or it could be the treatment settling sooner because it has the records, which is an effect rather than a confound -- on an outcome this measurement did not register and therefore cannot claim. Nothing about the analysis changes. Excluding truncated rows after seeing that their rate is asymmetric is precisely the move a pre-registration exists to stop, and M1 kept them, so dropping them would break the one comparison M5 was built to make. What is added is an obligation on the verdict to report all three shares per arm, registered now because after the table exists it would be worth nothing. Record-Id: r-m5trunc Limit: the rates come from 71 rows, so they indicate an imbalance rather than measure one; the verdict reports the final figures Ruled-out: Excluding truncated rows from the analysis set | the rule would change after the rate became visible and asymmetric, and M1 kept them Ruled-out: Raising the per-task token budget for the remaining shards | it changes the instrument mid-run and makes the completed rows a different measurement Ruled-out: Stopping to investigate the imbalance | the stopping rule forbids interim analysis, and the imbalance is reportable without resolving it Certainty: firm Blast: local Undo: easy Verified: over-tokens traced to tokens 60000 in the fixture YAML against a run that spent 74,314; invocation cap at 200,000,000 with about 2,958,350 spent; M1's truncation share computed from t702-m1-final.jsonl at 15.0% of 60 reproposal rows Unverified: which of the two readings is right, which the registered comparison cannot decide and this note does not claim to
CDEB-P ran sixteen times and returned three parameters that could not have been found by reading the protocol again. v1.3 changes exactly those, and nothing about the design, the matrix or the twenty locked decisions. The token threshold was 15%, chosen before anyone measured what the ON arm costs. It costs 45% more, so 15% required a 1.71x relative lift -- at a 50% control rate the ON arm would have had to reach 85.5% to clear a gate whose sibling asks for ten points. Injecting context costs tokens by construction, so the threshold has to know the overhead, and it cannot learn it from the outcomes without becoming a number chosen after the fact. It now derives from a calibration measured on the disposable smoke tasks, which never enter the corpus and therefore expose no outcome, and both the calibration and the derived threshold are frozen before the study runs. Token efficiency also leaves the full-headline conjunction. Three ANDed gates sounded like three measurements and behaved like one: the hardest could veto the other two, so a real behaviour improvement would have reported FAIL. It is still evaluated, still reported, still available as a partial claim, and its threshold is now stricter than the number it replaced. "Completable in one fresh agent session" was a wish with no check, and one task in four burned a quarter of the study hitting the wall in all four runs. A task now passes a wall-clock probe at 60% of budget before sealing, and the probe may read only wall time and stop reason -- never the oracle -- which is the same distinction that let this session read stop_reason mid-study without reading a result. Qualification required the record to exist, not to be reachable. Two tasks in four delivered nothing to the ON arm, which made those runs ON by assignment and OFF in substance and left one task carrying the whole question. Sealing now requires that a path the good control edits actually renders the expected record at the frozen snapshot -- a check that runs no agent. Limit: the calibration is measured on disposable tasks, so it estimates overhead on work that is not the corpus and a corpus with different injection density will sit off it Ruled-out: raising the 15% to a number that looked achievable | picking a threshold to be clearable is the same act as picking one after seeing results, and only the derivation makes it preregistered Ruled-out: keeping token efficiency in the conjunction with the new threshold | the objection was never the number, it was that one gate could erase a measured behaviour result while the report called it a three-metric study Ruled-out: widening the hook matcher so the two silent tasks would deliver | that redefines a delivery failure as a delivery success, and §9.5 exists to record exactly that failure Warn: the 0.6 probe budget and the 1.15 calibration factor are judgement calls with no measurement behind them, unlike the three findings that forced this revision Blast: local Undo: easy Certainty: firm Verified: schemas parse and now pin 1.3.x so a protocol change is a schema change CI notices, the token threshold is a range with a required calibration rather than a constant, and the eleven verifier cases pass against a 1.3.0 fixture Provenance: authored Record-Id: r-cdebv13
Write down what 0.7.0 changes, starting with the tier nobody could reach
An external production-readiness review rejected the first v1.3 draft, and it was right on every finding. This replaces that draft. The formula was wrong, and wrong in the direction that matters. TVPDSS reduction is `1 - o/q` for overhead o and relative lift q, so the overhead is already inside the metric. Deriving the threshold from o again counts it twice, and the algebra comes out as `q >= 1.15*o^2` -- at the pilot's measured o = 1.45 that is a 2.42x lift, which at a 50% control rate asks the ON arm for 120.9%. I replaced an unreachable gate with an impossible one. The stated 5% floor could never be selected either: the schema requires o >= 1.0 and the formula returns 13.04% there. The 15% threshold is restored, and the diagnosis that produced the draft is withdrawn with it. The gate was never unreachable -- it was strict because injecting context costs 45% more tokens, and asking for 15% fewer per usable result on top of that is a demanding bar rather than a broken one. Calibration stays as a feasibility note that sets no threshold. Removing token from the headline conjunction while leaving §17.1's sentence alone created a path where a failed token gate still produced "used Y% less token volume". Three gates now exist with fixed wording each, and only the combined one generates the three-metric sentence. The draft's claim that this loosened nothing was false and is recorded as false: `P and M` is strictly weaker than `P and T and M` whatever T's threshold becomes. The runtime probe named no arm, which lets one arm's runtime select the corpus; it discarded the row the freeze manifest was supposed to contain; and it called a returned process a completed task, which a no-op satisfies. It is now an ON and OFF pair, artifacts preserved, renamed to runtime-boundedness. Deliverability was checked with `commitlore context`, which is not the surface CDEB measures -- budget, trust grading, matcher, index and output parsing all sit between it and the agent, and zero shipping delivery was the defect being fixed. It now drives the frozen shipping inject path and passes only when the record appears in the forwarded payload. And the draft asserted the pilot showed task qualification caused the two silent tasks. The pilot's own result says its instrumentation could not tell that from a hook that never fired. The causal claim is withdrawn; v1.3 closes both possibilities separately instead. Limit: schemas and the verifier now require the qualification evidence, but nothing yet produces it -- CDEB-10 must build the probe and the deliverability check before a freeze can be written Ruled-out: keeping the derived threshold with the exponent corrected | any threshold that moves with measured overhead is a threshold the study can influence, and the fixed bar is the only one that means the same thing before and after the run Ruled-out: lowering 15% to something the pilot's overhead clears | that measures a different claim and calls it the same one; if 45% overhead makes the bar hard, the bar is reporting a true fact about the product Warn: the 0.6 probe budget is still a judgement call with no measurement behind it, and it now gates corpus membership through two agent sessions per task Blast: local Undo: easy Certainty: firm Verified: the double-count reproduces as q >= 1.15*o^2 giving 120.9% at o=1.45 against 85.3% for the fixed bar, and 18 verifier cases pass including a study with no freeze manifest, a hand-set threshold, an unqualified corpus, a build that disagrees with its freeze, delivery with zero executions, executions exceeding opportunities, and a hook that fired on a path with no records Provenance: authored Record-Id: r-v13fix
…uessing it Two of the review's remaining limits close here. The third does not, and says so. The delivery qualification exists now rather than being a paragraph. It builds a real PreToolUse payload and runs the pinned shipping injector -- same command, same budget, same trust configuration, same snapshot as the ON arm -- and passes only when the expected record id appears in the bytes the hook forwarded. It renders nothing itself. One test carries the whole argument for why the first version was wrong. A record sits in the repository, on the edited path, and the shipping budget is squeezed to one token: `commitlore context` would still show it, and this check fails. That is the entire difference between the surface that was checked and the surface CDEB measures, made into an assertion. The pilot's defect was zero shipping delivery, and only one of those two surfaces can see it. The 0.6 probe threshold was a judgement call, and the pilot's own wall times settle it. Completed runs topped out at 0.48 of budget and the failing task sat at 1.00, so the good and bad tasks separate anywhere in between and 0.6 touches neither end. The same data limits what the gate may claim: two repeats of one cell ran 89s and 431s, a 4.9x spread, so two probes cannot bound the tail. The gate screens the median. Study timeouts stay ordinary measured failures under intention-to-treat, and what it prevents is the thing that actually happened -- a task whose four runs all timed out and contributed nothing. Limit: the runtime probe still has no implementation, so a freeze cannot yet be written even though the schema now demands its evidence Ruled-out: qualifying delivery through `commitlore context` | budget, trust grading, the guard, lifecycle projection, the matcher and output parsing all sit between it and the agent, and the defect being fixed was zero delivery through exactly those Ruled-out: requiring every good-control path to carry every record | a record scoped to one file of a multi-file change still reaches an agent that opens that file, and demanding all of them rejects sound tasks Ruled-out: throwing on a non-zero injector exit | the hook is fail-open by design, so the exit code belongs in the freeze manifest where a reader can weigh it Warn: delivery qualification proves the record can arrive, not that it will -- an agent that edits another path first or uses a tool outside the matcher still gets nothing, and §9.5 is what records that Blast: local Undo: easy Certainty: firm Verified: six cases -- a path carrying its record qualifies, a path carrying none does not, a squeezed budget fails what context would pass, a missing second record fails the task, any one of several paths suffices, and the payload digest and exit code are recorded rather than thrown Provenance: authored Record-Id: r-delivchk
…lot's numbers The schema demanded runtime qualification evidence that nothing produced, which was the correct failure direction and still a gap. It is closed. The decision function is pure -- probes in, verdict out -- so the gate can be tested without spending two agent sessions per assertion, and `runProbe` is the only part that spends them. That split is what makes this suite runnable in CI at all. Every case is a pair the pilot actually produced. 80s and 89s qualifies. The task that sat at 902s and 903s in all four runs does not. 431s -- the slowest run that ever completed, at 0.48 of budget -- qualifies, which is the observed separation the 0.6 threshold sits inside without touching either end. Three refusals encode review findings rather than taste. A single arm is refused because runtime is treatment-sensitive and qualifying on one arm selects a corpus that arm finishes faster, with the bias inseparable afterwards. Two probes of the same arm are refused for the same reason. The verdict reads the slower arm rather than the mean, because the study runs each task six times and averaging lets a fast arm carry a slow one into the corpus. The last test asserts what the probe type does not contain. A selector that could see functional_pass or rejected_decision_revived would be selecting on the dependent variable, so the absence is pinned rather than assumed. Limit: the gate screens runtime and says so -- `completed` means the process returned, which a no-op satisfies, and the pilot's 4.9x within-cell spread means two probes cannot bound the tail Ruled-out: averaging the two arms | the study runs six sessions per task, so the slower arm is the one that decides whether they fit, and a mean lets the faster arm carry a task in Ruled-out: keeping the probe artifact out of storage | the freeze manifest is required to carry per-task probe results, and a qualification nobody can recheck is not a gate Warn: `runProbe` costs two agent sessions per task, so qualifying a 30-task corpus is 60 sessions before a single measured run Blast: local Undo: easy Certainty: firm Verified: eight cases against real pilot pairs -- 80/89s qualifies, 902/903s timeout does not, 431s at 0.48 of budget qualifies, a 700s slower arm fails on the slower arm, single-arm and same-arm probe sets are refused, an errored arm is refused however fast, and the probe type carries none of the three outcome fields Provenance: authored Record-Id: r-runprobe
The runtime gate decides corpus membership by wall time, so a probe on a different model than the study screens a distribution the study will never produce. §2.2 already forces a new study id when the observed model changes; this is the same rule reaching the qualification that selects the corpus, and `qualifyRuntime` now refuses a probe whose model is not the pinned one. The model is sonnet. The 0.6 threshold this gate applies was derived from the pilot's wall times -- completed runs topping out at 0.48 of budget against a failing task at 1.00 -- and every one of those runs was sonnet. Switching models does not adjust that derivation, it deletes it, and the threshold goes back to being a number somebody picked. M1 and M5 also measured sonnet, so this is the choice that leaves CDEB comparable with the evidence already published. The refusal is worth having whichever model is chosen later, because the failure it prevents is silent: a qualification run on a stronger model finishes faster, admits tasks the study cannot finish, and the resulting timeouts look like task difficulty rather than a mismatched screen. Limit: pinning to sonnet means CDEB will say nothing about whether a stronger model needs delivered decisions less, which is a question worth its own study rather than a variable inside this one Ruled-out: qualifying on opus while measuring on sonnet | the gate would screen runtimes the study never produces, and its threshold is derived from sonnet data Ruled-out: leaving the probe model unchecked and documenting the requirement | the mismatch produces timeouts that read as task difficulty, so a note cannot catch what a comparison can Warn: a later model change invalidates the 0.6 derivation as well as the pinning, so both must be re-established rather than only the constant edited Blast: local Undo: easy Certainty: firm Verified: nine cases pass, including a pair where one arm probed opus against a sonnet-pinned study and the qualification refuses by naming both Provenance: authored Record-Id: r-probemodel
Fix the three protocol numbers the pilot proved wrong
A repository with 815 commits, hooks installed and an index current with HEAD held zero CommitLore records, and doctor reported all ten of its checks ok. Four captures sat in its pending directory, one of them staged with a passing validation and a record ready to attach, all four eight days old. Every step behaved as designed. `capture-stage` stamps expiry five minutes after staging; the commit did not happen inside that window; the prepare-commit-msg gate skipped the record because it had expired; pending-gc protects the staged phase and so never collected the file. The net effect was that the product silently stopped producing records and said nothing. `pending ls` already prints `stale` and `never-collected` on exactly those rows. The information existed. The command people actually run did not carry it, which is #402 and #400's category and the reason this is a doctor change rather than a capture change. The check separates the staged loss from the drafts that never staged, because merging them hides which decisions had a record ready to attach. It names the oldest timestamp so the age is visible without a second command. Limit: the check reads staleness, so a capture whose base commit is still HEAD reports ok even if it has been waiting long enough that nobody remembers preparing it Ruled-out: lengthening the five-minute expiry | a staged record binds to the tree it was prepared for, and attaching it to a different tree is worse than dropping it -- the defect is the silence, not the expiry Ruled-out: letting gc collect expired staged files | that erases the only evidence a capture was ever prepared, and this repository found the defect precisely because the files were still there Warn: a repository that has never captured anything and one whose pending directory was deleted both report ok, so this check cannot distinguish a healthy install from a missing one Blast: local Undo: easy Certainty: firm Verified: six cases -- ok with nothing captured, ok while a capture can still apply, warn naming the staged loss and its wording, staged and never-staged counted separately, no loss claimed when nothing reached staged, and the oldest timestamp present; run against the real repository it reports the one staged loss and three drafts by name Provenance: authored Record-Id: r-doctorpend
…althy Eleven closed issues and one open one are the same defect. #40, #128 and #149 probed a path doctor reconstructed rather than the command that actually runs. #49 let the hook be redirected to any executable and still said ok. #382 called a stale hook pin ok. #335 reported 106 records where git had zero. #296 prescribed a fix that could not repair the condition, and #63's fix broke git fetch and then said ok. #458 reported ten checks ok on a repository that had silently stopped producing records eight days earlier. None of those is a missing check. They are one structural failure: a verdict with no evidence behind it, in a command that had no way to say "I could not look" and no way to say "this failure is that failure's consequence". The ADR decides the model. Every check carries typed evidence rather than prose, so a claim and the observation behind it travel together. A check that cannot run says skipped with a reason from a closed union rather than passing. A check whose dependency failed is annotated `blocked_by` and still renders its own evidence, because a suppressed row is how a screen lies by not looking. Checks become registry data instead of a hand-assembled array, so a new one cannot be added without a category and a place in the order. The PRD specifies it, and the two hardest calls are recorded with their reasons. Exit codes stay 0 and 1: the Swift reference maps degraded to 3, and SPEC §10 fixes 3 protocol-wide as "ran and answered but could not see everything", so that mapping would give 3 a doctor-private meaning §10 forbids. Profiles are rejected with a reopening condition, because a profile that omits checks reproduces #458 exactly -- the screen that lied by not looking. Limit: this lands the model and the specification only; no check changes behaviour and the shipping report is byte-identical until the registry work begins Ruled-out: a `manual` status, as the Swift reference has | it exists there for operator attestations the OS cannot query, and every CommitLore remediation is a command, so a status nothing emits is dead contract surface Ruled-out: the reference's 0/1/2/3 strict exit mapping | SPEC §10 fixes 2 as "could not run" and 3 as "could not see everything" for every command, and RELEASE-GATE §4 plus init's `doctor --fix` depend on warn exiting zero Ruled-out: per-surface profiles, for now | omitting checks to make a surface faster is the #458 shape, and `--only`/`--category` cover the subset uses until a measurement shows the full set is too slow Warn: `blocked_by` is the one new field that can hide a real defect if it ever suppresses a row rather than annotating it, so the invariant belongs in a test before the first collapse ships Blast: local Undo: easy Certainty: firm Verified: SPEC §10's table read directly -- 2 is "could not run", 3 is "ran and answered but could not see everything", and it states a code MUST NOT carry a meaning another command does not give it; the ADR number follows the retired 0023-0025 range that ends at 0022 and resumes at 0026 Provenance: authored Record-Id: r-adr32doc
Say when a capture was prepared and then dropped
An adversarial review of ADR-0032 and the doctor PRD returned three blocking findings, seven major, three minor. Every finding checked out against source, and two were not refinements but reversals of claims the documents stated as fact. This repository publishes its corrections, so the ADR now opens with a section listing what was wrong and what replaced it. The reversals. First, the status derivation made ok unreachable: any non-optional skip degraded the aggregate, and squash-conservation, inject-runtime and inject-version skip routinely on healthy repositories -- permanent degraded is the #458 first-screen lie with its polarity flipped. Skip reasons now carry a class: an observed true empty (not_applicable) does not degrade; only could-not-verify does. Second, the exit-code reasoning was wrong, not merely overstated: SPEC §10 does not forbid doctor exiting 3 -- an unfetched notes mirror and shallow history are code 3's own named examples, and RELEASE-GATE §1 already requires guard to exit 3 on an unfetched mirror. Warn keeps exiting 0, but on its true grounds: RELEASE-GATE §4 compatibility and SPEC's "a command need not use every code" -- named as compatibility debt, with a narrow vision-gap 3 left open rather than falsely closed. The claim that init depends on doctor's exit code was also false (init branches on needsAttention) and is deleted. The review also built three repositories that stayed healthy while broken. Closed: a notes-availability check consults the remote's advertisement so "refspec fixed, nothing fetched" can never aggregate ok; a capture-liveness check warns when the hook chain is green yet no commit ever carried a record (#458's literal shape); evidence is now required on every row, ok included, so an ok with empty evidence no longer typechecks. One adjacent case stays open and is named with its reason instead of papered over. The rest: the absolute network ban was false against the shipping command, which already runs git fetch --dry-run and git ls-remote -- the ban is rescoped to doctor's own process, with offline runs degrading transport rows to warn; registry order is frozen to the shipping runDoctor order the first revision's table contradicted; blocked_by gets one mechanical producer (a factory constructor, with the runner only normalising chains) and the misattribution of SetupDoctor's design is corrected; the severity field and the dedicated install-source check are cut; every mis-cited line number is re-verified against source. Limit: documents only -- src/ and test/ are untouched, and the two checks this revision specifies (notes-availability, capture-liveness) do not exist yet Ruled-out: keeping "SPEC forbids doctor exiting 3" | spec/SPEC.md line 274 names an unfetched notes mirror and shallow history as code 3's own examples, so the prohibition claim was false and saying so plainly beats softening it Ruled-out: degrading the aggregate on every non-optional skip | doctor.ts lines 628, 699 and 942 skip on ordinary healthy repositories, which made ok permanently unreachable Ruled-out: a capture-recency heuristic to close the produced-then-stopped case | SPEC line 155 makes a record per commit optional, so recent recordless commits are not evidence of breakage; the gap is named open in PRD §2.2 instead of guessed shut Ruled-out: dropping --only and --category as the review suggested | they are filters over registry data feeding the partial-honesty rule, not new code paths, and the disagreement is recorded in the ADR's Rejected section with that reason Warn: the earlier revision claimed init depends on doctor's exit code; init.ts line 104 branches on needsAttention and ignores the exit code, so do not reintroduce that claim when next touching §7 Blast: local Undo: easy Certainty: firm Verified: doctor.ts probes the network at lines 196 and 248; its ten skipped return sites and runDoctor's array order at 1021-1036 read directly; SetupDoctor.swift's severity comment found at lines 63-64 not 548, blockingCause at 634, and wc -l over SetupDoctor*.swift gives 3,754 lines across 21 files Provenance: authored Record-Id: r-adr32rev
Decide what a diagnostic must prove before it says a repository is healthy
Both are documented invariants -- ADR-0032 §8 promotes them, PRD §11 names the tests -- and neither had one. A check added next month that phoned home or left a state file behind would have passed the whole suite. Every later ticket in this milestone rebuilds doctor's internals, so the fences go in before the rebuild rather than after it. The read-only fixture leaves a check failing on purpose. A run where everything passes never reaches the remediation paths, and those are where a write would most plausibly appear. Writing the read-only assertion strictly found that the documented wording is false. A plain run touches `.git/commitlore/index.db-shm`, and `--fix` creates that and `-wal`. Those are SQLite's bookkeeping for a WAL database -- opening one creates them for readers as much as writers -- and they hold no committed data. Making the run literally write-free would mean opening the index outside WAL, trading a documentation problem for the concurrency one #420 was about. So the test asserts the invariant that carries the meaning: `index.db` is byte-identical and nothing else moves except those two sidecars, and #473 fixes the wording rather than the code. The socket fence stubs this process's own socket construction, which cannot see a socket opened by a spawned git. Two shipping checks contact the remote that way. PRD §8.1 has since been rescoped to say exactly that, and the header records which half this file can prove. Limit: the socket stub covers this process only, so a check that shelled out to curl would pass it -- the fence is Node-level construction, not egress Ruled-out: asserting zero writes literally, as §8.2 was worded | it would have failed on SQLite sidecars that carry no data, and the honest repair is the sentence, not the database mode Ruled-out: a fixture where every check passes | the remediation paths are where a write would appear, and a green fixture never reaches them Warn: the write inventory compares size and mtime, so a same-size rewrite inside one filesystem timestamp tick would slip through; a content hash would close it at the cost of walking the index on every run Blast: local Undo: easy Certainty: firm Verified: three cases -- a plain run over a fixture with a failing check leaves index.db byte-identical and touches nothing but the sidecars, every check completes with net.Socket/connect/createConnection and dns.lookup throwing, and --fix adds only remote.<name>.fetch config keys Provenance: authored Record-Id: r-docinv461
Fence doctor's read-only and no-network promises before rebuilding it
…nting severity
The row was seven fields. No category, so nothing could ever select a
subsystem; no severity, so ordering findings meant re-deriving it wherever
someone needed an order; no evidence slot, so a check stated its conclusion
with the observation interpolated into prose or dropped. `skipped` carried its
reason as free text, which is a skip nothing can act on.
Severity is now a total function of status computed inside the one factory, and
absent from its parameter list -- passing one is a type error rather than a
discouraged habit. Two axes that can disagree make every consumer resolve the
disagreement; deriving at a chokepoint makes the inconsistency unrepresentable.
`skipped` maps to info rather than warning, because a check that could not run
reported nothing, and giving it a warning's weight is how a report starts
ranking its own blind spots above its findings.
Category is a required positional argument rather than a lookup keyed by id.
The lookup was simpler and wrong in a specific way: a check added later would
silently take a default. This makes omitting one fail to compile.
The instrument landed before the change, which is the part worth keeping. It
caught the very first edit -- and the failure was the snapshot's own, not the
report's: mkdtemp's random suffix was unnormalised, so it varied per run. A
snapshot that fails for reasons unrelated to what it guards is deleted within a
week, so the normaliser was fixed rather than the expectation regenerated
blindly.
Migrating the sixty-one call sites took two attempts. The first rewrote
positional calls into object literals, converted fifteen, and raised the error
count from sixty-one to two hundred and forty-two; it was reverted whole rather
than repaired.
Limit: evidence is `{}` on every row until the ticket that populates it, so the field exists and proves nothing yet
Ruled-out: deriving category from a table keyed by check id | it is less code and gives a new check a silent default, which is the failure mode this milestone exists to remove
Ruled-out: mapping `skipped` to `warning` | a check that could not run has reported nothing, and weighting a blind spot above a finding inverts what the report is for
Ruled-out: repairing the object-literal migration that broke | it had converted a quarter of the sites and quadrupled the error count, and a half-applied mechanical rewrite is harder to audit than a fresh one
Warn: `skipReason` is still optional on the skipped overload; until the sites are mapped a bare skip is constructible, and the type will not catch it
Blast: local
Undo: easy
Certainty: firm
Verified: 102 files and 2,254 cases pass; the text snapshot and the check-order snapshot are unchanged across the model change, every v1 JSON key is present with its type on every row, severity agrees with status on every row of a full run, and the two rows that deliberately clear needsAttention still do
Provenance: authored
Record-Id: r-checkmodel
runDoctor was a hand-written array of thirteen ad-hoc functions. A throw in any of them killed the command, so the user who most needs a diagnosis -- the one whose repository is in a state some check did not anticipate -- got a stack trace instead of the other twelve answers. Nothing was timed, so §10's budget was an assertion with no instrument, and there was no data structure a filter could ever read. Checks are registry entries now. A throw becomes one failed row carrying the message's first line as evidence, and the rest of the report still renders. Every row is stamped from a monotonic clock, because a wall clock can go backwards and a negative duration is worse than none. One ordering fact drove the design. `commit-msg-hook` consumes `hook-runtime`'s result but presents before it, so the dependency runs backwards against the report order and cannot be satisfied by running earlier entries and reading their output. Memoising the computation keeps "each check runs exactly once" true without reordering anything a user reads. That edge is deliberately not declared as a dependency: §2 admits only earlier entries, and declaring a backward edge to make the data look complete would put a lie in the structure the fix plan will later walk. Effect injection is not here. §2.3 names five effects and only the clock is injected; threading the rest through thirteen checks touches all of them, and a half-injected context is worse than none because a test that stubs it would then exercise an arbitrary subset. Split to #476 with that reasoning. Limit: the registry is data but nothing filters it yet, so the ordering guarantees are tested and unused until the --only ticket Ruled-out: declaring commit-msg-hook -> hook-runtime now | the edge points backwards against report order, and a declared dependency that the emission order cannot satisfy is a lie in the structure the fix plan walks Ruled-out: injecting git and spawn alongside the clock in this change | it rewrites every check while the text must stay byte-identical, and partial injection makes a stubbed-context test cover an arbitrary subset Ruled-out: letting a throwing check abort the report | the twelve remaining answers are worth more than the one missing, and the containment row says which subsystem went unreported rather than implying it passed Warn: containment turns a crash into a `fail`, so a check that throws on every run now reports a steady failure rather than an obvious one -- the evidence line is the only thing distinguishing them Blast: local Undo: easy Certainty: firm Verified: 102 files and 2,259 cases pass with the text and check-order snapshots unchanged; ids are unique and kebab-case, every category is populated, every declared dependency resolves to an earlier entry, every row carries a whole non-negative durationMs, and a check forced to throw yields one fail row with its first message line while the other twelve still render Provenance: authored Record-Id: r-registry463
CI disagreed with the local run on two rows, and the disagreement was the interpreter's path: nvm under a home directory here, hostedtoolcache under /opt on the runner. The report was identical; the snapshot was a record of where it was first generated. This is the second time this instrument has failed for a reason that has nothing to do with what it guards -- the first was mkdtemp's random suffix. Both are the same mistake in different clothes: normalising the values a reader cares about and leaving in the ones that vary per machine. A snapshot that cries wolf gets deleted, and this one has to survive twelve more tickets. Limit: normalisation is a list of known-variable things, so the next machine-specific value to appear in a detail string will fail once before it is added Ruled-out: dropping the two rows that mention the interpreter | they carry the commitlore.bin and commitlore.node pairing, which is exactly what #382 and #71 were about Warn: the node path is replaced by exact match, so a detail string that renders the interpreter differently -- relative, or through a symlink the check did not resolve -- would slip past and reintroduce the drift Blast: local Undo: easy Certainty: firm Verified: the regenerated snapshot contains no home directory, no nvm path and no hostedtoolcache path; thirteen cases pass locally Provenance: authored Record-Id: r-snapnode
Give doctor a check model and a registry, and stop one throwing check taking the rest
200 rows, 100 per arm, seeds 11 through 20, produced by one harness at 788a9db against one model. Section 6 exposure is complete on all 200. No row stopped with an error. The 44 rows that did not reach a natural stop are kept rather than dropped: 40 over-turns and 4 over-tokens. Deviations 1 and 2 already oblige the verdict to report truncation per arm, and a shard that quietly excluded its truncated rows would make that obligation unmeetable. This is deviation 3's second replacement shard. The 400 rows lost to a temp reaper are now 400 rows re-run and committed, which is also why this lands the moment the shard finished instead of at the end of the study. The registered analysis still refuses: seeds 55-58 are outstanding, and it says so by name rather than by computing a table on 1,080 of the registered 1,160. Limit: this shard proves nothing on its own -- it is 200 of 1,160 and the stopping rule is the only thing entitled to read them together Ruled-out: dropping the 44 truncated rows to make the shard look clean | deviations 1 and 2 require per-arm truncation in the verdict, and a shard that excludes them cannot support that line Ruled-out: holding all shards and committing once at the end | that is the practice that lost 400 rows to a temp reaper, and the fix was durable storage per shard rather than more care Warn: 22% of this shard is over-turns against 7% in the first re-run shard, so the per-arm truncation split matters more than it looked when the deviation was written Blast: local Undo: easy Certainty: firm Verified: 200 rows, cond split 100/100, exposure complete 200/200, one harness commit and one model across the shard, and the registered analysis refuses by naming the missing 55-58 file rather than computing on a partial set Provenance: authored Record-Id: r-m5shard2
An agent handed the repository's active records re-proposed a ruled-out approach in 16 of 580 runs. Without them, 109 of 579. Fisher exact two-tailed p = 0.0000, difference -16.1pp, Newcombe 95% -19.6 to -12.7. The registered threshold was 6.6pp. Appendix A.2 predicted the direction and got the size badly wrong. It argued for an effect smaller than the threshold on three grounds -- that no arm passes --trusted-author so the treatment is its weakest form, that ROUTE-GAP records treatment runs implementing what the injected block ruled out, and that the truncation imbalance shrinks the gap. All three were conservative, and the effect is 2.4x the threshold anyway. The note on how the prediction could fail guessed a control base rate above the 23.3% planned against; it came in at 18.8%, below it, so the size is not explained by the explanation that was offered for it in advance. The truncation asymmetry is reported per arm because deviations 1 and 2 require it, and it points the same way: the control arm truncates at 28.5% against 21.2%, truncation suppresses re-proposal, so it removes control chances rather than manufacturing treatment ones. 16.1pp is a floor with respect to that artefact. What the number does not cover is stated where a reader meets the number. Every record rendered `[claim]` because no arm passed --trusted-author, and the payload's own legend told the agent not to act on it as an order. The `[directive]` tier became reachable only after this run. One model, one harness, ten constructed fixtures, and an oracle that reads the final tree rather than establishing that anything was read. Limit: this measures `[claim]`-graded delivery on one model and ten synthetic tasks, and says nothing about the `[directive]` tier that 0.7.0 made reachable Ruled-out: reporting the difference without the per-arm truncation split | deviations 1 and 2 exist because the arms truncate unequally, and a table that hides it invites the reader to assume they did not Ruled-out: dropping the excluded error row from the account | one row was excluded by the registered rule, and a verdict that mentions 1,160 without mentioning the exclusion is describing a different denominator Warn: the effect is large enough that the obvious next question is whether it survives tasks nobody designed to be discriminative, and these ten were designed to be Blast: local Undo: easy Certainty: firm Verified: 1,160 of 1,160 registered rows, exposure complete on 580 and 579 with zero failures in both arms, one harness and one dist digest across every shard, 80 original cells superseded by re-runs through a rule in the analysis code, and the seven shard production windows printed with re-runs marked Provenance: authored Record-Id: r-verdictm5
Report M5: 2.8% against 18.8%, and a magnitude prediction that was wrong
…ered The README carried a sentence admitting the central question was open: whether an agent handed a decision acts on it. That sentence has been true since the first release and is now false, so it is replaced by the measurement rather than left standing beside it. 2.8% against 18.8% across 1,160 registered runs. The three things that make the number worth reading go with it: the 6.6pp threshold was registered before the run, the preregistration predicted a smaller effect and was wrong, and the control arm truncated more, which removes control chances rather than manufacturing treatment ones. Three limits sit in the same paragraph as the number rather than a page away. Every record rendered `[claim]` and the payload told the agent not to act on it as an order, so this measures the weaker of the two tiers and the stronger one became reachable only in this release. It is one model, one harness, ten constructed fixtures. And the oracle reads the final tree: it shows that agents which received records re-proposed less often, not that any of them read anything. Limit: the README now leads its behaviour claim with a `[claim]`-tier number while shipping a `[directive]` tier nobody has measured, and that gap will widen until something measures it Ruled-out: putting the limits in docs/evidence.md and linking them | a reader who stops at the table has then read a claim without its scope, and the scope is what makes the claim survivable Ruled-out: rounding 6.7x into the headline | the ratio is arithmetic on two small counts and moves fast with either; the two rates and the interval are what the study actually bounds Warn: the sentence this replaces was the honest one for four releases, and its replacement will need the same treatment the day a task set nobody designed for discrimination gives a different answer Blast: local Undo: easy Certainty: firm Verified: 82 cases across the readme, compatibility-matrix and manifest suites pass, and the changelog suite accepts the entry Provenance: authored Record-Id: r-readmem5
Ten sites reported `skipped` and every one said why only in prose. A consumer that needs to tell "we did not look" from "we looked and it is fine" -- the distinction the whole model exists to make structural -- had nothing to match but sentences the next release is free to reword. The mapping is the ticket's table, detail strings byte-unchanged. The factory's skipped overload now requires a reason, so a bare skip does not compile. Tightening the overload found a live gap the prose had hidden. `commit-msg-hook` inherits `hook-runtime`'s status, and the compiler pointed out that a skipped runtime would make the inherited row a skip with no reason. `hook-runtime` has no skip site today, so the branch is unreachable -- and it is written out rather than cast away, inheriting the runtime's own reason, because adding one later must not be able to produce a reasonless skip here silently. Limit: the union has six members because ten sites needed six, and the next check that skips will need a seventh rather than one of these stretched to fit Ruled-out: a catch-all `other` member | it is the free-text problem with a type annotation, and a consumer branching on `other` is back to reading the detail string Ruled-out: casting the inherited status to satisfy the overload | the cast compiles today and lies the moment `hook-runtime` gains a skip site, which is exactly when nobody would be looking Warn: `nothing_applicable` covers two squash-conservation conditions that differ -- no candidate branch, and candidates that recorded nothing checkable -- so a consumer cannot tell them apart without the detail Blast: local Undo: easy Certainty: firm Verified: 103 files and 2,265 cases pass with the text snapshot unchanged; every skipped row across two fixtures carries a union member, no non-skipped row carries the key, the serialized report contains no null skipReason, and an unborn HEAD maps to unborn_head end to end Provenance: authored Record-Id: r-skipreason
The README gate rejected the first version of this change, and it was right to. It carried a p-value, a percentage-point figure and a Fisher result typed into prose, beside a generated block that CI regenerates and byte-compares. A statistic retyped into a paragraph drifts from the log that produced it, which is the precise failure the gate exists to prevent -- and the generated block is worth nothing if the paragraph above it carries a hand-written one. The two rates and their counts stay, because they are what the claim is; the significance test, the interval on the difference and the registered threshold move to bench/VERDICT-M5.md, and the README says why they live there. The generated block itself still publishes M4. Repointing it means declaring seven shards with a status note covering supersession, the excluded row and the truncation split, in the register the M4 note set -- deliberate machinery, not a config value, and not something to change between a CI failure and a tag. Filed as #480. Limit: the README's behaviour claim and the generated block below it now describe different studies until #480 lands Ruled-out: repointing README_SOURCES at M5 inside the release | its own comment explains why the declaration is deliberate, and writing that note in a hurry would produce exactly the careless publication it was built to stop Ruled-out: dropping the rates too and linking everything | the two rates are the claim, and a README that states a behaviour result without its magnitude sends every reader to a second document to learn what was measured Warn: the gate matches `p =`, a `pp` figure and `Fisher exact`, so a future statistic in another shape -- an odds ratio, a chi-square -- would pass it and reintroduce the drift Blast: local Undo: easy Certainty: firm Verified: 30 cases across the readme and readme-numbers suites pass, including the stray-statistic scan that rejected the first attempt Provenance: authored Record-Id: r-numgate
Give every skip a reason a machine can branch on
Put the measured number where the README said the question was unanswered
Eighteen version pins move together because the readme suite asserts they match package.json: the two manifests, and four READMEs carrying the install one-liner, the pinned `sh install.sh` invocation, the `git clone --branch` and the inspect-first variant. Splitting them across commits leaves the repository in a state where the documented install points at a tag that does not exist. `dist/` is unchanged. The CLI reads its version at runtime, so the built tree is byte-identical to the one every M5 row was produced against. The release carries the first measurement of the thing the product is for. An agent handed this repository's active records re-proposed a ruled-out approach in 16 of 580 runs against 109 of 579 without them. It also carries the change that makes the number describe the weaker case: `[directive]` was unreachable on every install that has ever existed, and became reachable here, after the run that measured `[claim]`. Limit: the README's behaviour claim now rests on M5 while the generated numbers block beneath it still publishes M4, which is #480 rather than a release-time edit Ruled-out: bumping the manifests first and the README pins after | the readme suite pins them to package.json, so the intermediate commit is one where CI is red and the documented install is wrong Ruled-out: rebuilding dist for the version change | the version is read at runtime, and a rebuild would move the digest every M5 row records without changing a byte of behaviour Warn: between the promotion merge and the tag push, every README install one-liner points at a tag that does not exist -- tag promptly and confirm the URL rather than assuming Blast: system Undo: easy Certainty: firm Verified: 103 files and 2,265 cases pass at this version, no v0.6.0 pin remains in any README, both manifests read 0.7.0, dist has no diff, and the readme, changelog, readme-numbers and compatibility-matrix suites accept the entry Provenance: authored Record-Id: r-rel070
Release 0.7.0
CommitLore — record lintTrailers: clean — 137 commits in Active constraints for the paths this PR touchesLimits (271)
Ruled out (602)
Truncated: 701 lines omitted — the comment hit GitHub's 65000 character limit. Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotion of
devintomainfor v0.7.0. 132 commits.This is the only PR where
promotion-range-lintruns — it validates every record in the promoted range.The headline
1,160 registered runs. Three things about how it was produced matter more than the number:
Method, exclusions, per-arm truncation and what it does not cover:
bench/VERDICT-M5.md.[directive]became reachable — after the run that measured[claim]No installed surface had ever passed
--trusted-author, so grading failed closed and every record every user had ever received was[claim], while the injected legend advertised a tier nobody could reach (#415).initnow records the installing user's git identity; another author's records still grade[claim], so the property that stops a contributor's commit from instructing someone else's agent is untouched.M5 therefore measures the weaker of the two tiers. That is stated beside the number in the README, the changelog and the verdict.
Also in this release
mode: auto, records stampedProvenance: drafted, capped at[claim]until a person promotes them withSupersedes:.docs/SELF-AUDIT.md— what this repository caught in itself, leading with the claims it published that turned out to be false.git push(The pre-push hook hangs every git push: sync's push re-triggers the hook that called it #422), a forgeabledirective(Notes-sourced records inherit the annotated commit author's trust: anyone who can write refs/notes/commitlore can forge a [directive] #409), and a forced refspec that silently destroyed unpushed records (The notes refspec doctor --fix writes is forced, so an ordinary git fetch silently destroys unpublished records #417).Verification at this exact tree
dist/is byte-identical to the tree every M5 row was produced against — the CLI reads its version at runtime.Known, stated rather than hidden
The README's behaviour claim rests on M5 while the generated numbers block beneath it still publishes M4 (#480). Repointing
README_SOURCESneeds a status note written with the care the M4 one took, not an edit squeezed between a CI failure and a tag.After the merge
Between this merge and the tag push, every README install one-liner points at a tag that does not exist. The tag goes on the merge commit immediately, and
RELEASE-GATE.md§4's seven install checks run against a fresh clone before the release is called done.