From an external review of v0.5.0.
What is proved, and what is not
The README is precise about this and should stay that way.
Proved. Current decisions are distinguished from superseded ones. A record does not attach to the wrong commit. Untrusted context is graded rather than trusted. Decisions survive rebase, squash and rename.
Not proved. That an agent using CommitLore needs fewer corrections, writes fewer redundant helpers, spends fewer tokens, or shortens review. M4 recorded no guard_exposure, so treatment exposure is unverifiable — the README says so.
Why this is now the bottleneck
Gate B closed the last technical gap. What remains between "a well-built tool" and "a tool worth adopting" is not another feature; it is one measurement nobody has taken.
The first measurement, and only this one
Not a productivity study — those need controls this project does not have. Something narrower and decidable:
Given a repository with recorded decisions, how much of the currently-active decision set does a fresh agent recover before its first edit?
It is checkable because the correct answer is knowable: the repository holds the records, so recall against them is countable, and a record that was superseded and still surfaced is countable as an error.
Only if that number is good do the downstream questions become worth their cost:
- corrections per task
- work done on a path already ruled out
- review comments about decisions already made
- tokens
- new helpers and compatibility branches per change
Why the order matters
If a fresh agent does not recover the decisions, no downstream number will move, and measuring them first would produce a null result that says nothing about the product. Recovery is the mechanism; everything else is its consequence.
Scope note
This is a measurement ticket, not a feature. Its output is a number and the method that produced it, in bench/, with the same provenance rules ADR-0018 sets for the existing benchmarks.
From an external review of v0.5.0.
What is proved, and what is not
The README is precise about this and should stay that way.
Proved. Current decisions are distinguished from superseded ones. A record does not attach to the wrong commit. Untrusted context is graded rather than trusted. Decisions survive rebase, squash and rename.
Not proved. That an agent using CommitLore needs fewer corrections, writes fewer redundant helpers, spends fewer tokens, or shortens review. M4 recorded no
guard_exposure, so treatment exposure is unverifiable — the README says so.Why this is now the bottleneck
Gate B closed the last technical gap. What remains between "a well-built tool" and "a tool worth adopting" is not another feature; it is one measurement nobody has taken.
The first measurement, and only this one
Not a productivity study — those need controls this project does not have. Something narrower and decidable:
It is checkable because the correct answer is knowable: the repository holds the records, so recall against them is countable, and a record that was superseded and still surfaced is countable as an error.
Only if that number is good do the downstream questions become worth their cost:
Why the order matters
If a fresh agent does not recover the decisions, no downstream number will move, and measuring them first would produce a null result that says nothing about the product. Recovery is the mechanism; everything else is its consequence.
Scope note
This is a measurement ticket, not a feature. Its output is a number and the method that produced it, in
bench/, with the same provenance rules ADR-0018 sets for the existing benchmarks.