A rebuild spent its time on work nobody reads (#776) - #790
Merged
Conversation
One `commitlore index --rebuild` started 4,345 git subprocesses. 4,329 of them were `interpret-trailers --parse`; the other sixteen were the whole rest of the work. Forty-five seconds, essentially all of it process startup. Three causes, and each was work whose answer was thrown away. **One process per trailer paragraph.** `parseRecordBlocks` ran `asIsolatedBlock` on every earlier paragraph and *then* dropped any result without a `Record-Id` -- 3,900 parses to keep 33. The raw-text test now runs first: a trailer key is always at the start of a line in the source and folding continues values, so a paragraph that never names the key cannot yield one. `explodeRecordBlocks` also skips messages the atom pass already covers. That second gate was wrong on the first attempt. ADR-0014 says a message with at most one `Record-Id` has exactly one block, so the gate counted -- but that is a statement about how many blocks exist, not about whether the one git handed the atom pass is it. A commit whose final paragraph is `Co-authored-by:` has its record one paragraph earlier, and counting alone dropped nine rows from exactly such a commit. The gate checks position too. **Verifying every signature nobody asked about.** `%G?` makes git verify each commit. Only signature mode reads the answer; `trusted-authors.ts` says so where the verifier generation is defined. Enabling the setting afterwards already rebuilds, by the same mechanism a changed keyring uses (#653). The field is emitted empty rather than dropped so the positional destructure keeps its shape, and an empty string is outside git's `%G?` vocabulary, so it cannot be misread as "no signature" -- which is `N`, and a different fact from "not asked". **A budgeted scan paying for its own deadline checks.** A batch is three `git log` processes however many commits it covers, and holding the budgeted batch at 64 made 177 of them on 10,000 commits against 30 unbounded. The batch now doubles to `LOG_BATCH`: small early so any repository still gets checked often, which is why 64 was chosen, large later so the process cost is paid a handful of times. Also here: `byteSnapshot` in the shadow test read a directory and then stat'd each entry, and git removes its own lock files in between. CI caught it on `maintenance.lock`. Vanishing entries are skipped; a file the shadow run wrote would still be there to be stat'd, so this cannot hide what the test is for. Record-Id: r-rebuildworknobodyreads Provenance: authored Certainty: firm Blast: module Undo: easy Limit: the deadline is still only checked between batches and before the expensive half of one, so a late batch of 1024 commits can overshoot by whatever that batch costs -- bounded by one batch rather than by the whole scan. And an unsigned-mode index now carries `''` where it carried git's verdict, so a reader wanting the cached status without turning signature mode on no longer gets it; none exists Verified: identical output, not merely faster -- every `trailers` row dumped before and after and compared field by field, 7250 rows both times. 45s to 2.5s here; on a synthetic 10,000-commit corpus, over 120s (killed, 9,437 interpret-trailers calls and still going) to 2.2s, with the cold consumer path going from a partial answer to a complete one in 2.9s. Negative controls for each: count-only gate reproduces the nine missing rows, an unconditional `%G?` fails the atom test, a pinned batch of 64 reproduces the incomplete-scan caveat. A first attempt at the 10k comparison showed no difference because the control never ran -- the change was already committed, so `git stash push <paths>` reverted nothing CommitLore-Version: 2.0.0
CommitLore — record lintTrailers: clean — 2 commits in Active constraints for the paths this PR touchesLimits (249)
Ruled out (384)
Truncated: 373 lines omitted — the comment hit GitHub's 65000 character limit. Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
…fire (#776) `readCommitRecords` worked out whether signature mode is on as its first statement, and that costs a `git config`. A scan with nothing to read -- which is every hook fire against an index that is already current -- paid it for an answer it never used, and the read path came out about 10 ms slower than the build it was meant to be faster than. Resolved on the first batch instead. A scan that reads nothing never asks. Found by measuring the hook against `origin/main`'s bundle rather than against the earlier number in my own notes: a single reading looked like 486 ms against a remembered 399 ms, which is the sort of gap a busy machine invents. Eleven runs of each, side by side, put the real difference at 10 ms -- one process, which is what it was. Record-Id: r-lazysignaturemode Provenance: authored Certainty: firm Blast: local Undo: easy Limit: a scan that reads even one commit still asks, so the cost returns on any invocation that has catching up to do. That is the invocation that can afford it Verified: the hook path is now within noise of `origin/main` (365 ms against 368 ms, eleven runs each, both 11/11 producing a real payload) and the rebuild is unchanged at 2.3 s. 304 tests across the scan, query, signature-generation and inject suites CommitLore-Version: 2.0.0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blocked only on CI. Replaces #789, which is the same change — its last commit carried CommitLore trailers without a
Record-Id, which this repository's own dogfooding test rejects. Rebuilt as one properly-recorded commit rather than force-pushing.Closes #776. The 23-second rebuild was not a floor — it was 4,329 git process spawns.
Not closing #776: this removes the reason the number was large, and the remaining product question (below) is unchanged in shape but much smaller.
What one rebuild actually did
The batch reader already avoids one process per commit, and the module says so in its own header:
explodeRecordBlockswent around it.%(trailers)returns only a message's last paragraph, so recovering a squashed earlier block means re-reading the message — and that pass parsed every earlier paragraph to find the ones carrying a second record. On this repository: 3,900 parses to keep 33.Both changes move a cheap test in front of an expensive one already being discarded
parseRecordBlocksranasIsolatedBlockon each earlier paragraph and then dropped any result without aRecord-Id. The raw-text test now runs first — a trailer key is always at the start of a line in the source and folding continues values, so a paragraph that never names the key cannot yield one.explodeRecordBlocksskips the parse entirely for messages the atom pass already covers.The gate was wrong the first time, and how is worth reading
ADR-0014 says "a message with at most one
Record-Idanywhere has exactly one block", so the first gate counted occurrences and skipped at ≤1.That is a statement about how many record blocks exist, not about whether the one git handed the atom pass is it. A commit whose final paragraph is
Co-authored-by:has its record one paragraph earlier —%(trailers)returns the conventional block, and the real record is only reachable through the explode pass. The count-only gate dropped nine rows from exactly such a commit here.The gate now checks position as well as count. The boundary is four tests.
Measured
trailersrowsIdentical, not merely smaller — every row of
trailersdumped before and after and compared field by field. That check is what caught the first gate; the timing alone looked like a win.Negative control: reverting to the count-only gate reproduces the nine missing rows.
What is left on #776
The product question does not go away, it shrinks: what first-use delay is acceptable in a freshly cloned repository, now against ~7 s rather than ~23 s, with
CONSUMER_SCAN_BUDGET_MS = 3_000still the consumer budget.Two further reductions are identified and not taken here:
%G?in the rebuild'sgit logformat costs 2.7 s on this tree and is only needed in signature mode.