test(web): compare work log update cost against a from-scratch derive - #102
Merged
Conversation
The 20,000-activity benchmark asserted a 100 ms wall-clock bound. On the 4-vCPU GitHub-hosted runners that replaced Blacksmith it measured 100.5 ms and 107.5 ms on two of the first three runs, for work that takes about 4 ms on a quiet laptop: the bound was measuring runner contention, not the code. Assert the property the optimization actually provides instead: an update over cached rows must be far cheaper than deriving the same rows from scratch, measured back to back in the same process and taking the best of three rounds so a single GC pause or scheduling stall cannot fail it. Locally the ratio is 0.13-0.27; without the per-activity cache both calls walk the same path and the ratio is about 1, so the bound of 0.5 has margin on both sides.
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
apps/web/src/session-logic.test.ts: the 20,000-activity work-log benchmark no longer asserts a 100 ms wall-clock bound. It now measures, in the same process, an update over cached rows and a from-scratch derivation of the same rows, takes the best of three rounds for each, and asserts the update is under half the from-scratch time.Why
Since #99 moved CI to GitHub-hosted standard runners (4 vCPU, four Vitest workers sharing them), the old bound failed on two of the first three runs — 100.5 ms and 107.5 ms — for work that takes ~4 ms on a quiet laptop. It was measuring runner contention, not
deriveWorkLogEntries, and every unrelated PR (#100, #101) had to be rerun to get green.The optimization the test was written for (upstream #8006, cherry-picked as
b49ad7918) is a per-activityWeakMapcache so an update does not re-derive every row. Measured here:So the ratio the test now asserts is ~0.13–0.27 with the cache and ~1.0 without it; a bound of 0.5 has margin on both sides and is independent of how fast or contended the machine is. The existing identity test two cases above (
appendedEntries[1]isinitialEntries[1]) still pins exact row reuse deterministically.Verified:
vp test run apps/web/src/session-logic.test.ts— 82 passed;vp run -F @t3tools/web typecheckclean.Checklist
Model: Claude Fable 5 · Harness: Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.