Skip to content

test(web): compare work log update cost against a from-scratch derive - #102

Merged
rynfar merged 1 commit into
pylonfrom
fix/session-logic-perf-test
Aug 26, 2026
Merged

test(web): compare work log update cost against a from-scratch derive#102
rynfar merged 1 commit into
pylonfrom
fix/session-logic-perf-test

Conversation

@rynfar

@rynfar rynfar commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

What Changed

apps/web/src/session-logic.test.ts: the 20,000-activity work-log benchmark no longer asserts a 100 ms wall-clock bound. It now measures, in the same process, an update over cached rows and a from-scratch derivation of the same rows, takes the best of three rounds for each, and asserts the update is under half the from-scratch time.

Why

Since #99 moved CI to GitHub-hosted standard runners (4 vCPU, four Vitest workers sharing them), the old bound failed on two of the first three runs — 100.5 ms and 107.5 ms — for work that takes ~4 ms on a quiet laptop. It was measuring runner contention, not deriveWorkLogEntries, and every unrelated PR (#100, #101) had to be rerun to get green.

The optimization the test was written for (upstream #8006, cherry-picked as b49ad7918) is a per-activity WeakMap cache so an update does not re-derive every row. Measured here:

this laptop (M5 Pro)
from-scratch derive, 20k rows 30–46 ms
update over cached rows, 20k+1 4–8 ms
derive with cache misses (the pre-#8006 path) 29–34 ms

So the ratio the test now asserts is ~0.13–0.27 with the cache and ~1.0 without it; a bound of 0.5 has margin on both sides and is independent of how fast or contended the machine is. The existing identity test two cases above (appendedEntries[1] is initialEntries[1]) still pins exact row reuse deterministically.

Verified: vp test run apps/web/src/session-logic.test.ts — 82 passed; vp run -F @t3tools/web typecheck clean.

Checklist

  • This PR is small and focused
  • I explained what changed and why
  • I included before/after screenshots for any UI changes (no UI change)
  • I included a video for animation/interaction changes (no motion change)

Model: Claude Fable 5 · Harness: Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

The 20,000-activity benchmark asserted a 100 ms wall-clock bound. On the 4-vCPU GitHub-hosted runners that replaced Blacksmith it measured 100.5 ms and 107.5 ms on two of the first three runs, for work that takes about 4 ms on a quiet laptop: the bound was measuring runner contention, not the code. Assert the property the optimization actually provides instead: an update over cached rows must be far cheaper than deriving the same rows from scratch, measured back to back in the same process and taking the best of three rounds so a single GC pause or scheduling stall cannot fail it. Locally the ratio is 0.13-0.27; without the per-activity cache both calls walk the same path and the ratio is about 1, so the bound of 0.5 has margin on both sides.
@github-actions github-actions Bot added size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. labels Aug 26, 2026
@github-actions

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.3 KiB 13.3 KiB −1 B (−0.0%) 15.1 KiB
Codex Thread snapshot wire 6.9 KiB 6.9 KiB +5 B (+0.1%) 7.3 KiB
Codex Live turn WebSocket wire 6.4 KiB 6.4 KiB −6 B (−0.1%) 7.8 KiB
Codex Live turn WebSocket decoded 55.5 KiB 55.5 KiB 0 B (0.0%) 66.4 KiB
Codex Live turn messages 9 9 0 (0.0%) 21
Claude Total thread wire 13.3 KiB 13.3 KiB −1 B (−0.0%) 15.1 KiB
Claude Thread snapshot wire 6.9 KiB 6.9 KiB +5 B (+0.1%) 7.3 KiB
Claude Live turn WebSocket wire 6.4 KiB 6.4 KiB −6 B (−0.1%) 7.8 KiB
Claude Live turn WebSocket decoded 56.3 KiB 56.3 KiB 0 B (0.0%) 66.4 KiB
Claude Live turn messages 9 9 0 (0.0%) 21

Baseline: 486065e · PR result: a2b1d18 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 109.4 KiB
  • Claude decoded thread snapshot: 110.0 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@rynfar
rynfar merged commit b48577f into pylon Aug 26, 2026
14 checks passed
@rynfar
rynfar deleted the fix/session-logic-perf-test branch August 26, 2026 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant