Skip to content

ci: shard Vitest and run browser jobs in the pinned Playwright image - #466

Closed
kalvinnchau wants to merge 5 commits into
mainfrom
peon/ci-vitest-shards-browser-container
Closed

kalvinnchau wants to merge 5 commits into
mainfrom
peon/ci-vitest-shards-browser-container

Conversation

@kalvinnchau

@kalvinnchau kalvinnchau commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What

Two CI changes, in one PR so their effect can be watched together.

  1. Vitest runs in three shards. A new vitest matrix job runs vitest run --shard=N/3 on three runners. JavaScript keeps Biome, the TypeScript check and the frontend build, so those still run once. Each shard keeps BUZZ_TEST_WORKERS=2. CI required now needs vitest.
  2. Browser jobs run in the pinned Playwright image. These are the four CI commits from feat: add inbox conversations and draft previews #422 by @tulsi-builder, cherry-picked with authorship preserved (3314dfbc, cf4a66be, 8f4346a0, b07233d9; import-block conflicts with Show agent activity in navigation #423 in browser-ci.test.mjs resolved) so they can land separately from the inbox feature. measurements and browser run in mcr.microsoft.com/playwright:v1.63.0-noble pinned by digest, with no apt-get or playwright install. A new browser_fixture job builds fixture-bridge once; the twelve shards download and verify it instead of each installing Rust and building it.

No test, assertion, engine or shard count is removed. Playwright workers and the 2×6 browser matrix are unchanged.

Why

Hosted CI on main, 77 successful first-attempt runs between 2026-09-29T21:29Z and 2026-09-30T17:42Z:

p50 p90
Run wall time 8.95 min 10.8 min
JavaScript job 8.3 min
WebKit journey shard 7.8 min
  • The Vitest step is p50 452 s of the JavaScript job, on one runner with two workers (run 36671118189, 5d2b08e2: 475 s).
  • Since 14:00Z on 2026-09-30, jobs have been hitting their 15-minute timeout inside apt-get against azure.archive.ubuntu.com. On main every browser job runs apt; after this change only Rust and tool integration does.

Timings

Vitest, local only (Apple M3 Max, Node 24.18.0, at 1c75f6c6), BUZZ_TEST_WORKERS=2 bin/pnpm exec vitest run --shard=N/3 --reporter=json:

Shard Files Tests Wall
1/3 155 1,938 86 s
2/3 154 1,778 83 s
3/3 154 1,982 101 s

The three JSON reports together contain all 463 files from vitest list --filesOnly exactly once (no duplicates, none missing), 5,698 tests, all passing. Hosted estimate, not yet measured: about 3–3.5 min per shard job against 8.3 min today.

Browser jobs, hosted CI. Time from run start to the start of the Functional journeys step, across the twelve shards:

Run Setup path min median max
36671118189 (main 5d2b08e2, healthy mirror) apt + Rust + fixture build per job 70 s 83 s 106 s
36753995236 (main bfe4c7f1, slow mirror) same 572 s 592 s 844 s
36755437303 (#422 982ce489) container + shared fixture 127 s 135 s 151 s

In the #422 run the fixture job took 78 s and each shard then spent 47–71 s before its journeys started. With a healthy mirror, journeys therefore start about 50 s later than on main today, because every shard waits for browser_fixture. With a slow mirror they start several minutes earlier. #422 carries different browser tests, so its journey durations are not comparable to main; the first run on this PR is the like-for-like measurement.

Known risks

  • browser_fixture is on the critical path for all twelve shards. It took 288 s in run 36751775314 (233 s inside Swatinem/rust-cache before it printed its configuration, then a cache miss) and 78 s in run 36755437303. The Rust cache is saved only on pushes to main, so runs on this PR will be cold.
  • In run 36751775314, message-actions.spec.mjs (Chromium 2/6) and sidenav-polish.spec.mjs (WebKit 3/6) failed once. feat: add inbox conversations and draft previews #422 later added 982ce489, which waits for popup and wheel completion in those two specs. It is not included here. It applies cleanly to this branch if the failures reproduce.
  • The vitest job gets its own Hermit cache key, so its first run provisions tools cold.

Validation

  • bin/node --test tests/integration/browser-ci.test.mjs: 11 pass, including the new guard on the shard denominator, worker limit and single sharded invocation.
  • bin/node --test tests/integration/ci-test-report.test.mjs: 5 pass.
  • bin/pnpm exec biome check --error-on-warnings on the changed JS files: clean.
  • The three local shard runs above.
  • .githooks/pre-push run by hand (the worktree hook installer refuses because a global core.hooksPath is set): no JS, design-system or Rust inputs selected.

Deferred checks

  • Hosted timings for the vitest shards and for the browser lane on main's test set: this PR's first CI run.
  • Cold and warm timing of browser_fixture on main after merge.
  • Branch protection: confirm nothing requires the JavaScript check by name for Vitest coverage; CI required covers the new job.
  • Human testing has not been done.

tulsi-builder and others added 5 commits September 30, 2026 11:05
Signed-off-by: tulsi <tulsi@block.xyz>
(cherry picked from commit 3314dfb)
Signed-off-by: peon <9ac6794b000690b7e814eb1805ad32405d0bec7d52838de3a86cf967565dacc0@buzz.block.builderlab.xyz>
Signed-off-by: tulsi <tulsi@block.xyz>
(cherry picked from commit cf4a66b)
Signed-off-by: peon <9ac6794b000690b7e814eb1805ad32405d0bec7d52838de3a86cf967565dacc0@buzz.block.builderlab.xyz>
Signed-off-by: tulsi <tulsi@block.xyz>
(cherry picked from commit 8f4346a)
Signed-off-by: peon <9ac6794b000690b7e814eb1805ad32405d0bec7d52838de3a86cf967565dacc0@buzz.block.builderlab.xyz>
Signed-off-by: tulsi <tulsi@block.xyz>
(cherry picked from commit b07233d)
Signed-off-by: peon <9ac6794b000690b7e814eb1805ad32405d0bec7d52838de3a86cf967565dacc0@buzz.block.builderlab.xyz>
The JavaScript job ran all Vitest files on one runner with two workers
and set the floor for CI wall time together with the WebKit shards. Move
Vitest into its own three-way matrix job so lint, types and the frontend
build still run once. Each shard keeps the two-worker contention limit.
CI required now needs the vitest job, and a guard pins the shard
denominator, the worker limit and the single sharded invocation.

Co-authored-by: peon <9ac6794b000690b7e814eb1805ad32405d0bec7d52838de3a86cf967565dacc0@buzz.block.builderlab.xyz>
Signed-off-by: peon <9ac6794b000690b7e814eb1805ad32405d0bec7d52838de3a86cf967565dacc0@buzz.block.builderlab.xyz>
@kalvinnchau
kalvinnchau marked this pull request as ready for review September 30, 2026 18:33
@kalvinnchau
kalvinnchau requested review from a team, comp615 and wesbillman as code owners September 30, 2026 18:33
@kalvinnchau
kalvinnchau marked this pull request as draft September 30, 2026 18:33
@kalvinnchau kalvinnchau reopened this Sep 30, 2026
@kalvinnchau
kalvinnchau marked this pull request as ready for review September 30, 2026 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants