ci: parallelise the test suite and stop paying for superseded runs

[padak]

What this is

CI cost and latency work. No product code changes — src/ is untouched.

The measurement

A PR run today (measured on run 32669264298, typical):

Job	Runner	Wall clock	Multiplier	Billed min
check	Linux	0:34	1×	1
test (3.12)	Linux	9:05	1×	10
test (3.13)	Linux	7:10	1×	8
build-windows	Windows	9:28	2×	20
		~9.5 min		39 min

Volume over the 13 days to 2026-08-23: 300 runs (219 PR + 81 main push) → ~692/month → ~27,000 billed minutes/month.

Step-level timing says the entire bill is one thing — pytest running single-threaded, three times per PR:

test (3.12):    setup  21s  +  pytest 516s
test (3.13):    setup  15s  +  pytest 402s
build-windows:  setup  72s  +  pytest 477s

Setup is noise next to 1,395s of serial pytest.

Changes

1. pytest-xdist + -n auto on all three test jobs

The ~6k tests are in-process CliRunner invocations with every HTTP call mocked and no shared mutable state, so they parallelise cleanly. Measured locally:

configuration	time	CPU
sequential, no coverage	152 s	85 %
sequential + coverage (what CI runs today)	230 s	91 %
-n 4 (matches a 4-vCPU GitHub runner)	37 s	337 %
-n 4 + coverage	68 s	331 %
-n auto (11 cores)	29 s	727 %

Verified stable: three consecutive full runs (6042 passed every time) plus three repeated runs of the suites most likely to break under parallelism — test_auth_pkce.py (binds loopback ports), test_token_provider.py and test_job_idempotency_store.py (cross-process locks). No port or lock collisions.

Coverage still combines correctly across workers — totals are identical to a sequential run (31296 / 5405 / 9186 / 1121, 80 %).

The Windows full-suite step was 477s of that job's 568s and bills at 2×, so it was the most expensive single step in the pipeline. If it ever turns flaky on Windows specifically, the comment in place says to swap -n auto for -p no:xdist rather than disable the suite — Windows-only file-locking and atomic-replace semantics are exactly what it is there to catch.

2. concurrency with PR-only cancel-in-progress

ci.yml and frontend.yml had no concurrency group, so pushing again to an open PR left the previous run to finish and bill in full. Measured: 29 of 219 PR runs (13 %) were still executing when the next push landed; branch ms/dmd-1833 alone accumulated 23 runs (~875 billed minutes).

Main pushes are deliberately never cancelled — that run is the post-merge verification of an already-shipped commit, and cancelling it because the next merge arrived would drop the signal for the commit in between.

3. Coverage on main pushes only

It costs ~50 % extra wall clock and enforces no --cov-fail-under, so it could never have blocked a PR. Locally it stays one make test-cov away.

4. 3.13 on main pushes only

PRs run 3.12 (the requires-python floor and what the wheel is built on); main runs the full matrix. A 3.13-only regression is caught on main instead of on the PR. Accepted trade — reversible with a one-word change to the matrix expression, which the comment says.

5. Three redundant Windows steps removed

-k export (#529), test_update_runner.py (#528) and test_job_idempotency_store.py (#427) each ran as a named step immediately before the full suite that contains them — duplicated Windows minutes at 2×. Their rationale is folded into the full-suite comment; a failure now names the test node id instead of the step, which is the same information.

6. uv dependency cache

enable-cache keyed on uv.lock, on all four jobs.

7. make test / test-unit / test-cov get -n auto

152s → 29s locally. make check end-to-end went from ~3.5 min to 53 s. -v is dropped with it — four workers interleaving 6k lines is unreadable, and a failing test prints its own node id. make test-file stays sequential and verbose for single-file debugging.

Projected result

billed min / run:    39  ->  ~15    (-62 %)
wall clock / run:   9.5  ->  ~4 min
billed min / month:  27,000  ->  ~10,400

Considered and rejected: path filtering for docs-only PRs

A first pass suggested ~19 % of PRs were "docs/plugin-only" and could skip the Python jobs. Checking which repo paths the suite actually reads from the live tree killed it:

• plugins/** — read by test_agent_prompt.py (prompt byte budget), test_skill_frontmatter.py, test_mcp_migration_recipe.py, test_license_consistency.py; test_migrate_cicd.py and test_generate_promotion_pipeline.py import scripts from under it
• README.md, LICENSE, build/package/chocolatey/*.nuspec — test_license_consistency.py
• CLAUDE.md — test_check_command_sync.py::test_live_command_tree_has_no_drift runs the gate against the live tree
• scripts/** — many

Once those are excluded, the genuinely safe skip set is docs/** and .claude/**, and zero of the last 58 commits on main qualify. Real drift risk, no measured benefit — left out.

Verification

• Both exact CI commands run green locally, including the Windows job's command (which also picks up the integration-marked tests): 6042 passed and 6045 passed
• make check green end-to-end
• Both workflow files parse; matrix expression and step conditions inspected after parsing
• Rebased onto current main; the Serve endpoint reference freshness check step from docs(serve): generate the endpoint reference from the app, gate it in CI (#656) #671 is intact in the check job

Not verified here

-n auto on the Windows runner. I have no Windows machine, so the 477s → ~150s projection for that step is inference from the Linux measurement. First run on this PR will show it — if it is flaky rather than fast, the fix is the one-line fallback documented in the comment.

---

[devin-ai-integration]

Devin Review found 2 potential issues.