Skip to content

Add opt-in parallel testing; fix registry and fixture isolation#539

Merged
tony merged 12 commits into
masterfrom
perf/pytest-optimizer
Jun 28, 2026
Merged

Add opt-in parallel testing; fix registry and fixture isolation#539
tony merged 12 commits into
masterfrom
perf/pytest-optimizer

Conversation

@tony

@tony tony commented Jun 28, 2026

Copy link
Copy Markdown
Member

Summary

  • Add opt-in parallel test execution via pytest-xdist (new dev dependency) with a just test-parallel recipe (uv run py.test -n auto). The default uv run pytest is unchanged and still runs serially — parallelism is purely opt-in.
  • Fix VCSRegistry sharing parser state across instances. Its parser map was a class variable, so constructing any second registry mutated the shared registry singleton (replacing its git/hg/svn parsers). Each registry now owns its own parser map.
  • Fix the git_repo / hg_repo pytest fixtures leaking state between tests. The first consumer of either fixture received a live handle to the session-cached master checkout, so mutating that checkout (adding a remote, switching branches) polluted the cache for every later test. The master copy is now a pristine, read-only cache and every consumer — including the first — gets its own copy.
  • Net effect: the suite is now order-independent and safe to run under xdist. The two isolation fixes also matter to downstream consumers of VCSRegistry and the pytest fixtures (e.g. vcspull), independent of parallelism.

Changes by area

Parallel testing (opt-in)

  • pyproject.toml: add pytest-xdist to the dev and testing dependency groups.
  • justfile: add a test-parallel recipe — uv run py.test -n auto.
  • .gitignore: ignore the .pytest-optimizer/ profiling-state directory.

Test-isolation fixes

  • src/libvcs/url/registry.py: initialize parser_map as an instance attribute in __init__ instead of a ClassVar, so each VCSRegistry is independent and the module-level registry is never mutated by constructing another.
  • src/libvcs/pytest_plugin.py: build the git_repo / hg_repo master checkout once as a pristine read-only cache, then always return an isolated copytree of it — the first consumer no longer receives a live handle to the cache.
  • src/libvcs/url/base.py: unregister the demo rule at the end of the RuleMap.register doctest, matching the git/hg/svn examples, so it doesn't leak into the global GitURL parser.

Design decisions

  • Parallelism is opt-in, not the default. The default uv run pytest stays serial and stable. The worker count is left to the operator (-n auto) rather than hardcoded in addopts, because the optimal count is machine-specific (high-core machines oversubscribe a subprocess-bound suite).
  • Fix isolation at the source, not by constraining test order. Rather than pin test order or use xdist --dist grouping to paper over the coupling, the registry and fixtures were made genuinely isolated. This is the more robust fix and it also corrects real bugs for downstream callers.

Before / after — VCSRegistry

Before, a second registry clobbered the global one's parsers:

from libvcs.url.registry import registry, VCSRegistry
VCSRegistry({"git": MyGitURLParser, ...})
registry.match("git+https://github.com/o/r")  # now matched by MyGitURLParser

After, each registry is independent; the global registry is untouched.

Verification

The suite is order-independent (run shuffled across several seeds):

$ uv run --with pytest-randomly pytest -p randomly -p no:cacheprovider

Parallel execution is green:

$ uv run pytest -n auto

Test plan

  • uv run ruff format . and uv run ruff check . — clean
  • uv run mypy — clean (strict, covers src and tests)
  • uv run pytest — default serial suite green and unchanged
  • uv run pytest -n auto — parallel run green
  • uv run --with pytest-randomly pytest -p randomly — green across multiple seeds (suite is order-independent)

tony added 3 commits June 28, 2026 09:15
why: The pytest-optimizer pipeline writes durable analysis state under
.pytest-optimizer/; it is tooling output, not source.

what:
- Ignore the .pytest-optimizer/ state directory
why: Enable optional parallel test execution for the subprocess-bound
suite; pytest-xdist is the runner that provides it.

what:
- Add pytest-xdist to the dev and testing dependency groups
- Regenerate uv.lock
why: Give contributors a one-word entry point for parallel runs without
memorizing the xdist flags.

what:
- Add `test-parallel` recipe: uv run py.test -n auto
@codecov

codecov Bot commented Jun 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.23077% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.18%. Comparing base (fa740d3) to head (68d19d1).

Files with missing lines Patch % Lines
src/libvcs/pytest_plugin.py 66.66% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #539      +/-   ##
==========================================
- Coverage   61.19%   61.18%   -0.01%     
==========================================
  Files          40       40              
  Lines        6563     6557       -6     
  Branches     1103     1103              
==========================================
- Hits         4016     4012       -4     
+ Misses       1950     1948       -2     
  Partials      597      597              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tony

tony commented Jun 28, 2026

Copy link
Copy Markdown
Member Author

Code review

Found 1 issue:

  1. The three new #### deliverable headings in the unreleased changelog block omit a PR ref (#539), unlike every other entry in the file (e.g. #### Command output is returned verbatim by default (#538)). (AGENTS.md says "PR refs (#NN) sit in each deliverable's #### heading.")

libvcs/CHANGES

Lines 23 to 42 in e918163

### Fixes
#### `VCSRegistry` no longer shares parser state across instances
{class}`~libvcs.url.registry.VCSRegistry` kept its parser map in a class
variable, so constructing any second registry mutated the shared
{data}`~libvcs.url.registry.registry`. Each registry now owns its own parser
map, leaving the global one untouched.
#### `git_repo` and `hg_repo` fixtures return isolated clones
The first consumer of the `git_repo` / `hg_repo` pytest fixtures received a live
handle to the session-cached master checkout, so mutating that checkout (adding
a remote, switching branches) polluted the cache for every later test. The
master copy is now treated as a pristine, read-only cache and every consumer —
including the first — gets its own copytree.
### Development
#### Opt-in parallel test runs

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

tony added 4 commits June 28, 2026 10:17
why: Document the new contributor-facing parallel test capability.

what:
- Add a Development entry describing pytest-xdist + test-parallel
why: parser_map was a class variable shared by every VCSRegistry, so
constructing a second registry (as the registry docs example does)
mutated the global `registry` singleton -- replacing its git parser.
This surfaced as order-dependent test failures, but it is a real bug
for any downstream caller that builds a custom registry.

what:
- Initialize parser_map as an instance attribute in __init__
- Drop the ClassVar declaration
why: The first consumer of git_repo/hg_repo got a live handle to the
session-scoped master_copy, so its mutations (e.g. adding a remote)
leaked into every later test that copied the cache. Surfaced as
order-dependent failures; a real isolation bug for plugin consumers.

what:
- Build master_copy once as a pristine read-only cache
- Always return an isolated copytree, including for the first consumer
- Apply the same fix to git_repo and hg_repo
why: The RuleMap.register doctest added 'gl-prefix' to the global
GitURL.rule_map without removing it, unlike the git/svn/hg examples
which clean up. The leaked rule persisted into later tests.

what:
- Unregister 'gl-prefix' at the end of the example
@tony tony force-pushed the perf/pytest-optimizer branch from e918163 to e088abc Compare June 28, 2026 15:17
why: The svn_repo cache was dead. The cache-miss branch checked out to
projects_path instead of master_copy, so master_copy was never written,
the cache never hit, and new_checkout_path/unique_repo_name were unused;
every svn test re-ran a full checkout. (Isolation was unaffected since
projects_path is function-scoped.)

what:
- Build master_copy once via obtain(), then copytree to an isolated
  checkout for every consumer, mirroring git_repo/hg_repo
@tony tony force-pushed the perf/pytest-optimizer branch from fe7ab57 to f8104de Compare June 28, 2026 15:36
tony added 4 commits June 28, 2026 10:47
why: The workflow guide's Tests section listed only serial runs; the new
opt-in xdist mode (just test-parallel) and the suite's order-independence
expectation were undocumented.

what:
- Add a "Running tests in parallel" subsection (just test-parallel / -n auto)
- Add an "Order independence" subsection with a shuffled-run check
why: The git_repo/hg_repo/svn_repo per-test isolation guarantee lived only
in inline comments, so downstream consumers (e.g. vcspull) could not see it
from the docstrings or rendered docs. A few neighboring fixture docs were
also stale.

what:
- Document the per-test isolation contract in the three repo fixtures'
  docstrings and add a "Repository isolation" note to the plugin doc page
- Correct the inaccurate "session-scoped" card in docs/api/index.md
- Fix the git_remote_repo docstring and the "Emphemeral" typo
why: The custom-registry example never stated that building a custom
VCSRegistry leaves the module-level registry untouched -- the exact
property whose absence was a recently fixed bug.

what:
- Note that subclassing with a local RuleMap is isolated, vs registering
  on GitURL.rule_map which mutates shared class state
- Add a doctest asserting the global registry still resolves git to GitURL
  after a custom registry is built (regression guard)
why: Record the decision behind the test-isolation fixes and the opt-in
xdist mode so the rationale and rejected alternatives survive.

what:
- Add ADR 0002 (order-independence invariant + opt-in parallelism, with
  alternatives, consequences, and prior art)
- Register it in the ADR index toctree
@tony tony merged commit 58c399e into master Jun 28, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant