Add end-to-end performance tests for JSONRPC server across platforms

[karthiknadig]

Summary

Since pet can be run as a JSONRPC server (via pet server), we should have dedicated end-to-end performance tests that exercise the server from a client perspective and measure discovery latency across platforms.

Current State

What exists:

• CI integration tests in .github/workflows/pr-check.yml across Windows, macOS (x86_64/aarch64), and Linux
• Discovery validation tests in crates/pet/tests/ci_test.rs that verify discovered environments are correct
• Sample JSONRPC client in docs/sample.js demonstrating client connection
• Telemetry infrastructure (RefreshPerformance struct) that tracks timing breakdown by locator

What's missing:

The existing tests validate correctness but don't systematically test performance from a client perspective.

Proposed E2E Performance Tests

Test Categories

1. Server Cold Start Performance

   • Measure time from process spawn to first successful JSONRPC response
   • Test with various configure payloads (empty workspace, large workspace, many environment directories)

2. Discovery Latency Tests

   • Full machine scan (refresh with no filters)
   • Workspace-scoped discovery (refresh with searchPaths)
   • Kind-specific discovery (refresh with searchKind)
   • Measure and assert on P50/P95/P99 latencies

3. Resolve Performance

   • Cold resolve (no cache)
   • Warm resolve (with cacheDirectory populated)
   • Batch resolution patterns

4. Concurrent Request Handling

   • Multiple refresh requests (should serialize per existing REFRESH_LOCK)
   • Multiple resolve requests in parallel
   • Mixed workload patterns

5. Platform-Specific Baselines

   • Windows (x86_64, aarch64)
   • macOS (x86_64, aarch64)
   • Linux (x86_64, musl, gnu)

Implementation Options

Approach	Pros	Cons
Rust integration tests	Native, can share code with existing tests	Harder to simulate realistic JSONRPC client
Node.js test harness	Matches VS Code Python extension usage pattern	Separate test infrastructure
Criterion benchmarks	Rust-native, regression detection built-in	Doesn't test full server spawn path
Hyperfine CLI benchmarks	Simple, cross-platform	Limited to CLI, not JSONRPC server

Suggested Metrics

• Discovery duration (returned in RefreshResult.duration)
• Time-to-first-environment (elapsed time until first environment notification)
• Total environments discovered per unit time
• Server startup latency
• Memory usage during discovery (if feasible)

CI Integration

• Add performance test job to pr-check.yml or separate workflow
• Store results as artifacts for comparison
• Consider GitHub Actions benchmark action for regression detection
• Run on representative matrices (at minimum: Windows, Ubuntu, macOS)

Acceptance Criteria

• E2E tests that spawn pet server and communicate via JSONRPC
• Performance baselines established for each platform
• CI job that runs performance tests and detects regressions
• Documentation for running performance tests locally

References

• JSONRPC API spec: docs/JSONRPC.md
• Sample client: docs/sample.js
• Existing tests: crates/pet/tests/ci_test.rs
• Performance telemetry: crates/pet-core/src/telemetry/refresh_performance.rs

[karthiknadig]

Closing this issue - all acceptance criteria have been implemented:

✅ E2E tests that spawn pet server and communicate via JSONRPC

Implemented in crates/pet/tests/e2e_performance.rs with comprehensive test coverage:

• Server cold start performance
• Full refresh performance
• Workspace-scoped refresh
• Kind-specific refresh (Conda, Venv, VirtualEnv, Pyenv)
• Resolve performance (cold vs warm cache)
• Concurrent resolve performance
• Cold vs warm cache comparison

✅ Performance baselines established for each platform

.github/workflows/perf-baseline.yml runs on every push to main for:

• Windows (x86_64-pc-windows-msvc)
• Linux (x86_64-unknown-linux-musl)
• macOS (x86_64-apple-darwin)

✅ CI job that runs performance tests and detects regressions

.github/workflows/perf-tests.yml runs on PRs with:

• Comparison against baseline metrics
• P50/P95/P99 statistical analysis (10 iterations)
• Sticky PR comments with performance reports per platform
• Delta indicators for regression detection

⚠️ Documentation for running performance tests locally

Run locally with:

cargo test --release --features ci-perf --test e2e_performance -- --nocapture