[Feature]: Opt-in flag to exclude statically skipped tests when calculating CI shard distribution

[michael-nishiguchi]

🚀 Feature Request

Currently, Playwright's test runner calculates shard distribution across the entire test suite before filtering for test status. When a repository contains a non-trivial number of skipped tests (e.g., test.skip(), conditional skips per platform, or temporarily disabled suites), these skipped tests are still allocated to shards as standard units of work.

Because statically skipped tests finish in milliseconds, shards that happen to receive a higher concentration of them finish almost immediately, while other shards receive mostly active tests. This imbalance causes CI runtimes to be bounded by the single slowest, overburdened shard, effectively diminishing the benefits of CI parallelization.

I'd like to work on this, if this is something the team agrees with.

Example

Introduce an opt-in configuration option in playwright.config.ts (and a matching CLI flag) to filter out statically determinable skipped tests before the sharding algorithm assigns tests to shards:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  shard: {
    // Current behavior remains default: 'all'
    // Proposed opt-in mode:
    mode: 'exclude-skipped', // or a boolean flag like excludeSkippedTests: true
  },
});

Or via the CLI:

npx playwright test --shard=1/4 --exclude-skipped-from-shards

Example

No response

Motivation

Predictable Shard Balancing: Teams running large test suites across matrix CI jobs can opt in to achieve significantly more uniform shard runtimes and lower total wall-clock CI duration.

Zero Breaking Changes: By making this opt-in, repositories relying on deterministic test indexes, exact total counts across combined blob/HTML reports, or specific shard indices will see no change in behavior by default.

Caveats to Consider: Tests dynamically skipped at runtime inside the test body (test.skip(condition)) cannot be predicted during pre-execution planning and would naturally still run on their assigned shard. This proposal targets tests marked with top-level annotations that are resolved during test discovery.

Note on contribution: I intend to work on this feature myself. If the core team is aligned with this approach, please assign this issue to me and I will proceed with developing the PR! (I understand from the contributing guidelines to wait for official approval/assignment before opening a PR).