Version
1.49.1
Steps to reproduce
I encountered an issue while using Playwright with the maxFailures configuration in our CI pipeline. Here's the scenario:
- We have
maxFailures set to 3.
- When running 70 tests, if the 4th test fails 3 times, Playwright stops execution as per the
maxFailures threshold.
- The
last-run.json file then only contains information about the failed tests (in this case, the 4th test).
- When rerunning the job with the
--last-failed flag, Playwright only runs the failed test(s) and skips the rest of the tests that were not executed in the initial run.
Is this the expected behavior of Playwright with maxFailures?
Are there alternative configurations or recommendations to ensure all tests are executed in reruns while still using the maxFailures feature?
Expected behavior
I would expect the --last-failed flag to respect the test configuration and include skipped tests from the initial run in the rerun. This would ensure comprehensive test coverage during subsequent executions.
Actual behavior
When rerunning the job with the --last-failed flag, Playwright only runs the failed test(s) and skips the rest of the tests that were not executed in the initial run.
This behavior creates an issue because skipped tests remain untested during reruns.
Additional context
No response
Environment
System:
OS: macOS 15.2
CPU: (10) arm64 Apple M1 Pro
Memory: 81.59 MB / 16.00 GB
Binaries:
Node: 20.12.2 - ~/.nvm/versions/node/v20.12.2/bin/node
npm: 10.5.0 - ~/.nvm/versions/node/v20.12.2/bin/npm
pnpm: 9.14.2 - /opt/homebrew/bin/pnpm
Languages:
Bash: 3.2.57 - /bin/bash
npmPackages:
@playwright/test: ^1.49.1 => 1.49.1
Version
1.49.1
Steps to reproduce
I encountered an issue while using Playwright with the
maxFailuresconfiguration in our CI pipeline. Here's the scenario:maxFailuresset to 3.maxFailuresthreshold.last-run.jsonfile then only contains information about the failed tests (in this case, the 4th test).--last-failedflag, Playwright only runs the failed test(s) and skips the rest of the tests that were not executed in the initial run.Is this the expected behavior of Playwright with
maxFailures?Are there alternative configurations or recommendations to ensure all tests are executed in reruns while still using the
maxFailuresfeature?Expected behavior
I would expect the
--last-failedflag to respect the test configuration and include skipped tests from the initial run in the rerun. This would ensure comprehensive test coverage during subsequent executions.Actual behavior
When rerunning the job with the
--last-failedflag, Playwright only runs the failed test(s) and skips the rest of the tests that were not executed in the initial run.This behavior creates an issue because skipped tests remain untested during reruns.
Additional context
No response
Environment
System: OS: macOS 15.2 CPU: (10) arm64 Apple M1 Pro Memory: 81.59 MB / 16.00 GB Binaries: Node: 20.12.2 - ~/.nvm/versions/node/v20.12.2/bin/node npm: 10.5.0 - ~/.nvm/versions/node/v20.12.2/bin/npm pnpm: 9.14.2 - /opt/homebrew/bin/pnpm Languages: Bash: 3.2.57 - /bin/bash npmPackages: @playwright/test: ^1.49.1 => 1.49.1