[Feature] Run tests in a random order

[humbienri]

For your test runner, playwright-test, please add the ability to be able to run the individual tests in a random fashion. I consider this one of those true acid tests for how well a test automation suite and overall architecture is evaluated. It can get rather hard to be watchful of not inadvertently building in dependency between tests and being able to specify a command line switch --random would be a gate to prevent that without exactly spending a whole lot of eyes time on it.

Thanks for the hard work you've been putting in so far.

[mxschmitt]

Linking #7040

[pavelfeldman]

This is unlikely to appear due to the reasons described in #7040. Leaving it open to collect voices. Once again, in every project I worked on, including Chromium, this has been enabled and then inevitably disabled. While great in theory, it never actually worked.

[dimkin-eu]

maybe a simple solution - run them in reverse order? Not a random, but still dependent tests will fail

[humbienri]

Pavel Feldman (@pavelfeldman), it never worked or it was never used? Or are you saying that when the tests were ran in random order, there would be failures and so folks would decide to just run them in order?

[pavelfeldman]

It is less about 'in order' and more about repeatability of the tests. The cost of non-determinism is just too high.

[Meemaw]

Pavel Feldman (@pavelfeldman) we have a very uneven test execution length between tests. Coupling that with sharding, it results into some shards to finish in < 2 minutes, and some taking 15 minutes.

This happens because most of the "expensive" tests are in same file which will fall into same shard.

Randomising/shuffling tests would partially mitigate this, and help us big time, unless you have any other ideas on how to solve this problem.

[Meemaw]

cc Max Schmitt (@mxschmitt) in case you have any ideas how to tackle ☝️ problem.

[Meemaw]

Wondering if you would be open to contributions for this feature? We'll have to add it somehow, and we really wouldn't want to fork playwright for a feature I think could be beneficial to others.

[zyulyaev]

Given fullyParallel option which IMO contradicts your statement in #7040, I don't see any reason not to allow running tests in random order. We have a very similar issue to the one Matej Šnuderl (@Meemaw) described. Our tests depend on the feature flags values, and in some environments, a lot of tests are skipped because of the features being disabled. So the load is distributed extremely unevenly between the shards.

[zyulyaev]

I created a patch using patch-package as a hotfix to this issue. Really hope to see this as a native feature though.

[AlexDaniel]

Pavel Feldman (@pavelfeldman) we have a very uneven test execution length between tests. Coupling that with sharding, it results into some shards to finish in < 2 minutes, and some taking 15 minutes.

Same. This is not strictly a problem with sharding, but with parallelization in general. Some tests can run longer than others, and not everything is strictly bound by CPU. To achieve the best wall clock time you have to take into consideration how fast/slow each test runs.

Back in the day, prove tool for perl5 tests had a slow option to run slow tests first. This was based on the results of the previous run. I wish something like this existed in Playwright too, then it'd be able to optimize both the parallelization and sharding. Note that there are many options for handling the stats file – you can either commit it to the repo from time to time, or use CI features to cache it from the previous run.

[Meemaw]

I created a patch using patch-package as a hotfix to this issue. Really hope to see this as a native feature though.

Nikita Zyulyaev (@zyulyaev) does this also shuffle tests from within a single file across multiple shards? We have most of our "expensive" tests in a single file, and when we tried patching Playwright to shuffle the tests, could only get it shuffled on the group/file level.

[zyulyaev]

I created a patch using patch-package as a hotfix to this issue. Really hope to see this as a native feature though.

Nikita Zyulyaev (@zyulyaev) does this also shuffle tests from within a single file across multiple shards? We have most of our "expensive" tests in a single file, and when we tried patching Playwright to shuffle the tests, could only get it shuffled on the group/file level.

Matej Šnuderl (@Meemaw) I believe so. We have a very similar issue to yours, and we've been successfully using this patch for several weeks now.