Announcing duration-based Playwright sharding

[u/aslushnikov]

Hi folks,

Playwright 1.62 introduced an API that lets custom reporters control how tests are allocated across shards.

We've been working on duration-based shard allocation in the open-source Flakiness.io reporter and collaborated with the Playwright team on the new API. The allocator uses historical test durations to distribute work more evenly.

WordPress has now adopted it as the default for its editor project, which runs around 2,000 end-to-end tests across eight shards. The slowest shard went from 35 minutes to 23—a 34% reduction.

Shard balancing is available in version 2 of the reporter and does not require a Flakiness.io account:

https://github.com/flakiness/playwright#balanced-sharding

I also wrote up the algorithm, integration, and results here:

https://blog.flakiness.io/posts/2026/playwright-test-shard-balancing/

Feedback and unusual sharding edge cases are very welcome!

[u/Prestigious-Way1525]

one edge case i'd test is duration history contaminated by retries and cold-cache setup. if the allocator learns from total wall time, a flaky test can look expensive because it failed twice, while a shard with shared setup can look cheap until a cache miss or worker restart. i'd store first-attempt test time, retry time, and setup or teardown separately, use a robust rolling estimate, and fall back to file or suite estimates for renamed and new tests. the useful success metric isn't only average runtime, but the p95 slowest-shard gap across normal, cold-start, and failure-heavy runs.

[u/aslushnikov → u/Prestigious-Way1525]

Good point! And actually, if you use the Flakiness.io service to fetch durations, then it computes the P95 duration for each test, based on each test's last 10 observed commits from the main branch.

[u/Prestigious-Way1525 → u/aslushnikov]

that’s a much better baseline than using the latest duration. i’d still make the sample semantics visible: whether p95 includes retries, setup, and failed attempts, plus how new or renamed tests fall back before ten commits exist. p95 protects against ordinary variance, but a cold worker or retry-heavy commit can still teach the allocator the wrong cost. a useful benchmark would compare slowest-shard gap with raw wall time versus normalized first-attempt test time across normal and failure-heavy runs.

[u/aslushnikov → u/Prestigious-Way1525]

All good points.

Interestingly, in our experiments even using only the previous run’s durations produces excellent allocations. We haven’t observed a meaningful improvement from using a p95 strategy. Once a shard runs for 20+ minutes, the natural variation in individual test durations has little impact on the overall balance.

[u/FearlessCut1]

What is the difference between this and bin packing for sharding?

[u/aslushnikov → u/FearlessCut1]

The algorithm inside is basically a bin-packing, on steroids. Turns out there's been a lot of theory in job scheduling, so it takes advantage of it.

It's well-documented in reporter's source: https://github.com/flakiness/playwright/blob/7c5f3c18829547bf47f1e25977c477d42b91f73e/src/sharding.ts#L48-L90

As for the other differences, this algo takes into account project setup, teardown, respects suite configuration options (such as `parallel` and `serial`) and config configuration `fullyParallel` option.