Repository navigation
Run scenarios in parallel workers - #405
Draft
schlessera wants to merge 5 commits into
Draft
schlessera wants to merge 5 commits into
schlessera wants to merge 5 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This was referenced Oct 30, 2026
Setting WP_CLI_TEST_PARALLEL to a number of workers spreads the scenarios over that many Behat processes. Every worker but the first gets its own temporary directory, and with it its own home directory, caches and run directories, as well as its own test database. Scenarios relying on resources shared across the system, such as fixed ports or fixed paths in the system's temporary directory, all run in the first worker. Scenarios are distributed by their expected cost, so that the workers finish at about the same time, and the ones that fail are recorded so that rerun-behat-tests runs exactly those again. composer prepare-tests now allows the test user to create the databases of the workers. Without that permission, and on Windows, when collecting code coverage or when selecting scenarios by path or name, the scenarios still run one after the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run Behat through the PHP binary, as a shell script cannot be run directly on Windows, and do not hand the run to the parallel runner there at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rerun of a parallel run has to go through the parallel runner, which knows which scenarios failed. Otherwise Behat's own rerun cache, which does not match the workers' arguments, runs every scenario again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
schlessera
force-pushed
the
perf/parallel-scenarios
branch
from
October 31, 2026 11:17
0a546f4 to
79d8dda
Compare
This was referenced Oct 31, 2026
This was referenced Oct 31, 2026
The database check before a run required the test database to exist, while the scenarios drop it themselves. After a run in parallel workers, the rerun of the failed scenarios therefore usually switched to SQLite, even when testing against MySQL. Check that the test user may create the database instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 31, 2026
The processes launched by the tests only received TEMP and TMP, not TMPDIR, so they used the system's temporary directory even when the tests ran in a parallel worker with its own. WordPress saves downloads there under names taken from the response, such as twentytwelve.4.9.zip, so workers installing the same theme at the same time could remove each other's download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #390.
Scenarios ran strictly one after another, although almost all of them are independent of each other and the CI runners have several cores.
Setting
WP_CLI_TEST_PARALLELto a number of workers now spreads the scenarios over that many Behat processes. The scenarios are selected with the same tag filter as before, so a parallel run covers exactly the scenarios of a serial one.wp_cli_test_1,wp_cli_test_2, ...).composer prepare-testsnow allows the test user to create these databases.composer behat-rerunre-runs exactly the scenarios that failed in the last parallel run.Locally, this package's own suite ran the same 97 scenarios and 393 steps with the same results as a serial run, in 27 instead of 47 seconds with 6 workers. A subset of 213
entity-commandscenarios went from 491 to 140 seconds with 8 workers in an earlier experiment.Parallel runs stay opt-in. Measurements on CI are recorded on #390.