Repository navigation
feat: run each benchmark in its own process - #101
not-matthias wants to merge 13 commits into
Conversation
Benchmarks of one process share allocator, JIT and thread state, so what one benchmark leaves behind (raised mmap threshold, fragmented heap, optimized code, retained caches) changes the measurement of every benchmark after it. Adding or changing one benchmark can then shift unrelated results. Add `Isolation` for the integrations. It is on by default and turned off with `CODSPEED_PROCESS_ISOLATION=false`. `Isolation.run` re-executes the current command with `CODSPEED_ISOLATED_BENCHMARK` set to one benchmark URI, so each benchmark gets a fresh process. The child only runs that benchmark, which overrides user filters. - `CODSPEED_PROCESS_ISOLATION`: unset, `1` or `true` is on, `0` or `false` is off, anything else is an error. - `CODSPEED_ISOLATED_BENCHMARK`: set by the parent on each child. - `CODSPEED_ISOLATION_COMMAND`: JSON command to re-run, for runners that measure in worker processes whose argv is not the benchmark command. The child is started with `process.execPath` and `process.execArgv`, so the V8 flags set by the runner survive. A failing child ends the parent with the same exit code (128 + N when signal N killed it). Children run sequentially with `spawnSync`, which keeps the output in order. Re-exec is used instead of fork because fork only copies the calling thread.
Route every instrumented benchmark through `Isolation.run`. A child process only runs its own benchmark and skips the banner, the summary line and the completion listeners that the parent already handles. A run call that owns no benchmark of the child does not touch the instruments.
The analysis runner measures every task through `Isolation.run` (async and sync). Walltime runs and runs without active instrument hooks are not isolated. A child only measures its own task and stays quiet about the banner and the summary line.
Both the Vitest 5 provider and the legacy analysis runner measure through `Isolation.run`. An isolated child still reports every benchmark to Vitest, but only measures its own. Vitest measures in forked workers, and the argv of a worker is not the command that starts the benchmarks. The plugin records the command of the main Vitest process in `CODSPEED_ISOLATION_COMMAND` from its `config()` hook, before the workers are forked, and each worker re-runs that command once per benchmark. This means one full Vitest run per benchmark.
Run fixtures in simulation mode with the instrument hooks stubbed (they only exist under the CodSpeed runner) and check, for core, benchmark.js, tinybench and Vitest 5, that: - every benchmark runs in a fresh process by default; - `CODSPEED_PROCESS_ISOLATION=false` runs them all in one process; - a failing benchmark fails the run; - core forwards the child exit code (128 + N for signals) and the execArgv. Tests that drive a plugin in-process against a mocked core now set `CODSPEED_PROCESS_ISOLATION=false`, because isolation would start the test runner again for every benchmark.
Document the environment contract, the parent and child roles, and the Vitest and test caveats.
a33e439 to
4bd0611
Compare
Merging this PR will degrade performance by 77.26%3 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
Footnotes
|
Avoid transforming and importing unrelated benchmark files in each child. Replace discovery globs before Vitest resolves its configuration, preserving fresh processes and profiling flags.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Keep environment alias validation in core instead of repeating it through every plugin. Retain real process, opt-out, failure, signal, and flag propagation checks while dropping getter-only assertions and redundant documentation.
 [Medium impact] The change reaches instrumented benchmark runs across three Node.js integrations. Fix the legacy Vitest resource conflict and missing chained Benchmark.js measurements before merging.
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| await optimizeFunction(async () => { | ||
| if (!isolation.shouldRun(uri)) return; | ||
|
|
||
| await isolation.run(uri, async () => { |
There was a problem hiding this comment.
The legacy runner starts each child while the parent's suite resources are still open. runAnalysisBenchmarkSuite calls beforeAll before this new isolation.run call, and afterAll runs only after the child finishes. If beforeAll starts a server on a fixed port, the child repeats that setup while the parent still owns the port and fails with EADDRINUSE.
Skip suite hooks in the process that only schedules children; run them in the child that measures the benchmark.
Knowledge Base Used: Vitest integration
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/vitest-plugin/src/legacy/analysis.ts
Line: 50
Comment:
**Suite setup conflicts with children**
The legacy runner starts each child while the parent's suite resources are still open. `runAnalysisBenchmarkSuite` calls `beforeAll` before this new `isolation.run` call, and `afterAll` runs only after the child finishes. If `beforeAll` starts a server on a fixed port, the child repeats that setup while the parent still owns the port and fails with `EADDRINUSE`.
Skip suite hooks in the process that only schedules children; run them in the child that measures the benchmark.
**Knowledge Base Used:** [Vitest integration](https://app.greptile.com/codspeed/-/custom-context/knowledge-base/codspeedhq/codspeed-node/-/docs/vitest-integration.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
| // The parent reports for every benchmark, the child ran only one of them. | ||
| if (!isChild) { | ||
| benchmarkCompletedListeners.forEach((listener) => listener()); |
There was a problem hiding this comment.
Skipping complete listeners in children breaks benchmarks started by those listeners. If the first suite's complete listener calls second.run(), the parent reaches the second suite, but its child restarts the script at the first suite. That first suite has no matching URI and returns without calling the listener, so second.run() is never reached. The child exits successfully, and the parent reports completion without measuring the second suite.
Preserve the callbacks needed to reach the selected benchmark, including when an earlier suite has no matching benchmark.
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/benchmark.js-plugin/src/index.ts
Line: 195-197
Comment:
**Chained benchmarks disappear**
Skipping `complete` listeners in children breaks benchmarks started by those listeners. If the first suite's `complete` listener calls `second.run()`, the parent reaches the second suite, but its child restarts the script at the first suite. That first suite has no matching URI and returns without calling the listener, so `second.run()` is never reached. The child exits successfully, and the parent reports completion without measuring the second suite.
Preserve the callbacks needed to reach the selected benchmark, including when an earlier suite has no matching benchmark.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.The legacy analysis runner ran beforeAll/afterAll in the process that only starts children, so a child setting up the same resources (e.g. a fixed port) failed while the parent still held them.
A child skipped the complete listeners of runs that don't own its benchmark, so a benchmark started from such a listener was never measured.
Isolation.run waited for the child with spawnSync. In a Vitest worker that blocked the worker's RPC with its main process for the whole child run, and under the simulation instrument the calls timed out (Timeout calling "onTaskUpdate"), failing the run.
Separators are already normalized to slashes; escaping every glob metacharacter, backslash included, makes the escaping complete.
TLDR: Benchmarks in one process share allocator and JIT state, so one benchmark shifts the next; each instrumented benchmark now runs in a fresh process (one Vitest run restricted to the target file per benchmark for the Vitest plugin).
CODSPEED_PROCESS_ISOLATION(unset/1/trueon by default,0/falseoff, other values error),CODSPEED_ISOLATED_BENCHMARK=<uri>set on each childprocess.execPath+process.execArgv+process.argv.slice(1), so the V8 flags from the runner survive128 + Nfor signalN)Review notes
CODSPEED_ISOLATION_COMMANDfrom the main process; each worker re-runs the Vitest command per benchmark, but isolated children replace benchmark discovery globs with their target file before configuration resolution. Unrelated benchmark files are not imported; configuration and setup files still run. The child's Vitest reporter output is not silencedCODSPEED_PROCESS_ISOLATION=falsesetupCore, so it writes a perf map and environment file for a PID that measures nothingMirrors the codspeed-rust implementation (CodSpeedHQ/codspeed-rust, process isolation)
Closes COD-3960