feat(vitest-plugin): add walltime forks and GC between rounds by not-matthias · Pull Request #111 · CodSpeedHQ/codspeed-node

not-matthias · 2026-10-28T19:42:44Z

TLDR: Two opt-in walltime knobs for the Vitest 3/4 plugin: run each benchmark in several fresh processes and merge their rounds, and run a full GC before every round. With several forks, each fork runs half the warmup and measurement budget.

CODSPEED_WALLTIME_FORKS=<N>: each benchmark runs in N fresh processes, one after the other. Each child re-runs the Vitest command restricted to its benchmark (CODSPEED_ISOLATED_BENCHMARK), writes its round samples to CODSPEED_FORK_SAMPLES_FILE, and the parent reports all rounds as one result. Unset: unchanged behavior.
With N > 1, each fork runs half the time and ceil(half) the rounds of both warmup and measurement (FORK_BUDGET_FRACTION). Every fork pays the process startup anyway; without this, N forks cost N full runs. A single fork keeps the full budget.
CODSPEED_WALLTIME_GC_BETWEEN_ROUNDS=true: adds --expose-gc and runs a full GC in tinybench's beforeEach, outside the timed call (Vitest 3/4/5). It re-baselines a benchmark's results.
Also fixes walltime URIs of top-level benches, which lacked the file path.

Review notes

Stacked on feat: run walltime benchmarks in forks #101 (process isolation); the fork children reuse its re-exec command.
Forks are not supported on Vitest 5 yet: the provider throws when CODSPEED_WALLTIME_FORKS is set.
The half budget comes from walltime experiments on dedicated machines: on small synthetic benchmarks, 3 forks at half the budget were as stable as 1 fork at the full budget without moving the measured min. On benchmarks with heavy warmup or state that builds up over rounds, smaller per-fork budgets did move the min (by a few percent at half the budget), so forks change what those benchmarks measure. That is why forks stay opt-in.

Refs COD-3984

Benchmarks of one process share allocator, JIT and thread state, so what one benchmark leaves behind (raised mmap threshold, fragmented heap, optimized code, retained caches) changes the measurement of every benchmark after it. Adding or changing one benchmark can then shift unrelated results. Add `Isolation` for the integrations. It is on by default and turned off with `CODSPEED_PROCESS_ISOLATION=false`. `Isolation.run` re-executes the current command with `CODSPEED_ISOLATED_BENCHMARK` set to one benchmark URI, so each benchmark gets a fresh process. The child only runs that benchmark, which overrides user filters. - `CODSPEED_PROCESS_ISOLATION`: unset, `1` or `true` is on, `0` or `false` is off, anything else is an error. - `CODSPEED_ISOLATED_BENCHMARK`: set by the parent on each child. - `CODSPEED_ISOLATION_COMMAND`: JSON command to re-run, for runners that measure in worker processes whose argv is not the benchmark command. The child is started with `process.execPath` and `process.execArgv`, so the V8 flags set by the runner survive. A failing child ends the parent with the same exit code (128 + N when signal N killed it). Children run sequentially with `spawnSync`, which keeps the output in order. Re-exec is used instead of fork because fork only copies the calling thread.

Route every instrumented benchmark through `Isolation.run`. A child process only runs its own benchmark and skips the banner, the summary line and the completion listeners that the parent already handles. A run call that owns no benchmark of the child does not touch the instruments.

The analysis runner measures every task through `Isolation.run` (async and sync). Walltime runs and runs without active instrument hooks are not isolated. A child only measures its own task and stays quiet about the banner and the summary line.

Both the Vitest 5 provider and the legacy analysis runner measure through `Isolation.run`. An isolated child still reports every benchmark to Vitest, but only measures its own. Vitest measures in forked workers, and the argv of a worker is not the command that starts the benchmarks. The plugin records the command of the main Vitest process in `CODSPEED_ISOLATION_COMMAND` from its `config()` hook, before the workers are forked, and each worker re-runs that command once per benchmark. This means one full Vitest run per benchmark.

Run fixtures in simulation mode with the instrument hooks stubbed (they only exist under the CodSpeed runner) and check, for core, benchmark.js, tinybench and Vitest 5, that: - every benchmark runs in a fresh process by default; - `CODSPEED_PROCESS_ISOLATION=false` runs them all in one process; - a failing benchmark fails the run; - core forwards the child exit code (128 + N for signals) and the execArgv. Tests that drive a plugin in-process against a mocked core now set `CODSPEED_PROCESS_ISOLATION=false`, because isolation would start the test runner again for every benchmark.

Document the environment contract, the parent and child roles, and the Vitest and test caveats.

Avoid transforming and importing unrelated benchmark files in each child. Replace discovery globs before Vitest resolves its configuration, preserving fresh processes and profiling flags.

Keep environment alias validation in core instead of repeating it through every plugin. Retain real process, opt-out, failure, signal, and flag propagation checks while dropping getter-only assertions and redundant documentation.

greptile-apps · 2026-10-28T19:46:36Z

Confidence Score: 4/5

[Medium impact] The changes affect Vitest walltime benchmark runs and their reported samples.

The PR appears safe to merge, with non-blocking fixes recommended for failed-fork cleanup and promise-like hooks.

Findings

Failed forks leave files behind ▶
Async setup can finish late ▶

Fix with agent prompt

### Issue 1
packages/core/src/isolation.ts:170
`runForks` relies on `finally` to remove its temporary directory, but `runInChildProcess` calls `process.exit()` when a child fails. That skips cleanup and leaves any earlier fork's sample files behind. Repeated failures can build up files on long-lived workers. Return or throw the failure so cleanup finishes before exiting.

### Issue 2
packages/vitest-plugin/src/instrument.ts:48-51
If a hook returns a promise from another JavaScript context or a promise-like object, `withGcAfter` fails the `instanceof Promise` check. It runs GC immediately and discards the returned value, so the benchmark can start before setup finishes. Recognize promise-like results and wait for them before collecting garbage, while keeping synchronous hooks synchronous.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds opt-in walltime forks for Vitest 3/4 and GC before rounds for Vitest 3/4/5.

Walltime benchmarks can combine rounds from several fresh processes.
Walltime rounds can start after a full garbage collection.
Top-level Vitest benchmarks include their file in the result URI.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Walltime benchmark] --> B{Forks configured?}
  B -->|No| C[Run benchmark locally]
  B -->|Vitest 3/4| D[Start fresh child for each fork]
  B -->|Vitest 5| E[Reject unsupported option]
  D --> F[Select one benchmark]
  F --> G[Warm up and measure]
  G --> H[Write round samples]
  H --> I[Parent combines samples]
  C --> J[Write walltime result]
  I --> J

_{Reviews (1) · Last reviewed commit: "feat(vitest-plugin): halve the walltime ..." · Reviewed by Greptile}

greptile-apps · 2026-10-28T19:46:39Z

+    let samples: number[] = [];
+    for (let fork = 0; fork < forks; fork++) {
+      const file = join(dir, `${fork}.json`);
+      runInChildProcess(uri, { [FORK_SAMPLES_FILE_ENV]: file });


Failed forks leave files behind

runForks relies on finally to remove its temporary directory, but runInChildProcess calls process.exit() when a child fails. That skips cleanup and leaves any earlier fork's sample files behind. Repeated failures can build up files on long-lived workers. Return or throw the failure so cleanup finishes before exiting.

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/core/src/isolation.ts Line: 170 Comment: **Failed forks leave files behind** `runForks` relies on `finally` to remove its temporary directory, but `runInChildProcess` calls `process.exit()` when a child fails. That skips cleanup and leaves any earlier fork's sample files behind. Repeated failures can build up files on long-lived workers. Return or throw the failure so cleanup finishes before exiting. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

greptile-apps · 2026-10-28T19:46:39Z

+    if (result instanceof Promise) {
+      return result.then(collectGarbage);
+    }
+    collectGarbage();


Async setup can finish late

If a hook returns a promise from another JavaScript context or a promise-like object, withGcAfter fails the instanceof Promise check. It runs GC immediately and discards the returned value, so the benchmark can start before setup finishes. Recognize promise-like results and wait for them before collecting garbage, while keeping synchronous hooks synchronous.

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/vitest-plugin/src/instrument.ts Line: 48-51 Comment: **Async setup can finish late** If a hook returns a promise from another JavaScript context or a promise-like object, `withGcAfter` fails the `instanceof Promise` check. It runs GC immediately and discards the returned value, so the benchmark can start before setup finishes. Recognize promise-like results and wait for them before collecting garbage, while keeping synchronous hooks synchronous. --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

The legacy analysis runner ran beforeAll/afterAll in the process that only starts children, so a child setting up the same resources (e.g. a fixed port) failed while the parent still held them.

A child skipped the complete listeners of runs that don't own its benchmark, so a benchmark started from such a listener was never measured.

codspeed · 2026-10-28T19:55:26Z

Merging this PR will degrade performance by 0.53%

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 34 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 28 benchmarks have no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 81 improved benchmarks
❌ 41 regressed benchmarks
✅ 151 untouched benchmarks
🆕 6 new benchmarks
⏩ 56 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	CPU Simulation	`test sync baz 100`	317.1 µs	1,064.9 µs	-70.22%
❌	Memory	`count is back to zero`	1.3 KB	3.1 KB	-57.5%
❌	Memory	`one`	1.3 KB	3.1 KB	-57.5%
❌	Memory	`two`	1.3 KB	3.1 KB	-57.5%
❌	CPU Simulation	`switch 2`	306.1 µs	682.6 µs	-55.16%
❌	CPU Simulation	`test_recursive_fibo_10`	340.1 µs	719.7 µs	-52.74%
❌	CPU Simulation	`test_recursive_cached_fibo_30`	345.3 µs	728.2 µs	-52.58%
❌	CPU Simulation	`short body`	121.6 µs	243.9 µs	-50.15%
❌	CPU Simulation	`short body`	121.5 µs	243.6 µs	-50.09%
❌	CPU Simulation	`recursive`	350.2 µs	626.1 µs	-44.07%
❌	CPU Simulation	`fibo 15`	350.1 µs	624.2 µs	-43.91%
❌	CPU Simulation	`recursive fibo 15`	357.7 µs	635.2 µs	-43.69%
❌	CPU Simulation	`test_recursive_cached_fibo_10`	70.9 µs	96.8 µs	-26.82%
❌	CPU Simulation	`recursive fibo 15 with hooks`	615.3 µs	834.2 µs	-26.25%
❌	CPU Simulation	`recursive fibo 15 with hooks`	615.7 µs	834.7 µs	-26.24%
❌	CPU Simulation	`short body`	121.6 µs	164.1 µs	-25.9%
❌	CPU Simulation	`short body`	121.9 µs	164 µs	-25.7%
❌	CPU Simulation	`test_recursive_cached_fibo_30`	93.1 µs	125 µs	-25.48%
❌	CPU Simulation	`test_recursive_fibo_10`	66.3 µs	87.6 µs	-24.24%
❌	CPU Simulation	`test_recursive_cached_fibo_20`	85.3 µs	112 µs	-23.84%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing cod-3984-walltime-gc-and-forks (4353e5d) with cod-3960-implement-process-isolation-in-codspeed-node (860d65c)}

56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Isolation.run waited for the child with spawnSync. In a Vitest worker that blocked the worker's RPC with its main process for the whole child run, and under the simulation instrument the calls timed out (Timeout calling "onTaskUpdate"), failing the run.

Separators are already normalized to slashes; escaping every glob metacharacter, backslash included, makes the escaping complete.

…e file path The walltime runner only registered URI prefixes for nested suites, so a bench declared outside a `describe` block reported `::<name>` to the runner while its results used `<file>::<name>`. The profile and results URIs then no longer matched and the benchmark had no flamegraph. Register the root file suite too, and add a top-level bench to the Vitest 3 and 4 examples. Refs COD-3984

Walltime rounds vary when a major GC lands in some of them, and a single process can carry JIT and allocator state that a fresh one would not. Two opt-in settings, modeled on JMH, target this: - CODSPEED_WALLTIME_GC_BETWEEN_ROUNDS=true adds --expose-gc and runs a full GC in tinybench's beforeEach hook, outside the timed call (Vitest 3, 4 and 5). - CODSPEED_WALLTIME_FORKS=<N> runs each benchmark in N fresh processes and reports all their rounds as one result (Vitest 3 and 4). Forks re-run the Vitest command restricted to one URI, like process isolation, and hand their samples to the parent through a file. The legacy runner now patches tinybench's Task prototype once, so new runner instances no longer stack root-frame wrappers.

With several forks, every fork paid the full warmup and measurement budget on top of its process startup, so F forks cost F full runs. Each fork now runs half the time and ceil(half) the rounds of both phases. A single fork keeps the full budget.

A failing fork exited the parent through process.exit(), which skipped the cleanup of the samples directory.

A beforeEach hook returning a promise from another realm or a thenable failed the instanceof check, so the GC ran before the hook finished.

runForks waited for each fork with spawnSync, blocking a Vitest worker's RPC with its main process for the whole run of all forks.

not-matthias added 9 commits October 21, 2026 11:55

docs: describe process isolation

4bd0611

Document the environment contract, the parent and child roles, and the Vitest and test caveats.

perf(vitest): restrict isolated children to their benchmark file

3105892

Avoid transforming and importing unrelated benchmark files in each child. Replace discovery globs before Vitest resolves its configuration, preserving fresh processes and profiling flags.

docs: describe file filtering for instrumented isolation

8f1c9a2

test: trim redundant process isolation coverage

8816d3e

Keep environment alias validation in core instead of repeating it through every plugin. Retain real process, opt-out, failure, signal, and flag propagation checks while dropping getter-only assertions and redundant documentation.

greptile-apps Bot reviewed Oct 28, 2026

View reviewed changes

not-matthias added 2 commits October 28, 2026 20:51

fix(vitest-plugin): leave suite hooks to isolated children

35e1d71

The legacy analysis runner ran beforeAll/afterAll in the process that only starts children, so a child setting up the same resources (e.g. a fixed port) failed while the parent still held them.

fix(benchmark.js-plugin): call complete listeners in isolated children

1d6e63f

A child skipped the complete listeners of runs that don't own its benchmark, so a benchmark started from such a listener was never measured.

not-matthias force-pushed the cod-3984-walltime-gc-and-forks branch from 4bde326 to a415741 Compare October 28, 2026 19:52

not-matthias force-pushed the cod-3984-walltime-gc-and-forks branch from a415741 to aec7410 Compare October 28, 2026 20:27

not-matthias added 9 commits October 28, 2026 21:28

fix(vitest-plugin): escape backslashes in the isolated benchmark glob

860d65c

Separators are already normalized to slashes; escaping every glob metacharacter, backslash included, makes the escaping complete.

docs: note that GC between rounds re-baselines walltime results

8d1f518

refactor(vitest-plugin): drop unused walltime task field

ebd690a

fix(core): remove fork samples when a fork fails

ef6aac6

A failing fork exited the parent through process.exit(), which skipped the cleanup of the samples directory.

fix(vitest-plugin): wait for any thenable before GC between rounds

8de5086

A beforeEach hook returning a promise from another realm or a thenable failed the instanceof check, so the GC ran before the hook finished.

fix(core): keep the event loop free while forks run

4353e5d

runForks waited for each fork with spawnSync, blocking a Vitest worker's RPC with its main process for the whole run of all forks.

not-matthias force-pushed the cod-3984-walltime-gc-and-forks branch from aec7410 to 4353e5d Compare October 28, 2026 20:28

not-matthias force-pushed the cod-3960-implement-process-isolation-in-codspeed-node branch from 860d65c to 8038822 Compare October 29, 2026 11:10

greptile-apps Bot mentioned this pull request Oct 29, 2026

feat: run walltime benchmarks in forks #101

Open

not-matthias force-pushed the cod-3960-implement-process-isolation-in-codspeed-node branch from 8038822 to 86bdbbb Compare October 29, 2026 11:39

not-matthias force-pushed the cod-3960-implement-process-isolation-in-codspeed-node branch from 86bdbbb to e9af7a1 Compare October 29, 2026 11:54

Navigation Menu

Repository navigation

feat(vitest-plugin): add walltime forks and GC between rounds - #111

feat(vitest-plugin): add walltime forks and GC between rounds#111
not-matthias wants to merge 21 commits into
cod-3960-implement-process-isolation-in-codspeed-nodefrom
cod-3984-walltime-gc-and-forks

not-matthias commented Oct 28, 2026

Uh oh!

greptile-apps Bot commented Oct 28, 2026 •

edited

Loading

Diagram

Uh oh!

greptile-apps Bot Oct 28, 2026

Uh oh!

greptile-apps Bot Oct 28, 2026

Uh oh!

codspeed Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

not-matthias commented Oct 28, 2026

Uh oh!

greptile-apps Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 28, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 0.53%

Performance Changes

Footnotes

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

greptile-apps Bot commented Oct 28, 2026 •

edited

Loading

codspeed Bot commented Oct 28, 2026 •

edited

Loading