Skip to content

Read the cached collected data only when the rules on CollectedDataNode run - #6373

Merged
ondrejmirtes merged 1 commit into
phpstan:2.3.xfrom
janedbal:lazy-collected-data
Nov 1, 2026
Merged

ondrejmirtes merged 1 commit into
phpstan:2.3.xfrom
janedbal:lazy-collected-data

Conversation

@janedbal

@janedbal janedbal commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem

restore() decodes the whole collectedData section into the main process before the analysis starts. Only the rules on CollectedDataNode need that data, after the analysis, in the main process. Until then it sits in memory for the whole run, and every forked worker inherits it. On a project with a collector-heavy extension this is the largest section of the cache by far: Dead Code Detector stores every member usage in it, hundreds of megabytes on a large codebase. That is why DCD moved its data into its own file store (shipmonk-rnd/dead-code-detector#319) and later optimised that store (shipmonk-rnd/dead-code-detector#427). With 2.2.13 and this change both can go: DCD emits its usages through the collector API again.

Change

  • An array-frame entry carries its key as a separate payload before the value, so a section can be indexed without decoding any value. Cache version bumped.
  • restore() walks the collectedData section into an index (file to offset and length) and hands it out as LazyCollectedData together with the run's fresh data. That is all the workers inherit.
  • save() copies the entries that stay valid byte for byte from the old file, records their positions in the new one, and serializes only the fresh entries. A warm run writes a byte-identical file.
  • AnalyserResultFinalizer is the only place that decodes the section, and the decoded array lives as long as the CollectedDataNode.
  • The file is opened per read; no handle survives a fork or blocks a rename on Windows. Both the reader and the copy verify the key at the offset, so a cache replaced by another run sharing the tmpDir is never copied misaligned: the save is skipped and the run reports an internal error once.
  • A value that fails to unserialize used to be caught in restore(). It now surfaces in the finalizer, after the bytes were copied into the new file, so the reader discards the file and the finalizer reports an internal error (shouldReportBug: false) instead of failing on every following run.

CollectedDataNode and the Collector API are unchanged. AnalysisResult (@api) accepts the collected data as a closure so a formatter that never asks for it never decodes the section.

Numbers

phpstan-src self-analysis with DCD data in the result cache (2467 files, 88 MB cache), main process, Linux:

before after
restore(), warm run 200 ms, +168 MB 175 ms, +74 MB
partial run of 486 files, main process peak 486 MB 440 MB
partial run of 486 files, largest forked worker 368 MB 276 MB

The peak during the CollectedDataNode rules is unchanged; they still get the whole array. A streaming API for rules would be a separate, additive change.

Notes

@janedbal
janedbal marked this pull request as ready for review September 4, 2026 12:31
@phpstan-bot

Copy link
Copy Markdown
Collaborator

This pull request has been marked as ready for review.

…de run

restore() decoded the whole collectedData section into the main process
before the analysis started. The section is the largest part of the cache on
a project using collectors - a dead code detector stores every member usage
in it, hundreds of megabytes on a large codebase - and only the rules that
run on CollectedDataNode need it, after the analysis, in the main process.
Decoding it up front kept it in memory through the whole run, and the forked
workers inherited it.

The array frames now carry the key in its own payload, so a reader can index
the entries without decoding the values. readCacheFile() walks the
collectedData section recording where each file's entry is, and restore()
hands that index out as LazyCollectedData together with the fresh data of the
run. save() copies the entries that stay valid byte for byte from the old
file into the new one and records their new positions, so the finalizer reads
them back from the file that exists after the rename. The section is decoded
once, in AnalyserResultFinalizer, and lives as long as the CollectedDataNode.

The file is opened for each read and closed again: an open handle would be
inherited by the forked workers, and a rename over the file must not find it
open on Windows. A read checks that the entry it finds belongs to the file it
was indexed for, so a cache file replaced by another run fails loudly instead
of handing out someone else's data.

AnalysisResult takes the collected data as a closure too, so a formatter that
never asks for it does not decode the section a second time.

Measured on phpstan-src self-analysis with a dead code detector that keeps
its usages in collected data (2467 files, 88 MB cache), main process:

                            before              after
  restore(), warm run       200 ms, +168 MB     175 ms, +74 MB
  partial run, 486 files    486 MB main         440 MB main
                            368 MB largest worker  276 MB largest worker

A warm run that copies every entry writes a byte-identical cache file.
The cache version is bumped because the entry framing changed.

Amp-Thread-ID: https://ampcode.com/threads/T-01a19219-6260-7460-a52b-a418f709e4bf
@ondrejmirtes
ondrejmirtes merged commit 9c8ed05 into phpstan:2.3.x Nov 1, 2026
271 of 273 checks passed
@ondrejmirtes

Copy link
Copy Markdown
Member

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants