Repository navigation
Read the cached collected data only when the rules on CollectedDataNode run - #6373
Merged
Merged
Conversation
janedbal
marked this pull request as ready for review
September 4, 2026 12:31
Collaborator
|
This pull request has been marked as ready for review. |
staabm
force-pushed
the
lazy-collected-data
branch
from
September 11, 2026 07:56
769b3d6 to
b3e5982
Compare
ondrejmirtes
force-pushed
the
lazy-collected-data
branch
from
October 31, 2026 14:23
b3e5982 to
67d6615
Compare
…de run
restore() decoded the whole collectedData section into the main process
before the analysis started. The section is the largest part of the cache on
a project using collectors - a dead code detector stores every member usage
in it, hundreds of megabytes on a large codebase - and only the rules that
run on CollectedDataNode need it, after the analysis, in the main process.
Decoding it up front kept it in memory through the whole run, and the forked
workers inherited it.
The array frames now carry the key in its own payload, so a reader can index
the entries without decoding the values. readCacheFile() walks the
collectedData section recording where each file's entry is, and restore()
hands that index out as LazyCollectedData together with the fresh data of the
run. save() copies the entries that stay valid byte for byte from the old
file into the new one and records their new positions, so the finalizer reads
them back from the file that exists after the rename. The section is decoded
once, in AnalyserResultFinalizer, and lives as long as the CollectedDataNode.
The file is opened for each read and closed again: an open handle would be
inherited by the forked workers, and a rename over the file must not find it
open on Windows. A read checks that the entry it finds belongs to the file it
was indexed for, so a cache file replaced by another run fails loudly instead
of handing out someone else's data.
AnalysisResult takes the collected data as a closure too, so a formatter that
never asks for it does not decode the section a second time.
Measured on phpstan-src self-analysis with a dead code detector that keeps
its usages in collected data (2467 files, 88 MB cache), main process:
before after
restore(), warm run 200 ms, +168 MB 175 ms, +74 MB
partial run, 486 files 486 MB main 440 MB main
368 MB largest worker 276 MB largest worker
A warm run that copies every entry writes a byte-identical cache file.
The cache version is bumped because the entry framing changed.
Amp-Thread-ID: https://ampcode.com/threads/T-01a19219-6260-7460-a52b-a418f709e4bf
ondrejmirtes
force-pushed
the
lazy-collected-data
branch
from
November 1, 2026 11:07
67d6615 to
78b51d2
Compare
Member
|
Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
restore()decodes the wholecollectedDatasection into the main process before the analysis starts. Only the rules onCollectedDataNodeneed that data, after the analysis, in the main process. Until then it sits in memory for the whole run, and every forked worker inherits it. On a project with a collector-heavy extension this is the largest section of the cache by far: Dead Code Detector stores every member usage in it, hundreds of megabytes on a large codebase. That is why DCD moved its data into its own file store (shipmonk-rnd/dead-code-detector#319) and later optimised that store (shipmonk-rnd/dead-code-detector#427). With 2.2.13 and this change both can go: DCD emits its usages through the collector API again.Change
restore()walks thecollectedDatasection into an index (file to offset and length) and hands it out asLazyCollectedDatatogether with the run's fresh data. That is all the workers inherit.save()copies the entries that stay valid byte for byte from the old file, records their positions in the new one, and serializes only the fresh entries. A warm run writes a byte-identical file.AnalyserResultFinalizeris the only place that decodes the section, and the decoded array lives as long as theCollectedDataNode.restore(). It now surfaces in the finalizer, after the bytes were copied into the new file, so the reader discards the file and the finalizer reports an internal error (shouldReportBug: false) instead of failing on every following run.CollectedDataNodeand theCollectorAPI are unchanged.AnalysisResult(@api) accepts the collected data as a closure so a formatter that never asks for it never decodes the section.Numbers
phpstan-src self-analysis with DCD data in the result cache (2467 files, 88 MB cache), main process, Linux:
restore(), warm runThe peak during the
CollectedDataNoderules is unchanged; they still get the whole array. A streaming API for rules would be a separate, additive change.Notes
CollectedDataNodelists the cached files first and the re-analysed ones after them. A full run already yields worker-completion order, so nothing can depend on it.ResultCacheManager::process()dropsworkerCounton the merged result, so cached runs never print the per-worker peak. Not included here; the worker numbers above come from a one-line local fix.