You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[PERF] Reduce cold graph extraction work on large Python projects #59
[PERF] Reduce cold graph extraction work on large Python projects
Problem
The current 5,000-leaf-module comparison corpus contains 5,005 Python files
and 13,738 import edges. The published end-to-end fresh-process median for
one ArchUnitPython boundary rule was 3.59 s (v1.5.0, commit 09c9d199dec095aaa60be227bfd7b5c2b7ffb107, Python 3.13.4, Windows 11).
Exploratory timing of one direct rule.check() found graph extraction taking
roughly 1.8–2.6 s, depending on machine load. Within that phase, _extract_located_imports() accounted for about 1.2–1.7 s over 5,005 files,
and import resolution about 0.37–0.51 s over 13,738 imports. These components
are cumulative instrumentation measurements, not independent benchmark bars.
The source suggests two specific sources of avoidable work:
Each parsed file is walked separately to find TYPE_CHECKING blocks,
conditional import ranges, and actual imports/calls.
Absolute imports are resolved through repeated filesystem existence checks,
even though the scan already has the set of project Python files.
Try a single AST visitor that records contextual import kinds and imports
in one traversal. Compare it with current behavior on nested conditions.
Build a module/file lookup from the discovered project files and use it
for internal absolute-import resolution; retain current semantics for
package __init__.py, relative imports, parent roots, and external modules.
Profile again before considering parallel parsing or a persistent disk
cache. A disk cache needs explicit invalidation semantics and a separate
correctness test matrix.
Acceptance criteria
Existing tests and seeded direct-boundary/cycle correctness gates pass.
Tests cover relative and absolute imports, TYPE_CHECKING, optional imports,
literal dynamic imports, ignore directives, excluded paths, and malformed
Python files.
Report before/after fresh-process p50/p95 at 100/1,000/5,000 modules,
extraction phase time, and peak memory. Preserve zero runtime dependencies.
Implementation progress, now in PR #60 (not yet merged or released):
Tried a single AST walk and a per-graph absolute-import lookup; neither produced a repeatable gain on the generated corpus, so those experiments were reverted.
A larger win came from direct file-dependency rules: discover all project files for correct internal-target classification, but parse imports only from files selected by the rule's source filters. Parsed per-file edges are shared within the process, so later direct rules or a full-graph rule can reuse them. Clearing the graph cache also clears this session.
Four paired in-process checks on the 5,005-file corpus parsed 1,251 files with selection versus 5,005 with the full graph. Full/selective times were 6.37/3.13, 6.73/3.14, 6.26/3.51, and 7.20/3.18 seconds. Both paths returned the same result. A separate tracemalloc diagnostic measured 12.98 MiB versus 5.25 MiB peak Python-tracked allocations; this is not total process RSS, and its instrumented timings are not suitable as speed results.
The fresh-process ArchUnitPython harness (one warm-up, five samples per size) measured p50/p95 of 0.830/0.868 s at 100 modules, 1.008/1.194 s at 1,000, and 1.743/1.892 s at 5,000. The previous matcher-cache-only 5,000-module median was 2.826 s, so the separate-run median is 38% lower. The clean, seeded forbidden-edge, and cycle correctness gates passed.
In the 10-additional-rule diagnostic, elapsed time fell from 7.62 to 5.58 s, with the same 7,494 violations. Full suite: 299 passed; Ruff and Mypy pass.
These were preliminary local trials on the older 1.5.0 checkout, not a claim that ArchUnitPython is fastest. PR #60 is based on current 1.7.0, includes an eight-pair alternating comparison against clean 1.7.0, and passes CI on Python 3.10–3.13. No persistent disk cache or new runtime dependency was added.
Activity
TristanKruse commentedon Sep 24, 2026
Implementation progress, now in PR #60 (not yet merged or released):
tracemallocdiagnostic measured 12.98 MiB versus 5.25 MiB peak Python-tracked allocations; this is not total process RSS, and its instrumented timings are not suitable as speed results.These were preliminary local trials on the older 1.5.0 checkout, not a claim that ArchUnitPython is fastest. PR #60 is based on current 1.7.0, includes an eight-pair alternating comparison against clean 1.7.0, and passes CI on Python 3.10–3.13. No persistent disk cache or new runtime dependency was added.