Repository navigation
fix(pypi): fetch retained wheel archives lazily - #4316
charleysaintNV wants to merge 6 commits into
Conversation
aignas
left a comment
There was a problem hiding this comment.
Thank you for the PR.
So all things added up, this way is nice, because:
- it retains the whl downloaded via bazel_downloader in the repo cache.
- it extracts and removes the whl and if the user is not using
whltargets then the user may be paying less cost for storing the extracted wheels in the reproducible repo cache. - If the user asks for the whl target, we should take it from the bazel_downloader cache without a problem.
- This may facilitate migrating to whl extraction in the build phase.
Things that are not so good or should be discussed:
- Should we do patching in the extra
_whlrepository? Then we extract the source from the extra_whlrepository and do not do any patching in the main repo that haspkg? - Naming: should we name the repo
whl_file_repoinstead ofwhl_download? - Gazelle is pulling in wheels to get top level imports. I think we should instead change it to use
py_library. That would work better with the promises of this PR and users would not need to pay the cost of extracting the wheels there. The extraction is cached in the build cache, but I wonder how this should be done.
|
Thanks for the review, I’ve renamed the rule to whl_file_repo, moved matching patches into the separate wheel repository, consolidated the integration tests, and added the (pypi) news prefix. Unpatched archive repositories remain lazy. For patched wheels, extraction and the public wheel targets now consume the same patched archive. The CI failures exposed an example that globbed for a local wheel. That now uses the public target, with coverage checking that its patched metadata matches the extracted library. I'm not too familiar with the gazelle workflow but it should work the same after this gets merged so we can probably update in a follow up. |
URL-resolved wheels currently retain both the downloaded archive and extracted files even when a build only consumes the Python library. Download the archive temporarily for extraction and remove it afterward, while preserving the existing
whlandwhl_filetargets through a lazy sibling<repo>_whlrepository. Ordinary Python builds do not fetch that sibling repository.Apply matching patches in the separate wheel-file repository and extract from its patched archive. This keeps both original and patched archives out of the extracted repository. Preserve caller-owned wheel inputs, using a distinct
_patched_whlrepository for their patched replacements to avoid collisions with existing source repositories. Source-built wheels and directwhl_archiveURL calls retain their current archive behavior. Both download paths share checksum validation, authentication, and URL/environment substitution.Extend the existing wheel-library integration test with a generated wheel served over HTTP. Separate Bazel invocations check archive cleanup, lazy repository creation, both wheel targets, matching and nonmatching patches, preservation of caller-owned inputs, and the full
pip.parse(uv_lock = ...)path. Retain the existing pip-backend, source-distribution, directwhl_archive, and dependency-wiring assertions in the same integration target. Add analysis coverage for archive ownership and exclusion from runtime files.Gazelle's module mapping still consumes wheel archives and therefore materializes their archive repositories. Changing that consumer to use extracted libraries is separate follow-up work.
Validation:
bazel vendor //...and building all targets with the resulting vendor directory pass locally.--config=fast-tests, including the consolidated//tests/integration:whl_library_test_bazel_selftarget and its five test methods. The nested integration target is run with tag filters overridden and local test concurrency limited to one.whltarget materializes the archive. Its extracted repository uses 57.74 MiB instead of 73.61 MiB allocated on the test filesystem. Shared download caches are outside that measurement.Implements the delete-by-default / optional-archive-access suggestion in #3091 (comment). Preserves the wheel-consumer functionality introduced in #364 and fixed in #458. Related to #3968 and complements the extraction-reuse work in #3856.