Skip to content

fix(pypi): fetch retained wheel archives lazily - #4316

Open
charleysaintNV wants to merge 6 commits into
bazel-contrib:mainfrom
charleysaintNV:fix/lazy-wheel-archives
Open

charleysaintNV wants to merge 6 commits into
bazel-contrib:mainfrom
charleysaintNV:fix/lazy-wheel-archives

Conversation

@charleysaintNV

@charleysaintNV charleysaintNV commented Oct 22, 2026 •

Copy link
Copy Markdown

URL-resolved wheels currently retain both the downloaded archive and extracted files even when a build only consumes the Python library. Download the archive temporarily for extraction and remove it afterward, while preserving the existing whl and whl_file targets through a lazy sibling <repo>_whl repository. Ordinary Python builds do not fetch that sibling repository.

Apply matching patches in the separate wheel-file repository and extract from its patched archive. This keeps both original and patched archives out of the extracted repository. Preserve caller-owned wheel inputs, using a distinct _patched_whl repository for their patched replacements to avoid collisions with existing source repositories. Source-built wheels and direct whl_archive URL calls retain their current archive behavior. Both download paths share checksum validation, authentication, and URL/environment substitution.

Extend the existing wheel-library integration test with a generated wheel served over HTTP. Separate Bazel invocations check archive cleanup, lazy repository creation, both wheel targets, matching and nonmatching patches, preservation of caller-owned inputs, and the full pip.parse(uv_lock = ...) path. Retain the existing pip-backend, source-distribution, direct whl_archive, and dependency-wiring assertions in the same integration target. Add analysis coverage for archive ownership and exclusion from runtime files.

Gazelle's module mapping still consumes wheel archives and therefore materializes their archive repositories. Changing that consumer to use extracted libraries is separate follow-up work.

Validation:

  • All 31 bzlmod example tests pass, including an assertion that the public wheel archive contains the same patched metadata as the extracted library. Both bazel vendor //... and building all targets with the resulting vendor directory pass locally.
  • All 24 affected Bazel test targets pass with --config=fast-tests, including the consolidated //tests/integration:whl_library_test_bazel_self target and its five test methods. The nested integration target is run with tag filters overridden and local test concurrency limited to one.
  • The storage analysis suite passes under bzlmod. Under Bazel 7.4.1 WORKSPACE mode, the bzlmod-specific ownership assertion is skipped and runtime archive exclusion passes.
  • All applicable pre-commit hooks pass, including buildifier, Ruff, deleted-package generation, and Gazelle.
  • The NumPy 2.3.3 PoC imports successfully without materializing an archive repository; explicitly building its whl target materializes the archive. Its extracted repository uses 57.74 MiB instead of 73.61 MiB allocated on the test filesystem. Shared download caches are outside that measurement.

Implements the delete-by-default / optional-archive-access suggestion in #3091 (comment). Preserves the wheel-consumer functionality introduced in #364 and fixed in #458. Related to #3968 and complements the extraction-reuse work in #3856.

@aignas aignas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR.

So all things added up, this way is nice, because:

  • it retains the whl downloaded via bazel_downloader in the repo cache.
  • it extracts and removes the whl and if the user is not using whl targets then the user may be paying less cost for storing the extracted wheels in the reproducible repo cache.
  • If the user asks for the whl target, we should take it from the bazel_downloader cache without a problem.
  • This may facilitate migrating to whl extraction in the build phase.

Things that are not so good or should be discussed:

  • Should we do patching in the extra _whl repository? Then we extract the source from the extra _whl repository and do not do any patching in the main repo that has pkg?
  • Naming: should we name the repo whl_file_repo instead of whl_download?
  • Gazelle is pulling in wheels to get top level imports. I think we should instead change it to use py_library. That would work better with the promises of this PR and users would not need to pay the cost of extracting the wheels there. The extraction is cached in the build cache, but I wonder how this should be done.

Comment thread news/lazy-wheel-archives.changed.md Outdated
Comment thread .bazelrc.deleted_packages Outdated
@charleysaintNV

Copy link
Copy Markdown
Author

Thanks for the review, I’ve renamed the rule to whl_file_repo, moved matching patches into the separate wheel repository, consolidated the integration tests, and added the (pypi) news prefix.

Unpatched archive repositories remain lazy. For patched wheels, extraction and the public wheel targets now consume the same patched archive.

The CI failures exposed an example that globbed for a local wheel. That now uses the public target, with coverage checking that its patched metadata matches the extracted library.

I'm not too familiar with the gazelle workflow but it should work the same after this gets merged so we can probably update in a follow up.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants