Skip to content

Compress bundle-oci and retire outdated module versions - #489

Merged
rahultee merged 3 commits into
mainfrom
bundle-oci-hybrid-compression
Oct 14, 2026
Merged

rahultee merged 3 commits into
mainfrom
bundle-oci-hybrid-compression

Conversation

@rahultee

@rahultee rahultee commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Why

The full bundle is 64.4 GiB. In goval#24955 it could not download within replvisor's 300 s startup limit, so infra-staging pools restarted in a loop.

This PR makes the bundle smaller in two ways: compress all files, and retire old runtime versions.

Bundle (revision 159a0c2) Uncompressed Compressed (zstd)
Today 64.4 GiB ~15.3 GiB
This PR ~31.7 GiB ~8.2 GiB

The new sizes are estimates, not a full build. Method: hash every file in the published image, find module dependencies from store-path references, and use a per-file zstd model checked against a real mkfs.erofs build of a 2.7 GB sample (corrected for a 13% overestimate).

What changed

  • Compression: mkfs.erofs -zzstd,level=15 -C65536 -Eztailpacking for all files. Timestamps, UUID, ownership, media type and provenance stay deterministic. Compressed files cannot use DAX, so each VM decompresses into its own memory.
  • Retired modules: old IDs stay as upgrade-map aliases to a newer module. Projects keep their config but get the newer runtime, so some code or dependencies may need updates. Projects pinned to an older nixmodules revision do not change.
Old IDs Now resolves to
haskell-ghc9.0, 9.2, 9.4 haskell-ghc9.10
python-3.8, python-3.9, python-with-prybar-3.10 python-3.10
nodejs-14, 16, 18, 19, nodejs-with-prybar-18 nodejs-20
nodejs-23 nodejs-24
vue-node-18 vue-node-20
dotnet-7.0 dotnet-8.0
go, go-1.19, 1.20, 1.21, 1.23 go-1.25
dart-2.18, 3.0 to 3.5 dart-3.10
deno-1 deno-2
php-8.1 php-8.2
r-4.2, 4.3 r-4.5
elixir-1_15, 1_16 elixir-1_18
clojure-1.11 clojure-1.12
swift, swift-5.6 swift-5.8

Most retired versions are end of life upstream (for example Python 3.8/3.9, .NET 7, Node 14 to 23, PHP 8.1, GHC 9.0 to 9.4); the rest have newer replacements. The prybar aliases now use the standard Python and Node runners. Java (java-graalvm22.3, the only Java module), Rust, Ruby, Bun and documentation outputs are unchanged. Development bundles use Go 1.25.

Test plan

  • Passed locally: Nix formatting, deprecated shell-helper check, dev-shell smoke test, full modules and bundle evaluation, and bundle-oci-check (compressed layout, full content check, ownership, hardlinks, OCI metadata).
  • Every old ID resolves to its new module in modules, deployment modules and modules.json. Retired implementations are not in the bundle build graph.
  • Fixture image rebuilds to the same digest with 1, 2 and default worker counts.
  • Not done: full bundle build and real size (too large for the sandbox), and an end-to-end test in Goval.

Rollout

  • Publishing is a manual Semaphore promotion. Kargo's nixmodules-bundle Warehouse then picks the new digest for infra-staging. No pool uses OCI right now.

  • A cold node would still download ~8.2 GiB of bundle plus ~7.1 GiB of other images (1.7 to 2.2 min at 120 to 150 MB/s). Add a node pre-pull with a readiness gate before turning OCI back on.

  • Goval's guest kernel supports zstd EROFS. Host zstd EROFS support is not confirmed; it only matters if containerd mounts the layer on the host.

  • Before wider rollout, smoke-test VM startup and a few migrated projects in infra-staging.

  • Revert this PR and republish to restore the previous bundle and modules.

  • This is fully backward and forward compatible

~ written by ⠕ Replit

Keep development module closures in DAX-eligible chunks and compress cold files with zstd. Check layouts, contents, and worker-independent layer digests with the small fixture.

Assisted-by: Replit
@rahultee
rahultee marked this pull request as ready for review October 13, 2026 23:14
@rahultee
rahultee requested a review from a team as a code owner October 13, 2026 23:14
@rahultee
rahultee requested review from ryantm and removed request for a team October 13, 2026 23:14
Use zstd throughout bundle-oci and remove historical GHC 9.0, 9.2, and 9.4 implementations. Preserve their module IDs through the existing upgrade map to GHC 9.10. Drop chunking now that there is no deliberately uncompressed hot set.

Assisted-by: Replit
@rahultee rahultee changed the title Compress cold bundle-oci files while keeping hot closures DAX-eligible Compress the full bundle-oci layer and remove retired GHC versions Oct 14, 2026
Remove historical implementations and active Python 3.9, Node 18/23, and Vue Node 18 definitions. Preserve old module IDs by mapping them to newer remaining runtimes. Keep Swift 5.8, move both Swift aliases to it, and use Go 1.25 in development bundles.

Assisted-by: Replit
@rahultee rahultee changed the title Compress the full bundle-oci layer and remove retired GHC versions Compress the full bundle-oci layer and retire outdated module versions Oct 14, 2026
@rahultee rahultee changed the title Compress the full bundle-oci layer and retire outdated module versions Compress bundle-oci and retire outdated module versions Oct 14, 2026
@rahultee
rahultee merged commit 5a48aff into main Oct 14, 2026
8 checks passed
@rahultee
rahultee deleted the bundle-oci-hybrid-compression branch October 14, 2026 01:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants