Repository navigation
Investigate mini-CTK cache duplication after HTTP 429 restore failures #3074
Description
Maintainer investigation; not currently open for implementation PRs. This affects shared CI infrastructure. Please wait for maintainer agreement on the approach before starting implementation; observations and suggestions are welcome.
Summary
Transient GitHub Actions cache restore failures can lead to redundant mini-CTK archives on PR branches, consuming capacity that could otherwise retain useful compiler caches.
This issue follows up on #3071. We are not pursuing its default-branch-only save proposal because PRs introducing new toolkit configurations benefit from caching between iterations. The focus here is understanding and preventing unnecessary copies while preserving that behavior.
Observed behavior
Two examined jobs show the same sequence:
- A CUDA 12.9.1 cache restore receives HTTP 429: Too Many Requests.
- The action warns that restoration failed, then reports Cache not found.
- The workflow downloads the toolkit and saves another PR-scoped copy using the same cache key.
Examples:
- PR #3035: job 112907497057. The warning says the rate limit resets in eight seconds.
- PR #3046: job 113043762946. The warning says the rate limit resets in six seconds. A sibling job restores the same key successfully before the failing job saves its copy.
The matching shared cache was already present on main. GitHub's cache lookup includes default-branch fallback; these observations point to transient restore failures rather than a missing fallback configuration.
Log timestamps and affected key
All timestamps below are UTC on 2026-10-07.
| Job | HTTP 429 / reported miss | Save of the same key |
|---|---|---|
| #3035, 112907497057 | 16:03:52 | 16:05:10 |
| #3046, 113043762946 | 22:16:54 | 22:18:05 |
The sibling job in #3046 restores the key at 22:17:21.
Affected key:
mini-ctk-12.9.1-linux-64-d12979cd4e49e7936d5e589fcded5123fe7abaeef89cb5c8e3fd4b88c8cb021d
The current restore step and save condition are relevant starting points.
Impact and caveats
The October 9 inventory documented in #3071 identified 23 PR-scoped entries totaling 10.91 GiB that matched a main entry's key and cache version. This establishes potentially redundant storage, but does not establish why every copy was created. Only the two jobs above have been traced to HTTP 429 failures.
The earlier 11-13 GiB estimate assumed broadly removing PR-scoped saves. It is not a supported savings forecast for this investigation. Preventing new duplicates would also not remove existing entries automatically.
Investigation questions
- Can transient restore failures be retried with bounded backoff, taking account of the service's reset information?
- How can we distinguish a genuinely absent key from a failed lookup before deciding to save another copy?
- How can we validate reduced duplication while preserving useful PR caching for new toolkit configurations and reliable toolkit downloads when the cache service is unavailable?
Removing mini-CTK's extra compression layer remains a separate optimization; it is outside this investigation.
Activity