CI: Lychee reliability improvement ideas #2652

Why this matters

CUDA Python CI already has significant infrastructure and test flakiness
that first-attempt failures and manual reruns are causing significant
friction. Each false red interrupts review, costs triage time, obscures genuine
regressions, and stretches feedback loops, especially during stacked work and
release preparation.

The rendered-docs link check compounds that existing flakiness. A cold sweep
checks roughly 13,000 unique URLs, and currently one transient response from
one external service can turn an otherwise healthy workflow red:

• PR #2641 job 94978076797
  checked 13,166 unique links. Its only failure was an HTTP 500 for the valid,
  merged CUDA Python PR [pathfinder] RTLD_DI_LINKMAP-based new implementation of abs_path_for_dynamic_library() #834.
• PR #2650 job 95260538160
  checked 12,711 unique links. Its only failure was an HTTP 429 for a valid
  docs.conda.io page.

Both were cold-cache checks, both failures were unrelated to the pull request
under test, and both became manual-rerun candidates. The important point is
that these were not useful broken-documentation findings: they were isolated
external-service transients that made the entire Docs job look unhealthy.

The two incidents do not establish the overall first-attempt failure rate, but
they identify one source of avoidable friction that seems reasonably easy to address:
A bounded second pass can remove that source without weakening the link check, and recovery
notices will let us measure how often it helps.

Recommendation

Add one conditional, delayed second Lychee pass to the rendered-docs job.
The first pass should remain the normal fast path. If it reports unresolved
links, wait for a bounded cooldown (initially 60 seconds) and run Lychee once
more against the same files and the same on-disk cache.

This is a small, useful reliability improvement:

• A successful first pass adds no delay.
• A transient 429, 5xx, timeout, or network error gets one additional chance
  to recover without a manual workflow rerun.
• Successful external checks from the first pass are reused; failed external
  requests are not persisted and therefore go back to the network.
• A persistent broken or unavailable link still makes the job fail.

As a separate small improvement, give every workflow execution a new cache key
generation so a rerun can save refreshed successful results instead of being
stuck with an immutable exact-SHA cache.

Correct cache model

This is not a cache-poisoning problem.

Lychee v0.24.2 does not persist failed responses such as 429 or 500.
CacheStatus::Error entries are discarded when loading and skipped when
storing. The cache saved after a failed sweep therefore contains successful
responses but not the failed URL.

The configured max cache age of one day applies to successful entries:

• An immediate rerun can reuse the successful checks and must re-request the
  failed URL.
• A rerun after the one-day age limit performs a broader fresh check.
• Both reruns remain meaningful; the failed response is never replayed from
  disk.

References:

• Lychee v0.24.2 cache implementation
• Lychee PR #2105: never cache errors on disk
• GitHub Actions cache behavior

Phase 1: one bounded automatic second pass

Desired behavior

1. Run Lychee normally, but ask the pinned action to expose rather than enforce
   Lychee's first exit code.
2. If the action itself completed successfully and Lychee reports its
   broken-link exit code (2), emit a notice and wait 60 seconds.
3. Run the same Lychee check once more with normal hard-failure behavior.
4. Fail immediately for any other nonzero or missing Lychee exit code.
5. Save the cache after either outcome, as the workflow does today.

The pinned lychee-action exposes the raw exit_code output and supports
fail: false. That is preferable to broadly applying continue-on-error:
action download/setup failures still fail immediately, while a normal Lychee
link-check result can drive the conditional retry.

The retry condition should check both facts:

• The first action step outcome is success.
• Its exit_code output is exactly 2, Lychee's broken-link result.

The outcome check prevents a skipped or failed setup step with an empty output
from accidentally triggering the retry. A separate guard must fail the job for
an empty output or Lychee exit code 1 (runtime/input error) or 3
(configuration error). Otherwise fail: false would accidentally turn those
deterministic errors into success.

Illustrative workflow shape

The exact argument list is omitted here; both action invocations should use the
existing arguments unchanged and should share one centrally defined value.

- name: Check rendered docs links
  id: check-rendered-docs-links
  if: ${{ <existing rendered-docs condition> }}
  uses: lycheeverse/lychee-action@<current pinned SHA>
  with:
    args: &rendered-docs-lychee-args >-
      <existing arguments>
    fail: false
    failIfEmpty: true
    output: lychee-rendered-html-first.md
    <other existing inputs>

- name: Wait before retrying rendered docs links
  if: ${{ steps.check-rendered-docs-links.outcome == 'success' && steps.check-rendered-docs-links.outputs.exit_code == '2' }}
  run: |
    echo "::notice::Lychee found unresolved links; retrying in 60 seconds."
    sleep 60

- name: Retry rendered docs links
  id: retry-rendered-docs-links
  if: ${{ steps.check-rendered-docs-links.outcome == 'success' && steps.check-rendered-docs-links.outputs.exit_code == '2' }}
  uses: lycheeverse/lychee-action@<current pinned SHA>
  with:
    args: *rendered-docs-lychee-args
    fail: true
    failIfEmpty: true
    output: lychee-rendered-html-retry.md
    <other existing inputs>

- name: Fail on non-link-check Lychee errors
  if: ${{ steps.check-rendered-docs-links.outcome == 'success' && steps.check-rendered-docs-links.outputs.exit_code != '0' && steps.check-rendered-docs-links.outputs.exit_code != '2' }}
  run: |
    echo "::error::Lychee failed with exit code '${{ steps.check-rendered-docs-links.outputs.exit_code }}'."
    exit 1

- name: Report recovered rendered-docs link check
  if: ${{ steps.retry-rendered-docs-links.outcome == 'success' && steps.retry-rendered-docs-links.outputs.exit_code == '0' }}
  run: |
    echo "::notice::Rendered-docs link check recovered on its second pass."

Keep the existing cache-save step after both action invocations and retain its
always() condition. The second pass reads the .lycheecache file directly from
the workspace, so it does not need an intermediate Actions cache save/restore.

The second action invocation still reparses the rendered documents and checks
fast local links. The important optimization is that successful external
requests can be served from the first pass's on-disk cache; only unresolved
external requests need network rechecking.

Use separate first-pass and retry in-workspace output names and log sections.
The files disappear with the runner unless we also upload them as artifacts;
durable reports are optional rather than required for the first change. The
logs should clearly say when a retry happened and whether it recovered.
Otherwise the improvement would reduce failures while hiding the frequency of
the underlying flakiness.

A second lychee-action invocation also repeats action setup and installation.
That small cost and additional setup-failure opportunity occur only after a
first-pass link failure. Calling the installed binary directly could avoid the
cost, but would duplicate more of the action's token, output, and failure
handling; the second pinned action is the safer initial implementation.

Phase 2: make cache generations refreshable

The current primary key is effectively:

docs-rendered-lychee-<PR_NUMBER>-<SHA>

GitHub Actions cache entries are immutable. On a rerun of the same SHA after
Lychee's one-day success-entry age limit:

1. GitHub restores the exact old cache.
2. Lychee discards its expired entries and checks them again.
3. The workflow skips saving because the restore was an exact cache hit.
4. A later rerun of that same SHA restores the same stale cache again.

This did not cause either observed first-attempt failure. It is a separate
efficiency/lifecycle weakness.

Use a versioned, per-execution primary key:

key: docs-rendered-lychee-v1-${{ env.PR_NUMBER }}-${{ github.sha }}-${{ github.run_id }}-${{ github.run_attempt }}
restore-keys: |
  docs-rendered-lychee-v1-${{ env.PR_NUMBER }}-${{ github.sha }}-
  docs-rendered-lychee-v1-${{ env.PR_NUMBER }}-

This preserves the preference for the newest cache from the same PR and SHA
while allowing every execution to save a refreshed immutable snapshot.
Including both run_id and run_attempt also avoids collisions between separate
runs of the same SHA, not just attempts within one run. The v1 component gives
us an intentional cache-format invalidation mechanism.

The tradeoff is increased cache churn because every Docs execution uploads a
new snapshot, even when the restored entries remain fresh. Measure the cache
file size and expected run volume before deciding whether that cost is
material. This change is orthogonal to the same-job second pass and could be a
separate commit.

Phase 3: report the 5xx retry gap upstream

There is a relevant Lychee v0.24.2 implementation gap:

• Rejected HTTP 429 responses are considered retryable.
• Ordinary rejected 5xx responses appear not to be retried through the same
  error path.
• The retry classification for RejectedStatusCode matches only 429, even
  though a separate StatusCode helper classifies 5xx as retryable.
• Current Lychee upstream still appears to retain this mismatch as of this
  analysis.

Therefore the Conda 429 likely received the configured internal retries, while
the generic retry loop likely did not retry the rejected 500 response for the
original GitHub URL. Other GitHub-specific fallback requests may still have
occurred, so this is not a claim about the total number of network requests.
We should avoid saying that both incidents exhausted three retries.

An upstream Lychee issue or PR should propose routing rejected status codes
through the common status-code retry predicate and adding 5xx regression
tests:

• Lychee v0.24.2 retry loop
• Lychee v0.24.2 retry classification
• Related Lychee issue #2193

The CUDA Python outer retry should not wait for that upstream fix. It covers
429, 5xx, timeouts, network errors, and transients lasting longer than the
built-in retry window while retaining a strict final result.

Conditional ideas, not first choices

Host-specific throttling

Lychee suggested host-specific throttling after the docs.conda.io 429:

[hosts."docs.conda.io"]
concurrency = 1
request_interval = "1s"

This syntax is valid, but CUDA Python currently appears to contain only one
unique docs.conda.io URL. Per-host concurrency therefore may not have caused or
prevented this incident; shared runner-IP rate limiting is another plausible
explanation. Add this only if Conda 429s recur or the number of links to that
host grows. It also cannot address unrelated GitHub 5xx responses.

More built-in retries

Increasing max-retries or the wait time could help 429 responses, but it has
lower value than one outer pass:

• The current configuration already retries 429 with increasing waits.
• More retries extend every persistent 429 failure.
• The v0.24.2 5xx classification gap remains.

Warm new PRs from a trusted default-branch cache

Every new PR currently starts cold because the cache key is PR-specific and
pull-request caches are isolated. A scheduled or trusted-main cache warmer
could reduce cold sweeps for new PRs.

Treat this as a later optimization. Main currently skips this rendered-docs
check, and a useful shared cache would need a trusted producer, regular
generations, compatible restore prefixes, and careful freshness/security
review. It must not require pull_request_target or allow untrusted PR code to
write caches consumed by the default branch.

Non-goals

• Do not accept 429 or 5xx responses as successful.
• Do not exclude an entire domain because it occasionally flakes.
• Do not make the final rendered-docs link result advisory.
• Do not add cache-exclude-status for these errors; Lychee already avoids
  persisting them.
• Do not increase the cache age merely to conceal stale or broken links.
• Do not add an unbounded retry loop.
• Do not claim this addresses all CUDA Python CI flakiness.

Acceptance criteria

• A successful first pass runs no delay and no second pass.
• A first-pass broken-link result (exit code 2) triggers exactly one bounded
  delayed retry.
• A recovered retry makes the job green and emits a visible recovery notice.
• A persistent bad or unavailable URL leaves the job red with the final URL
  and status visible.
• Successful external responses from the first pass are reused on the second
  pass.
• Action setup failures, absent exit-code output, Lychee runtime/configuration
  errors, and empty input are not silently converted into success or treated
  as transient link failures.
• The cache-save step still runs after success or final failure.
• A manual rerun can save refreshed successful results under a new cache key.
• The first-pass and recovered-retry rate can be determined from job logs.

Validation plan

Before merging:

1. Run actionlint and the repository YAML checks.
2. Confirm that the pinned action sets outputs.exit_code when fail is false.
3. Confirm the first-pass success path performs no sleep and no second action
   invocation.
4. Use a deterministic local HTTP server in a test PR, with one route that
   fails once and then succeeds and another that always fails. Avoid adding a
   new external test dependency.
5. Use request counters to confirm the second pass reuses successful external
   results, rechecks the transient route, and makes the job green.
6. Confirm the persistent failing route leaves the job red.
7. Exercise invalid arguments/configuration, absent exit-code output, and zero
   discovered links; confirm each remains red without a link retry.
8. Rerun the same SHA and confirm the new cache generation restores prior
   successes and saves a new refreshed snapshot.

Suggested rollout

1. Add the conditional second pass, recovery notice, and distinct reports.
2. In the same PR or a separate small commit, add per-execution cache
   generations.
3. Monitor recovery notices for a few weeks to quantify the first-pass failure
   rate and the hosts/statuses involved.
4. File the 5xx retry-classification issue upstream and consume the fix when a
   release is available.
5. Add host-specific throttling or a shared trusted cache only if the measured
   results justify the extra policy and complexity.