CI: Consolidate the two release workflows

At certain point in time (#1626) the release workflow was rage-forked into two

https://github.com/NVIDIA/cuda-python/blob/main/.github/workflows/release-cuda-pathfinder.yml (new, for pathfinder only)
https://github.com/NVIDIA/cuda-python/blob/main/.github/workflows/release.yml (old, for all components)

with the understanding that eventually we want to join them. But it never happened after the team moved on. It's time to get it cleaned up and organized.

Proposed consolidation plan

release.yml already exposes cuda-pathfinder as a component, so consolidation should move the dedicated workflow's additional release policy into the shared graph. Simply deleting release-cuda-pathfinder.yml would lose verification, concurrency, release metadata, and automatic finalization.

Target flow

tag/input validation --+-- determine exact tag-triggered CI run
                       +-- check release notes
                       +-- create/reuse draft
                                   |
                                 docs
                          +--------+--------+
                     upload assets      TestPyPI
                          |                  |
                          |       [Pathfinder verify]
                          |                  |
                          |                PyPI
                          |                  |
                          |       [Pathfinder verify]
                          +--------------+---+
                               [Pathfinder finalize]

For cuda-core, cuda-bindings, and cuda-python, the bracketed jobs remain skipped and the GitHub release remains a draft, preserving today's behavior.

1. Keep the shared workflow interface

A full Pathfinder release should use:

component: cuda-pathfinder
release-action: full-release
git-tag: cuda-pathfinder-vX.Y.Z
empty run-id, backport-git-tag, and dry-run-docs-branch

For full Pathfinder releases only:

Require dispatch from the default branch.
Reject a supplied run-id, or validate that it is a successful tag-push run for the exact tag SHA with the expected artifacts. Rejecting it is simpler and matches the dedicated workflow.
Serialize runs under release-cuda-pathfinder with cancel-in-progress: false.

Do not impose those restrictions on the other components: maintenance/backport releases may intentionally use other workflow branches or explicit run IDs.

2. Preserve Pathfinder draft behavior

For Pathfinder, draft preparation should:

Resolve an exact refs/tags/<tag> ref.
Use gh release view "$TAG" rather than searching a potentially limited release list.
Reject an already-published release and reuse an existing draft.
Create a missing draft with --verify-tag, --latest=false, title cuda-pathfinder vX.Y.Z, and the existing release-notes, versioned-docs, PyPI, and Conda links.

Other components should retain the current generic body and manual publication. Tag, notes, and CI-run validation should finish before draft creation so invalid requests do not leave orphan drafts.

3. Reuse the generalized machinery

The shared implementation already supports Pathfinder:

ci/tools/check_release_notes.py knows the Pathfinder tag namespace and notes path.
build-docs.yml builds Pathfinder docs from the selected tag/run.
release-upload.yml creates the source archive/checksum, downloads wheels, supports dry-run artifacts, and uploads release assets.
ci/tools/download-wheels collects Pathfinder artifacts.
ci/tools/validate-release-wheels checks the exact version and rejects development/local wheels.
release.yml already owns the TestPyPI and PyPI OIDC publishing jobs.

These should replace the dedicated workflow's duplicated artifact-download/upload blocks. Keep the actual pypa/gh-action-pypi-publish steps directly in top-level release.yml; see the OIDC note below.

4. Preserve Pathfinder's docs-failure policy

upload-archive and publish-testpypi may continue after doc == failure only when all tag, notes, and run-ID prerequisites succeeded and the run is a full Pathfinder release. Cancellation must remain blocking. Every other component must continue to require doc == success.

This requires explicit !cancelled() and needs.<job>.result checks; otherwise GitHub propagates a failed/skipped dependency before the product exception is evaluated.

5. Add reusable registry verification

Add one small helper such as ci/tools/verify-published-package, called by separate verify-testpypi and verify-pypi jobs. Preserve:

A fresh virtual environment.
Six installation attempts with 30-second delays.
Exact cuda-pathfinder==X.Y.Z installation.
Successful cuda.pathfinder import.
Exact __version__ comparison.
No package cache while checking registry propagation.

Keep verification separate from publishing. If registry propagation is slow, rerunning verification must not retry an immutable package upload.

publish-pypi must explicitly tolerate the Pathfinder-only TestPyPI verifier being skipped for other products:

publish-pypi:
  needs:
    - determine-run-id
    - publish-testpypi
    - verify-testpypi
  if: >-
    ${{
      !cancelled() &&
      inputs.release-action == 'full-release' &&
      needs.determine-run-id.result == 'success' &&
      needs.publish-testpypi.result == 'success' &&
      (
        inputs.component != 'cuda-pathfinder' ||
        needs.verify-testpypi.result == 'success'
      )
    }}

Without this explicit condition, adding a skipped verifier to needs would also skip PyPI publication for every non-Pathfinder component.

6. Add Pathfinder-only finalization

After both verify-pypi and upload-archive succeed:

Check the versioned docs URL; HTTP and transport failures should be warnings.
Confirm the GitHub release is still a draft.
Run gh release edit --draft=false --latest=false.
Write the docs URL to the job summary.

Do not generalize finalization to other components; doing so would materially change their release process.

7. Harden shared input, permissions, and concurrency handling

Pass every free-form dispatch value through env and use quoted shell variables. Do not inject ${{ inputs.git-tag }}, ${{ inputs.run-id }}, etc. directly into run: source.
Qualify release source checkouts as refs/tags/....
Grant actions: read where cross-run artifacts are read, contents: read for checkout, contents: write only for draft/assets/finalization, and id-token: write only for registry publishing.
Validate wheels before uploading any GitHub assets, avoiding partially populated drafts after an artifact failure.

release-upload.yml currently groups concurrency by workflow and branch and cancels in progress. Since releases are normally dispatched from main, unrelated products/tags can cancel one another's uploader. Key it by component and tag instead, for example:

group: ${{ github.workflow }}-release-upload-${{ inputs.component }}-${{ inputs.git-tag }}

Prefer cancel-in-progress: false for release mutations. At minimum, the new key removes unintended cross-product cancellation.

8. Migrate trusted-publisher configuration before cutover

PyPI and TestPyPI Trusted Publishers bind authorization to the exact workflow filename and optionally the GitHub environment. Before deleting the dedicated workflow:

Register .github/workflows/release.yml for cuda-pathfinder on TestPyPI and PyPI.
Keep the existing testpypi and pypi environment names and protections.
Retain the old publisher records during the transition.
Remove the old release-cuda-pathfinder.yml publishers only after a successful unified release.

PyPI documents the workflow-filename requirement and says reusable workflows cannot currently be registered as Trusted Publishers. Therefore, moving the actual publishing action into another reusable workflow would be the wrong kind of deduplication.

9. Resolve the Pathfinder post-release version edge case

The shared notes checker skips new notes for .postN releases, while the dedicated workflow requires a literal .postN-notes.rst. Pathfinder docs intentionally publish post releases under the base X.Y.Z directory, while the existing release body uses the full version. The integration should explicitly maintain separate package and docs versions so generated docs URLs are correct. Alpha/beta policy was deferred in #1626 and should remain outside this issue.

10. Validation and rollout

Run actionlint via pre-commit and unit-test the new verification helper.
Exercise the existing release.yml dry-run matrix for mainline bindings, bindings/Python backports, and mainline Python where relevant.
Add a Pathfinder dry run using a real successful tag-triggered CI run, leaving run-id empty.
Test wrong-prefix/nonexistent tags, non-default-branch full release, supplied Pathfinder run ID, existing published release, and manually undrafted release.
Verify that TestPyPI failure blocks PyPI, and that PyPI verification or asset failure leaves the GitHub release as a draft.
Verify that docs failure remains nonblocking only for Pathfinder.
Verify that concurrent Pathfinder releases serialize and that another product cannot cancel the Pathfinder uploader.
Prove OIDC, propagation checks, and finalization during the next scheduled Pathfinder release.

After successful cutover, delete .github/workflows/release-cuda-pathfinder.yml and update the stale reference in .github/RELEASE-core.md.

(this was done with 6.1 sol)
I would make release.yml the single release entry point, reuse its existing stages, and preserve Pathfinder's different release policies through narrowly scoped conditions. One behavior should deliberately change: a failed docs build must fail the Pathfinder release, just as it does for the other products.

The two workflows already share much of their machinery. The important differences are in preparation, failure handling, and finalization:

Behavior	Pathfinder today	Other products through `release.yml`
Dispatch	Default branch only; supplying a tag starts a full release	Existing inputs, including a default of `dry-run`
Draft creation	After notes and CI-run validation; standard title and body	Independent preparation job; generic title and body
Docs failure	Assets and package publication may continue; this is a bug to correct during consolidation	Blocks assets and TestPyPI publication
Package verification	Install/import/version checks after both indexes	No post-publication checks
Publisher checkout	No checkout; consumes selected CI-run artifacts	Checkout plus shared download/validation scripts
GitHub finalization	Automatically publishes the draft, never marks it latest	Leaves the draft for manual publication
Concurrency	Serializes Pathfinder releases without cancelling the active run	No equivalent entry-point serialization

Those distinctions are visible in release-cuda-pathfinder.yml and release.yml. The proposal preserves the component-specific behavior except the docs-failure exception: successful docs are a release prerequisite for every product.

Keep the existing dispatch interface and preparation jobs.

I would retain component, release-action, git-tag, backport-git-tag, run-id, and dry-run-docs-branch, including the existing dry-run default. Pathfinder already appears in the component choices.

A Pathfinder full release would require selecting cuda-pathfinder, selecting full-release, and supplying its tag. It would still complete through one dispatch, although the interface would no longer be literally tag-only. Changing the shared default to full release would affect every other product.

I would also retain the existing determine-run-id, check-tag, and check-release-notes jobs, preserving their existing behavior for the other products. Pathfinder would receive small conditional branches:
- Require default-branch dispatch for a full release. Allow its new dry-run mode from a development branch.
- Resolve the exact refs/tags/<git-tag> ref, preserving the dedicated workflow's guarantee that the input names an existing tag.
- Preserve automatic selection of a successful tag-triggered CI run. If the shared run-id override is accepted for Pathfinder, validate that it belongs to the same tag commit and qualifying CI workflow, with the expected push event and successful conclusion. Other products would retain their current override behavior.
The existing lookup-run-id helper already implements the automatic tag-run selection; there is no reason to duplicate that lookup.

Release-note checking needs special care. Pathfinder currently requires the versioned notes file to exist, including for .post releases. The generic checker rejects empty notes but skips .post versions. Substituting it unchanged would alter Pathfinder's policy. I would preserve the Pathfinder file-presence check and leave the generic checker's backport and post-release rules unchanged. See check_release_notes.py.
Add one Pathfinder-only draft job after preparation.

In check-tag, the generic draft-creation logic would run only for non-Pathfinder products. Pathfinder would validate its tag there, then use a new ensure-pathfinder-draft job requiring all three preparation jobs to succeed.

This preserves an important ordering difference: Pathfinder currently validates notes and finds its CI run before creating a draft. Moving its draft creation into the existing generic step would create drafts even when one of those prerequisites fails.

The new job would preserve the dedicated workflow's release behavior:
- Query the requested release directly by tag.
- Reject an already published release.
- Reuse an existing draft without overwriting its body.
- Otherwise create the same cuda-pathfinder v<version> title and body, including release-note, versioned documentation, PyPI, and Conda links.
- Retain --verify-tag and --latest=false.
It would have contents: write, and run only for a Pathfinder full release. Dry runs would skip it without creating or changing a GitHub release.
Share the docs and asset stages, requiring successful docs for every product.

There should be one docs job calling the existing build-docs.yml, using the selected component, tag, and resolved CI run. Its existing release-docs routing, dry-run artifacts, and optional non-production deployment branch can remain.

The shared docs job would depend on the three preparation jobs and the optional Pathfinder draft job. Its condition must accept the draft job being skipped for other products and dry runs, but require it to succeed for Pathfinder full releases.

Both asset upload and TestPyPI publication would require needs.doc.result == 'success' for every product, in addition to successful preparation and the applicable draft prerequisite. Failed, skipped, or cancelled docs must block those jobs. This intentionally removes Pathfinder's current docs-failure exception and matches the other products' existing release gate.

PyPI publication, package verification, and GitHub release finalization would also remain downstream of that successful docs gate. A failed docs build must leave the release run failed, with no asset uploads, package publication, or GitHub release finalization. Any draft created during preparation would remain a draft.

Conditions that handle optional Pathfinder-only dependencies still need explicit status functions and prerequisite-result checks. GitHub otherwise applies an implicit success() condition, and failed or skipped dependencies can skip downstream jobs. Adding a Pathfinder-only prerequisite without handling its expected skipped state could prevent another product from releasing. Those expressions must explicitly require successful docs wherever docs are a direct prerequisite; !cancelled() alone must never bypass a failed docs build. GitHub documents both behaviors.
Reuse release-upload.yml, including archive creation and checksums.

Pathfinder duplicates archive creation, checksum generation, wheel downloading, and release uploads that already exist in release-upload.yml. I would remove its separate asset job implementation and call that reusable workflow.

There are two details to preserve.

First, upload order differs. Pathfinder currently collects the archive and wheels before uploading them together. The reusable uploader uploads the archive before downloading and validating wheels. For Pathfinder, I would defer that first upload and upload the archive, checksum, and wheels together at the end. Other products would retain their existing two upload steps and ordering.

Second, the shared uploader validates wheel versions, which the dedicated Pathfinder asset job currently does not. Applying that validation to Pathfinder is a useful additional safeguard, but I would identify it as a behavior change and test it explicitly. New extracted tooling must also be available when releasing older tags: a helper added on the workflow branch will not automatically exist in a checkout of a historical release tag.

Archive naming, directory prefixes, checksum format, and --clobber behavior should remain unchanged.
Share the two publishing jobs without adding a Pathfinder checkout.

I would retain one publish-testpypi job and one publish-pypi job, preserving their existing environments, publishing action, and index endpoints.

For other products, the checkout, download-wheels, and validate-release-wheels steps would remain as they are, guarded by component != 'cuda-pathfinder'.

Pathfinder would instead use its existing checkout-free artifact download. I would define that step once with a YAML anchor and reference it from both publishing jobs. This removes the two substantial duplicated download blocks while retaining their component selection, test-artifact exclusion, wheel flattening, and selected-run provenance. GitHub Actions supports YAML anchors and aliases.

This is one place where complete unification is not immediately appropriate. The generic download-wheels helper additionally excludes Python 3.15 artifacts; the Pathfinder publishing blocks do not. Using that helper also requires obtaining repository code, which would erase the deliberately checkout-free publishing path.

I would initially share the identical Pathfinder download block through an anchor. A later extraction could unify the underlying download implementation, but it should preserve those different policies explicitly.
Keep verification separate and Pathfinder-only.

I would add verify-testpypi and verify-pypi, retaining the dedicated workflow's checks:
- Fresh virtual environment.
- Exact package-version pin.
- Six installation attempts, separated by 30 seconds.
- Existing index options.
- Import of cuda.pathfinder and comparison of __version__ with the release version.
One anchored verification script, with index settings supplied through each job's environment, would remove the duplicated retry and version-check logic.

Separate jobs preserve the existing separation between publishing with OIDC permissions and installing/importing the published package. Other products would skip both jobs.

The shared PyPI publisher would require successful docs and successful TestPyPI publication for every product, plus successful TestPyPI verification specifically for Pathfinder. Its condition would explicitly accept skipped verification for other products. For example:
```
needs:
  - determine-run-id
  - doc
  - publish-testpypi
  - verify-testpypi
if: >-
  ${{
    !cancelled() &&
    inputs.release-action == 'full-release' &&
    needs.determine-run-id.result == 'success' &&
    needs.doc.result == 'success' &&
    needs.publish-testpypi.result == 'success' &&
    (
      (inputs.component == 'cuda-pathfinder' &&
       needs.verify-testpypi.result == 'success') ||
      (inputs.component != 'cuda-pathfinder' &&
       needs.verify-testpypi.result == 'skipped')
    )
  }}
```
The verification jobs would require their corresponding publishing job to succeed. Handling the optional verification job's skipped state for other products must not introduce a path around the docs-success gate: TestPyPI publication carries that gate through to PyPI and both verification stages.
Add Pathfinder-only finalization.

A finalize-pathfinder job would require successful docs, successful asset upload, and successful PyPI verification. It would preserve:
- Checking the versioned documentation URL.
- Warning on a non-200 HTTP response.
- Rechecking that the GitHub release is still a draft.
- Publishing with --draft=false --latest=false.
- Adding the documentation URL to the job summary.
This final URL check is for deployment propagation after a successful docs job, not a substitute for a successful docs build. The current URL check warns on HTTP errors, but a curl transport failure can still fail the job; preserving that distinction avoids quietly broadening its failure tolerance.

No other product would acquire automatic finalization. Leaving those releases drafted is part of their existing process, including the documented manual cuda-core publication step.
Preserve concurrency at both levels.

Pathfinder full releases would retain the constant release-cuda-pathfinder concurrency group with cancel-in-progress: false. Other products and dry runs would use unique run-based groups, preserving their current lack of entry-point serialization.

The reusable uploader also needs adjustment. Its current group is based on the caller workflow name and dispatch ref, with cancellation enabled. After consolidation, Pathfinder and another product dispatched from main would share that group and could cancel each other's upload.

I would use a Pathfinder-specific, run-based upload group with cancellation disabled, while retaining the exact existing group expression and cancellation setting for other components. Since the uploader already receives component, this can be done through component-specific expressions without adding more reusable-workflow inputs.

The upload group must differ from the enclosing Pathfinder release group. GitHub's documentation also explains that reusable workflows inherit their caller's workflow name for this context. Reusable-workflow concurrency guidance.

Before retiring the dedicated workflow, I would validate the scheduling rules as carefully as the scripts: all four products, dry and full modes, successful and failed docs, skipped Pathfinder-only jobs, failed verification, failed asset upload, and cancellation. For every product, a failed docs build must fail the release run and prevent asset uploads, TestPyPI/PyPI publication, package verification, and GitHub release finalization. Other products must still skip Pathfinder verification/finalization and leave the GitHub release drafted.

I would then run actionlint, relevant helper tests, and the existing mainline/backport dry-run matrix documented at the top of release.yml, adding Pathfinder and cuda-core coverage. Dry runs cannot establish that OIDC publication and finalization work, so those paths also need mocked failure tests and confirmation during the first authorized Pathfinder release.

Finally, Pathfinder's trusted-publisher registrations must authorize release.yml on both PyPI and TestPyPI before removing the old workflow. The workflow filename is part of the publisher configuration, so the YAML change alone is insufficient. PyPI's configuration requirements. I would also update the Pathfinder workflow reference in RELEASE-core.md.

This would leave one dispatch workflow, one docs implementation, one archive implementation, two shared publication stages, and single definitions of Pathfinder's download and verification logic.

We'll ignore the fact that pathfinder build can continue if doc build fails, as Ralf K suggests it's probably a bug.

Navigation Menu

Repository navigation

CI: Consolidate the two release workflows #3012

Proposed consolidation plan

Target flow

1. Keep the shared workflow interface

2. Preserve Pathfinder draft behavior

3. Reuse the generalized machinery

4. Preserve Pathfinder's docs-failure policy

5. Add reusable registry verification

6. Add Pathfinder-only finalization

7. Harden shared input, permissions, and concurrency handling

8. Migrate trusted-publisher configuration before cutover

9. Resolve the Pathfinder post-release version edge case

10. Validation and rollout

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CI: Consolidate the two release workflows #3012

Description

Activity

jpascucci-nv commented on Oct 6, 2026

Proposed consolidation plan

Target flow

1. Keep the shared workflow interface

2. Preserve Pathfinder draft behavior

3. Reuse the generalized machinery

4. Preserve Pathfinder's docs-failure policy

5. Add reusable registry verification

6. Add Pathfinder-only finalization

7. Harden shared input, permissions, and concurrency handling

8. Migrate trusted-publisher configuration before cutover

9. Resolve the Pathfinder post-release version edge case

10. Validation and rollout

jpascucci-nv commented on Oct 7, 2026

jpascucci-nv commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions