Skip to content

ci: temporarily pause Pixi lockfile maintenance #3065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rwgk merged 2 commits into from
Oct 9, 2026
Merged

ci: temporarily pause Pixi lockfile maintenance #3065

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0ba120d
Identify when pixi toml file has updated relative to lock file
jpascucci-nv Oct 9, 2026
286dc85
ci: temporarily pause Pixi lockfile maintenance
jpascucci-nv Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci-nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:

- name: Run CI tools unit tests
run: |
python -m pip install pytest
python -m pip install pytest pyyaml==6.0.3
# Standalone CI tool tests; skip repo-root conftest.py (imports cuda.pathfinder).
python -m pytest -v --noconftest ci/tools/tests

Expand Down
330 changes: 10 additions & 320 deletions .github/workflows/ci-pixi-lockfile-freshness-check.yml
View file
Open in desktop

Large diffs are not rendered by default.

221 changes: 10 additions & 211 deletions .github/workflows/ci-pixi-lockfile-refresh.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,11 @@
#
# SPDX-License-Identifier: Apache-2.0

# Scheduled (and on-demand) pixi lockfile refresh. All workspaces are updated
# in one job and, when anything changes, one PR. See #2298 and #2849.

# Scheduled refreshes are paused. Manual dispatch reports the suspension only.
# Re-enable only after the released Pixi validation described in ci/README.md.
name: "CI: pixi lockfile refresh"

on:
schedule:
# Monday 03:17 Eastern. GitHub handles the DST offset for the IANA zone,
# and a non-zero minute avoids the start-of-hour window where GitHub
# documents scheduled runs being delayed or dropped.
- cron: "17 3 * * 1"
timezone: "America/New_York"
workflow_dispatch: {}

defaults:
Expand All @@ -25,212 +18,18 @@ permissions: {}
jobs:
refresh:
name: pixi update (all workspaces)
if: ${{ github.repository_owner == 'nvidia' }}
runs-on: ubuntu-latest
timeout-minutes: 195
permissions:
contents: write
pull-requests: write
concurrency:
group: pixi-lock-refresh
cancel-in-progress: false
timeout-minutes: 5
steps:
- name: Checkout ${{ github.event.repository.name }}
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
# Full history and tags for package metadata, without historical
# file contents. See ci/README.md for the checkout policy.
fetch-depth: 0
filter: blob:none
ref: ${{ github.event.repository.default_branch }}

- name: Setup pixi
id: setup-pixi
uses: ./.github/actions/setup-pixi

# Pixi 0.73.0's update and lock-read paths can serialize the same solve
# differently (#2804). Canonicalize with the same operation used by the
# freshness workflow before opening the PR. If cuda_core hits the exact
# samples-source pruning defect guarded there, retain its committed lock
# and continue refreshing the other workspaces.
- name: Update and canonicalize all lockfiles
id: update
env:
PIXI_VERSION: ${{ steps.setup-pixi.outputs.pixi-version }}
WORKSPACE_TIMEOUT: 30m
- name: Report temporary suspension
run: |
workspaces="$(python3 ci/tools/list_pixi_workspaces.py)"
workspace_rows="$(
jq -ce '
if type != "array" or length == 0 then
error("workspace inventory must be a non-empty array")
elif any(.[];
type != "object"
or (.manifest | type != "string")
or (.lockfile | type != "string")
or .manifest == ""
or .lockfile == ""
) then
error("each workspace must have non-empty string manifest and lockfile fields")
else
.[]
end
' <<<"${workspaces}"
)"

while IFS= read -r workspace; do
manifest="$(jq -r '.manifest' <<<"${workspace}")"
lockfile="$(jq -r '.lockfile' <<<"${workspace}")"
echo "::group::pixi update (${manifest})"

workspace_status=0
# The single-quoted script expands its variables in the child shell.
# shellcheck disable=SC2016
timeout --kill-after=1m "${WORKSPACE_TIMEOUT}" \
bash --noprofile --norc -xeuo pipefail -c '
manifest="$1"
lockfile="$2"
pixi_version="$3"

committed_blob="$(git hash-object "${lockfile}")"
pixi update --no-install --manifest-path "${manifest}"
# The detector reads this uncommitted post-update state through
# git cat-file, so explicitly persist the blob in the runner-local
# temporary object database.
updated_blob="$(git hash-object -w "${lockfile}")"

check_status=0
pixi lock --check --manifest-path "${manifest}" \
|| check_status=$?
if ((check_status != 0)); then
defect_status=1
if [[ "${manifest}" == "cuda_core" ]]; then
defect_status=0
python3 ci/tools/check_pixi_samples_source_pruning.py \
--original-blob "${updated_blob}" \
--lockfile "${lockfile}" \
--manifest-path "${manifest}/pixi.toml" \
--pixi-version "${pixi_version}" \
--check-status "${check_status}" \
|| defect_status=$?
fi

if ((defect_status == 0)); then
echo "::warning title=Known Pixi samples lockfile bug::Pixi ${pixi_version} removed only the nine samples local-source references after updating ${lockfile}. Keeping the committed cuda_core lockfile and continuing with the other workspaces; all other refresh failures remain strict."
git cat-file blob "${committed_blob}" > "${lockfile}"
echo "known-defect-skipped=true" >> "${GITHUB_OUTPUT}"
exit 0
elif [[ "${manifest}" == "cuda_core" ]] && ((defect_status != 1)); then
echo "::error::Could not determine whether ${lockfile} matches the known Pixi samples-source pruning defect. Resolve the detector error above and rerun."
exit "${defect_status}"
fi

exit "${check_status}"
fi

canonical_blob="$(git hash-object "${lockfile}")"
verify_status=0
pixi lock --check --manifest-path "${manifest}" \
|| verify_status=$?
verified_blob="$(git hash-object "${lockfile}")"
if ((verify_status != 0)) || [[ "${canonical_blob}" != "${verified_blob}" ]]; then
echo "::error::pixi ${pixi_version} did not reach a successful byte-stable check for ${lockfile} (verification exit ${verify_status}, canonical blob ${canonical_blob}, verified blob ${verified_blob}); refusing to open an unstable refresh PR."
exit 1
fi
' bash "${manifest}" "${lockfile}" "${PIXI_VERSION}" \
|| workspace_status=$?

if ((workspace_status == 124 || workspace_status == 137)); then
echo "::error::Timed out after ${WORKSPACE_TIMEOUT} while refreshing '${manifest}'."
elif ((workspace_status != 0)); then
echo "::error::Failed to refresh '${manifest}' (exit ${workspace_status})."
fi
echo "::endgroup::"
if ((workspace_status != 0)); then
exit "${workspace_status}"
fi
done <<<"${workspace_rows}"

echo "::warning title=Pixi lockfile maintenance suspended::Pixi lockfile freshness enforcement and automated refreshes are temporarily suspended pending a verified released Pixi fix. Freshness was not checked."
{
echo "lockfiles<<EOF"
jq -r '.[].lockfile' <<<"${workspaces}"
echo "EOF"
} >> "${GITHUB_OUTPUT}"

- name: Compose PR body
id: pr-body
env:
KNOWN_DEFECT_SKIPPED: ${{ steps.update.outputs.known-defect-skipped }}
PIXI_VERSION: ${{ steps.setup-pixi.outputs.pixi-version }}
run: |
body_path="${RUNNER_TEMP}/pixi-lock-refresh-body.md"
{
echo "Automated lockfile refresh using pixi" \
"\`${PIXI_VERSION}\` (\`pixi update --no-install\`, followed by" \
"byte-stable \`pixi lock --check\` canonicalization for each" \
"refreshed workspace)."
if [[ "${KNOWN_DEFECT_SKIPPED}" == "true" ]]; then
echo
echo "> [!NOTE]"
echo "> \`cuda_core/pixi.lock\` was left unchanged because Pixi" \
"\`${PIXI_VERSION}\` reproduced the exact known samples-source" \
"pruning defect. Other workspaces were refreshed normally."
fi
echo
echo "Lockfiles covered by the refresh:"
python3 ci/tools/list_pixi_workspaces.py \
| jq -r '.[] | "- `\(.lockfile)`"'
echo "### Pixi lockfile maintenance temporarily suspended"
echo
echo "See #2298 and #2849."
echo "Freshness enforcement and automated refreshes are temporarily suspended. This successful job reports the suspension; it does not verify manifest/lock consistency."
echo
echo "> [!IMPORTANT]"
echo "> This PR was opened with \`GITHUB_TOKEN\`. A maintainer with write access must:"
echo ">"
echo "> 1. Select **Approve workflows to run** in the merge box."
echo "> 2. Assign themselves to this PR. The human-generated \`assigned\` event"
echo "> creates the required **PR has assignee, labels, and milestone** check"
echo "> and gives the PR a clear owner."
echo "Source builds and tests continue using the committed lockfiles with \`PIXI_FROZEN=true\`. Dependency changes require manual review and validation."
echo
echo "If the diff looks wrong, close this PR and leave the branch deleted; the"
echo "next scheduled run opens a fresh one if an update is still needed."
} > "${body_path}"
echo "path=${body_path}" >> "${GITHUB_OUTPUT}"

- name: Open or update a lockfile-only PR
id: cpr
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
with:
token: ${{ github.token }}
add-paths: ${{ steps.update.outputs.lockfiles }}
commit-message: "chore: refresh pixi lockfiles"
signoff: true
sign-commits: true
branch: ci/pixi-lock-refresh/all
delete-branch: true
title: "chore: refresh pixi lockfiles"
labels: |
CI/CD
dependencies
body-path: ${{ steps.pr-body.outputs.path }}
author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
committer: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

- name: Warn about required maintainer action
if: ${{ steps.cpr.outputs.pull-request-operation == 'created' || steps.cpr.outputs.pull-request-operation == 'updated' }}
run: echo "::warning::The refresh PR requires a maintainer to approve its workflow runs and assign themselves to create the required metadata check."

- name: Summarize
run: |
{
echo "### pixi lock refresh (all workspaces)"
echo ""
echo "- Operation: \`${{ steps.cpr.outputs.pull-request-operation }}\`"
if [ -n "${{ steps.cpr.outputs.pull-request-url }}" ]; then
echo "- PR: ${{ steps.cpr.outputs.pull-request-url }}"
else
echo "- No lockfile diff; PR not opened."
fi
if [[ "${{ steps.update.outputs.known-defect-skipped }}" == "true" ]]; then
echo "- \`cuda_core/pixi.lock\` retained its committed state because the exact known Pixi samples-source pruning defect was detected."
fi
} >> "$GITHUB_STEP_SUMMARY"
echo "See [CI documentation](https://github.com/NVIDIA/cuda-python/blob/main/ci/README.md#temporary-pixi-lockfile-maintenance-pause) for the tradeoff and re-enable criteria."
} >> "${GITHUB_STEP_SUMMARY}"
6 changes: 3 additions & 3 deletions .github/workflows/ci-pixi-source-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,9 +65,9 @@ env:
# Temporarily bypass Pixi's workspace-wide freshness check: the unrelated
# samples environment oscillates because excluded PyPI dependencies are
# still checked. Frozen mode keeps the committed solution and source builds,
# including nested `pixi run` calls. The separate freshness workflow rejects
# every change except the exact known samples-source pruning defect.
# Restore PIXI_LOCKED after the upstream fix is released and verified.
# including nested `pixi run` calls. Freshness enforcement and automated
# refreshes are temporarily suspended; dependency changes need manual review.
# Restore PIXI_LOCKED after the released-fix validation in ci/README.md.
PIXI_FROZEN: "true"

jobs:
Expand Down
61 changes: 30 additions & 31 deletions ci/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@ For those jobs, use `fetch-depth: 0` with `filter: blob:none`. The checkout
action fetches complete ancestry for all branches and tags, while the blob
filter skips historical file contents. This avoids downloading the large
historical `gh-pages` contents discussed in issue #2197. Git retrieves file
contents as needed when checking out the requested commit or creating the
freshness check's base-commit worktree.
contents as needed when checking out the requested commit.

CI change detection uses this full-history checkout for PR mirrors, so the
actual PR base branch is available when computing its merge base. Non-PR runs
Expand All @@ -25,35 +24,35 @@ The pinned checkout action's
[`README`](https://github.com/actions/checkout/blob/3d3c42e5aac5ba805825da76410c181273ba90b1/README.md#fetch-all-history-for-all-tags-and-branches)
documents `fetch-depth: 0` for fetching all history, branches, and tags.

## Temporary Pixi Samples Lockfile Workaround

Pixi 0.73.0 checks PyPI dependencies that the `samples` environment explicitly
excludes, then drops its local Conda source references during a PyPI-only
refresh. The next refresh restores them, so repeated lock commands do not
converge. A general maintenance refresh cannot repair this defect.

Source-build CI temporarily uses `PIXI_FROZEN=true`: it builds the checkout's
source packages using the committed dependency solution without triggering
Pixi's workspace-wide freshness check.

The separate freshness workflow remains strict except for one exact pattern:
Pixi 0.73.0 exits 1 and removes precisely the nine `samples` source references,
with every other lockfile byte unchanged. The detector also verifies the
expected manifest overrides, local paths, platforms, and source-record
definitions. It rejects the reverse cycle, which would add missing references.
For PRs, the base must produce identical original and repaired blobs, and no
`pixi.toml`, `pyproject.toml`, or Pixi-version input may differ from the base.
Recognized cases emit a warning and restore the committed references.

The refresh workflow uses the same exact detector after updating `cuda_core`.
On a match, it restores the pre-update committed lockfile and continues
refreshing the other workspaces. This postpones `cuda_core` dependency updates
without allowing an unstable lockfile into the generated refresh PR; every
unrecognized refresh failure remains fatal.

Remove the detector and exception, and restore `PIXI_LOCKED=true`, once a
released Pixi passes repeated lock and check commands without changing either
the reduced upstream reproducer or our full workspaces.
## Temporary Pixi Lockfile Maintenance Pause

Pixi lockfile freshness enforcement and automated refreshes are temporarily
suspended because of [Pixi #7215](https://github.com/prefix-dev/pixi/issues/7215)
and [#7000](https://github.com/prefix-dev/pixi/issues/7000). Repeated lockfile
operations can change the committed solution without converging, so a general
refresh is not a reliable repair.

The `CI: pixi lockfile freshness check` workflow still runs on every PR with its
existing `pixi lock --check (all workspaces)` check name. It succeeds with a
warning annotation and job summary explicitly reporting the suspension;
success does not mean freshness passed. The refresh workflow has no schedule.
Manual dispatch emits the same notice without solving dependencies, rewriting
lockfiles, pushing a branch, or creating a PR.

Manifest/lock consistency is not automatically enforced during this pause.
Necessary dependency changes require manual review and validation of the
affected manifests and lockfiles. Real source builds and tests continue using
`PIXI_FROZEN=true`, including nested `pixi run` calls, to build the checkout's
source packages against the committed dependency solution. The pinned Pixi
version and committed lockfiles are unchanged.

Re-enable maintenance only after a released Pixi passes both upstream issue
reproducers and repeated byte-stable update/lock/check cycles across all six
workspaces: the repository root, `cuda_pathfinder`, `cuda_bindings`, `cuda_core`,
`benchmarks/cuda_bindings`, and `benchmarks/cuda_core`. Verify both `pixi update
--no-install` and `pixi lock`, followed by repeated `pixi lock --check` commands,
converge to unchanged lockfile bytes. Then restore strict freshness checks,
scheduled refreshes, and `PIXI_LOCKED=true` in source-build CI.

## Repository Customizations

Expand Down
Loading
Loading