build: add CUDA_PYTHON_TOOLCHAIN override for compiler/linker selection

[juenglin]

Description

Introduces a CUDA_PYTHON_TOOLCHAIN build-time override (mirroring CUDA_PYTHON_PARALLEL_LEVEL) that switches the C/C++ compiler and linker as a unit. Valid values: gnu (default on Linux), llvm (clang + lld) on Linux, msvc (default on Windows). The defaults reproduce the previous build behavior exactly.

Motivation: we want to be able to experiment with alternative toolchains without committing to them yet. Based on experiments, switching to llvm reduces build time.

Note: there is no support guarantee for CUDA_PYTHON_TOOLCHAIN. It may be removed or changed without notice.

Naming

gnu / llvm / msvc name toolchain families (compiler + linker + binutils), not just a compiler, which matches switching compiler and linker together. llvm (clang + lld) is preferred over clang because the linker also switches to lld; clang would under-describe that. Values are case-insensitive; an invalid value or platform mismatch raises a clear error listing the platform's valid values.

Resolution rules

Only CUDA_PYTHON_TOOLCHAIN governs toolchain selection; CC/CXX are never read.

• CUDA_PYTHON_TOOLCHAIN unset — the platform default (gnu on Linux, msvc on Windows). The backend does not touch CC/CXX, so an externally-set compiler (e.g. CC="sccache cc" in CI) keeps working, and the default flag set is applied.
• CUDA_PYTHON_TOOLCHAIN=gnu — selects gcc/g++ explicitly. CC/CXX/LDSHARED are overridden to gcc/g++ so the gnu-specific flags (including -fpermissive, -fno-var-tracking-assignments in cuda.bindings) reach the actual GNU compiler even on systems where cc/c++ alias to clang.
• CUDA_PYTHON_TOOLCHAIN=llvm — selects clang/clang++ and lld. CC/CXX/LDSHARED are overridden to clang/clang++; -fuse-ld=lld is added to link args. The gcc-only flags (e.g. -fpermissive, -fno-var-tracking-assignments) are dropped since clang rejects them.

Preflight

_check_toolchain_available probes the toolchain's tools on PATH and fails fast with a helpful message (tool name, install hint, and how to fall back). No-op for the implicit platform default. For llvm: checks clang, clang++, and ld.lld.

Stale-binary protection

A stale .so from a previous toolchain looks perfectly fresh to setuptools' mtime check. Two stamps prevent this:

• cuda_core: generalizes the existing .build-cuda-major stamp into .build-config covering (cuda_major, toolchain, debug, coverage). build_ext is forced and the cythonize build_dir is keyed by the full config tag when any of these change.
• cuda_bindings: adds a new minimal .build-toolchain stamp. build_ext is forced when the toolchain changes (the cythonize step is toolchain-independent, so its build_dir is left flat).

Changes

• cuda_bindings/build_hooks.py, cuda_core/build_hooks.py: add the toolchain helpers. The genuinely-shared logic (constants, name resolution/validation, env application, preflight) is in a single block delimited by # --- begin/end shared toolchain helpers --- markers, duplicated verbatim with a "keep in sync" comment — mirroring the existing _import_get_cuda_path_or_home precedent. PEP 517 build isolation forbids a shared module (the sibling package is not installed in the isolated build env), so the block is duplicated rather than imported. The per-package _resolve_toolchain() is a thin wrapper that assembles only its own package-specific flags (cuda.bindings: c++14, -fpermissive, -O3; cuda.core: c++17, -O2).
• cuda_bindings/setup.py: dropped the now-redundant _is_clang strip (_resolve_toolchain selects the correct flag set upfront); added force_build_ext check and record_build_toolchain() call.
• cuda_core/setup.py: updated to call record_build_config() (was record_build_major()).
• toolshed/check_build_hooks_sync.py + .pre-commit-config.yaml: new pre-commit hook (check-build-hooks-sync) that verifies the shared block is byte-identical across both files at commit time, replacing the test that was fragile outside a full monorepo checkout.
• Tests: toolchain resolution, preflight, stamp/force, and keyed-build-dir tests in both test_build_hooks.py files.
• Docs: CUDA_PYTHON_TOOLCHAIN intentionally not documented in environment_variables.rst (no support guarantee; keeps the surface from becoming relied-upon automation).

Build paths verified

All current build entry points keep working: cibuildwheel (build-wheel.yml, Linux CC="sccache cc", Windows MSVC), python -m build/pip wheel (test-sdist-linux.yml, test-sdist-windows.yml), coverage.yml, pixi pixi-build-python, the Cython-test build_tests.py drivers, and local rebuild-cuda-python. The default path (unset CUDA_PYTHON_TOOLCHAIN) is unchanged.

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[mdboom]

No real objection to this as-is, but it would be nice to reduce the duplication between the two build_hooks.py scripts somehow if possible.

I think in the long run, we will want to migrate to a more robust build backend, the top contender for which is probably scikit-build-core (based on CMake). In that environment, you wouldn't hardcode the flags to run explicitly, but have the configure step figure out which ones are available etc. My worry is that if we implement this now, we would have to duplicate this somehow inside of a better build system for backward compatibility and it may not make sense there. Would it be better to migrate now?

[sylvesterkaczmarek]

Removing the _is_clang handling breaks the externally supplied clang path that this PR says should keep working. When CUDA_PYTHON_TOOLCHAIN is unset, _resolve_toolchain() selects gnu and keeps the GCC-only -fno-var-tracking-assignments; because the default path intentionally does not override CC/CXX, a caller with CC=clang / CXX=clang++ now gets that unsupported flag. Previously build_ext detected clang and removed it. Please preserve compiler-based flag filtering on the default path (or resolve flags from the actual external compiler) and add a regression with externally supplied clang and no toolchain override.

[juenglin]

No real objection to this as-is, but it would be nice to reduce the duplication between the two build_hooks.py scripts somehow if possible.

...

I tried extracting the shared toolchain helpers into a small distributable package (cuda-python-build-helpers, parallel to cuda-python-test-helpers), and I had it working locally. On reflection I kept the duplication, for two reasons:

1. PEP 517 build isolation. A build backend can only import what's in build-system.requires. cuda-python-build-helpers would have to be installable in the isolated build env for every wheel build — meaning publishing it to PyPI (or wiring a local wheelhouse into each cibuildwheel container), the same release burden cuda-pathfinder already carries. That's heavy for ~60 lines of code.
2. Test-environment coupling. Once build_hooks.py imports the package at module top, the build-hooks tests (which load build_hooks from source via importlib) need it installed in the test env too — so it has to be added to the test dependency groups, and pixi.lock regenerated. The self-contained build_hooks.py has none of that.

FWIW there is precedent: _import_get_cuda_path_or_home is already duplicated verbatim between the two build_hooks.py with a "keep in sync" comment.

... scikit-build-core ...
My worry is that if we implement this now, we would have to duplicate this somehow inside of a better build system for backward compatibility and it may not make sense there.

Hm, I don't think we promise backward compatibility for how the packages are built. Am I missing your point?

[juenglin]

/ok to test f8245a5

[juenglin]

/ok to test 8ce0a8b

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-2903/
https://nvidia.github.io/cuda-python/pr-preview/pr-2903/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-2903/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-2903/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[leofang]

I am very concerned in this. Whose problem are we addressing here? We don't have any CI infra to validate this (and full-fledged source build support is not something that we commit to -- 99.9% of our users should go use wheels or conda packages, not building from source).

[leofang]

Clang is not something we officially commit to. The original purpose was to be able to do a quick performance study (#1649). If anything, we should remove _is_clang from our build system.

[leofang]

@sylvesterkaczmarek Are you AI agent? Are you actually using Clang to build cuda-bindings (and why?), or did you just leave random drive-by comments?

[mdboom]

... scikit-build-core ...
My worry is that if we implement this now, we would have to duplicate this somehow inside of a better build system for backward compatibility and it may not make sense there.

Hm, I don't think we promise backward compatibility for how the packages are built. Am I missing your point?

My point was that this environment variable becomes part of the interface that others will use to build our package in their automation, and we might have to continue to support that surface indefinitely.

[juenglin]

My point was that this environment variable becomes part of the interface that others will use to build our package in their automation, and we might have to continue to support that surface indefinitely.

I see. I believe we both agree that this is not something we promise nor want to promise. We could be more explicit about that in environment_variables.rst.

[juenglin]

/ok to test 0b2ce38

[rwgk]

The synchronization test fails outside a monorepo checkout.

Maybe that tips the scale towards using pre-commit?

---

codex gpt-5.6-sol ultra findings findings

• P1 — Toolchain changes can silently reuse binaries from the previous toolchain. Toolchain selection only changes environment variables and extension flags in cuda.bindings and cuda.core. Setuptools freshness checks do not include those flags; core only forces rebuilds when the CUDA major changes, while bindings never forces them. A GNU build followed by LLVM in the same checkout can therefore package the still-fresh GNU .so. Include the effective toolchain/configuration in the build stamp or output paths and add a GNU→LLVM→GNU round-trip test.

• P1 — CC=clang does not select Clang for these C++ extensions. The resolver falls back from CXX to CC and declares LLVM at cuda_bindings/build_hooks.py#L143-L151, but inferred mode leaves CXX unset. Setuptools ≥80 compiles and links .cpp sources using CXX, so the actual compiler remains Python’s sysconfig C++ compiler—typically g++—while LLVM/lld flags are applied. Infer from the configured C++ compiler or set CXX consistently; the current helper-only test does not catch this integration failure.

• P1 — Inferred Clang unexpectedly makes lld mandatory. The LLVM branch always adds -fuse-ld=lld, while the shared preflight requires bare clang, clang++, and ld.lld names on PATH. Consequently, CXX=clang++ builds that previously used the system linker now fail without separately installed lld; absolute, versioned, or wrapped compiler paths can also fail despite being valid. Reserve clang+lld switching for the explicit override and preserve the linker for inferred external compilers.

• P2 — Explicit gnu does not guarantee a GNU compiler. The mapping uses generic cc and c++, then applies GCC-only flags. Where those aliases resolve to Clang, CUDA_PYTHON_TOOLCHAIN=gnu invokes Clang and fails. Use gcc/g++ or resolve and verify the aliases.

• P2 — The synchronization test fails outside a monorepo checkout. test_shared_toolchain_block_is_in_sync unconditionally reads sibling cuda_bindings. That sibling is absent in a standalone cuda-core sdist or sparse checkout, so the documented standalone test command fails. Move this invariant to a repository-level test or skip it when sibling source is unavailable.

• P3 — The new synchronization test lacks a required authorship marker. The same test has none of agent_authored, human_reviewed, or human_authored, contrary to the repository’s test provenance policy.

[juenglin]

/ok to test adebd02

[juenglin]

/ok to test b281903

[juenglin]

/ok to test baeff17

[rwgk]

codex gpt-5.6-sol ultra review findings and suggested fixes

Reviewed PR head: baeff1783669135a754c75491896058ad1392790

The suggested fixes are organized as three cherry-pickable commits on rwgk/review/toolchain-override-backend2. Together they are tree-identical to the version exercised by the upstream CI run.

[P1] Scope build stamps to the extension ABI

The new configuration stamps were single files in each checkout (.build-toolchain and .build-config). Multiple Python interpreters can reuse that checkout while producing different extension ABIs. One interpreter could therefore overwrite the checkout-global stamp for another interpreter.

For example, after Python A builds configuration X, Python B builds configuration Y, and Python A requests Y, Python A sees B's Y stamp. Its existing X extension can look up to date by timestamp, so build_ext is not forced even though Python A never built Y. The same issue applies to architecture and free-threaded versus GIL-enabled builds whenever the checkout is shared.

Suggested fix: include Python's EXT_SUFFIX in each stamp filename. EXT_SUFFIX carries the interpreter ABI and platform extension identity, including distinctions such as CPython version, Windows architecture, and free-threaded builds. Keep the existing configuration value inside each ABI-specific stamp.

• Fix commit: c5be9b69d6c5aa619e07f334e8e5bd7ba4338bb3 (build: scope configuration stamps to extension ABI)
• Bindings implementation: cuda_bindings/build_hooks.py
• Core implementation: cuda_core/build_hooks.py
• Regression coverage verifies that two mocked extension ABIs resolve to different stamp paths in both packages.

[P1] Stamp the exact core configuration that actually completed

cuda_core checked the requested debug mode in the PEP 517 backend, but wrote the completed-build stamp later from setuptools' build_ext.debug. Those values are not guaranteed to match. In particular, a backend debug build could be recorded as optimized. A later optimized request could then accept the debug artifact as current instead of forcing a rebuild.

Re-deriving the other stamp inputs at write time also made the stamp describe current ambient state rather than the exact configuration prepared earlier in the build.

Suggested fix: make _build_cuda_core() return the exact configuration key it checked, carry that immutable key through the backend, and record it only after the wheel build succeeds. For editable builds, write the stamp only after the .pth patch succeeds as well. Remove the inaccurate setup.py-side write.

• Fix commit: d40f92a00f19cc684027b48f13d35370003988a3 (build(core): stamp the exact successful configuration)
• Exact-key recording: cuda_core/build_hooks.py
• Wheel and editable success ordering: cuda_core/build_hooks.py
• Regression coverage checks the precise key and ordering for successful wheels and editables, plus the absence of a stamp after wheel or editable-patch failure.

[P2] Isolate toolchain environment mutations in build-hook tests

The build-hook tests modify CUDA_PYTHON_TOOLCHAIN, CC, CXX, and LDSHARED. Several paths assign directly to os.environ, so ordinary monkeypatch cleanup does not necessarily know the original state of every variable. Values can leak between tests or escape the module, making results order-dependent and capable of contaminating later build tests.

Suggested fix: add an autouse fixture in each build-hook test module that snapshots these four variables, clears them before each test, and restores the exact original state afterward, including the distinction between an unset variable and a set value.

• Fix commit: 60b55cef872b4cc21045128613debc114e80a6c6 (tests: isolate toolchain environment changes)
• Bindings fixture: cuda_bindings/tests/test_build_hooks.py
• Core fixture: cuda_core/tests/test_build_hooks.py

Validation

• cuda_bindings/tests/test_build_hooks.py: 15 passed.
• cuda_core/tests/test_build_hooks.py: 49 passed.
• All pre-commit hooks applicable to the five changed files passed.
• The clean three-commit branch has tree b416652ea2dc6f81dcd5696d625113ec514999e7, identical to the CI-tested branch.
• In upstream ci.yml run 35691868451, all 104 source, build, and test jobs passed; one matrix job was skipped.
• The run's only substantive failure was the pre-existing docs job: manual workflow_dispatch runs have no pull-request event, while build-docs.yml unconditionally asks get_pr_number for a PR on non-release branches. The final Check job status aggregator consequently failed because it requires docs success. Rerunning without creating a PR would deterministically produce the same result; .github is unchanged by these fixes.

[juenglin]

/ok to test f0a1b78

[rwgk]

Approving, based on a codex gpt-5.6-sol ultra re-review (no findings anymore).