ci: run cuda.core tests in heap-bounded batches, merge JUnit XML

[juenglin]

Description

Related to the "cuda.core test time" investigation discussed in #3041 and #3045 (no dedicated issue filed yet).

The init_cuda teardown's per-test gc.collect() (added by #2484) gets superlinearly slower as a pytest session proceeds. Two things accumulate for the life of a single session and are walked by every gc.collect():

1. pytest-repeat's --count=N (used on nightly runs) expands the collected item list N-fold up front.
2. Executed-item bookkeeping — TestReport objects, and (since --junitxml entered the CI command) the JUnit XML node per item — grows as the run proceeds.

Measured effect: nightly's 5 repeats cost ~10x one repeat, not ~5x. #3041 addressed term 1 with a single gc.freeze() right after collection. This PR addresses term 2 (and keeps term 1 bounded too) by capping how much of the suite any one pytest process ever sees.

ci/tools/pytest_batched.py collects the full, already---count-expanded item list once via pytest --collect-only, then runs it across several pytest subprocesses of at most --batch-size items each (default 1000, overridable via --batch-size or $PYTEST_BATCH_SIZE). Each subprocess starts with an empty heap, so neither accumulation term grows past one batch, regardless of how large --count or the suite gets.

The per-batch JUnit XML files are merged back into a single file at the original --junitxml path, so the script is a drop-in replacement for the plain pytest invocation it replaces in ci/tools/run-tests: same CLI, same final junit-core.xml (verified byte-for-byte equal tests/failures/testcase counts against an unbatched run of ci/tools/tests itself), same aggregate exit code. --runner-prefix forwards SANITIZER_CMD to each batch's pytest invocation.

Scoped to the "Run cuda.core tests" step only (ci/tools/run-tests), since that is the step the investigation identified as the bottleneck (cuda.bindings stayed flat). The bindings/pathfinder/cython invocations are untouched.

Two non-obvious correctness requirements this script depends on, each covered by a dedicated regression test:

• pytest --collect-only -q's flat, one-ID-per-line format needs a verbosity this script controls exactly. pytest sums -v/-q occurrences rather than letting the last one win, so a caller-supplied -v (CI always passes one) would otherwise partially cancel the -q this script adds internally, silently switching to the indented tree format and collecting zero node IDs.
• PYTEST_ADDOPTS (e.g. --count=N) must stay active for the batch runs, not just for collection. A repeated item's node ID (test_foo[1-3]) only exists once pytest-repeat's pytest_generate_tests hook re-parametrizes the module, which requires --count to still be in effect — even though pytest still only runs the items named explicitly on the command line, so this does not re-execute anything beyond what each batch names.

Verification (CPU-only, no GPU needed for this script itself):

• ci/tools/tests/test_pytest_batched.py: 18 unit tests for the pure helpers (node-ID parsing, batching, path stripping, --junitxml extraction, JUnit merging) and 7 end-to-end subprocess tests (multi-batch execution, failure propagation, empty-collection handling, --count interaction, the -v/-q regression).
• python -m pytest -v --noconftest ci/tools/tests (the suite CI runs) passes: 136 passed, 36 subtests passed — unchanged from before this PR.
• Ran pytest_batched.py against ci/tools/tests itself with the exact flag set ci/tools/run-tests uses (-rsxXfE -v --randomly-dont-reorganize --exclude-warning-annotations --junitxml ...) and confirmed the merged junit-*.xml's tests/failures/testcase counts exactly match an unbatched run of the same suite (both report tests="172", testcase count 136 — the gap is pytest's own pre-existing pytest-subtests counting quirk, reproduced identically, not something this PR introduces).

This has not been verified against the real cuda_core suite under CUDA/GPU CI; that needs a CI run on this PR.

Checklist

• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[juenglin]

Closing: #3053 removed the superlinear repeat scaling this targeted. Real nightly data (linux-64/py3.12/l4, "Run cuda.core tests"):

	nruns=1	nruns=5	ratio
pre-#3053	197s	1689s	8.6x
post-#3053	164s	702s	4.28x

Ratio now under 5x means no superlinear residual left to recover. A synthetic benchmark confirms batching is net-negative once gc.collect() frequency drops this low (subprocess startup overhead dominates). No further action needed here.