[ericspod]

Fixes # .

Description

This adds a testing runner for the KCL GPU runner available to MONAI. The goal is to develop a GPU testing process, either as a per-PR test or nightly GPU tests.

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[vikasreddy11]

Hi @ericspod , I'm interested in helping with the GPU test runner. I noticed the PR is still in progress. Is there a specific area where an independent investigation or contribution would be useful?

[ericspod]

Thanks @vikasreddy11 do you have access to a GPU? I could use some help testing the GPU resampling accuracy with the compiled extensions. I'm seeing affine tests failing like this

Details

FAIL: test_affine_003 (tests.transforms.test_affine.TestAffine.test_affine_003)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/monai/tests/transforms/test_affine.py", line 197, in test_affine
    test_resampler_lazy(
  File "/opt/monai/tests/lazy_transforms_utils.py", line 77, in test_resampler_lazy
    assert_allclose(lazy_out, non_lazy_out, rtol=rtol, atol=atol)
  File "/opt/monai/tests/test_utils.py", line 167, in assert_allclose
    np.testing.assert_allclose(actual, desired, *args, **kwargs)
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 1688, in assert_allclose
    assert_array_compare(compare, actual, desired, err_msg=str(err_msg),
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/numpy/testing/_private/utils.py", line 889, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=0.001, atol=0.001

Mismatched elements: 15 / 16 (93.8%)
Max absolute difference among violations: 1.3333333
Max relative difference among violations: 1.7777779
 ACTUAL: array([[[0.      , 0.231481, 0.601852, 0.694444],
        [0.462963, 0.833333, 1.277778, 1.296296],
        [1.203704, 1.722222, 2.166667, 2.037037],
        [1.388889, 1.898148, 2.268518, 2.083333]]], dtype=float32)
 DESIRED: array([[[0.      , 0.083333, 0.416667, 0.25    ],
        [0.166667, 0.5     , 1.166667, 0.666667],
        [0.833333, 1.833333, 2.5     , 1.333333],
        [0.5     , 1.083333, 1.416667, 0.75    ]]], dtype=float32)

This occurs in tests.transforms.test_affine and tests.transforms.test_affined. If you can try testing these files in the Docker image on in a similar environment with the CUDA extensions built and get to the bottom of the issue that would go a long way for the testing process. Be sure to have BUILD_MONAI=1 exported even when you run the tests to be sure compiled code is used. I encountered some other issues here that's related to the testing host I think that only I can sort out.

[vikasreddy11]

Thanks @ericspod, I tested on a GPU outside Docker (RTX 3050 Laptop, WSL2, Python 3.12.15, PyTorch 2.14.1+cu132, dev at cb35d45).

I can reproduce it, and it depends on the compiled extension:

• Default (USE_COMPILED=False): test_affine.py gives 125 passed, 0 failed.
• BUILD_MONAI=1 (USE_COMPILED=True): 72 failed, including test_affine_003. test_affine_116 fails alone and via unittest with the same mismatch numbers as yours.
• The failure is at the final pixel comparison in test_resampler_lazy (lazy_transforms_utils.py:77), after the pending-affine and shape assertions have passed. So the lazy metadata matches eager, and only the values resampled by apply_pending differ.

Not yet checked: test_affined, and PyTorch 2.9.x (mine is newer than CI's range). I'll report back.