SCC26: Collapse only spatial loops in per-variable GPU kernels

[AppleChen17]

Summary

Eight GPU loops in m_rhs, m_riemann_state, and m_weno copy or difference every variable of a field using collapse=4 over the variable and three spatial dimensions. This PR collapses only the three spatial loops and executes the variable loop sequentially inside each GPU thread.

On H200, this reduces grind time by 5.4–10.3% across three example cases, with 8.0–10.3% improvements on one GPU. Results were measured with NVHPC 25.9 and 26.3 on one GPU, and NVHPC 26.3 on two GPUs. Nsight Compute shows 59–67% fewer executed instructions in the changed kernels, with essentially unchanged DRAM traffic.

Outputs are bit-identical to master in our comparisons, and the full GPU and CPU test suites pass.

This PR is from the NTHU team ( Team NTHU SCC ) participating in the SC26 Student Cluster Competition.

Changes

The modification applies to the following routines:

File	Routine	Modified Operation
m_rhs.fpp	s_compute_rhs	Conservative-variable copy
m_rhs.fpp	s_compute_advection_source_term	x/y/z flux differences
m_riemann_state.fpp	s_finalize_riemann_solver, s_finalize_riemann_solver_hatR	Directional flux copies
m_weno.fpp	s_pack_weno_input_arr	WENO input packing

Each loop follows the same transformation (example from m_weno.fpp):

Before:

$:GPU_PARALLEL_LOOP(collapse=4)
do i = 1, v_size
    do l = ...
        do k = ...
            do j = ...
                v_rs_weno(j, k, l, i) = v_vf(i)%sf(j, k, l)

After:

$:GPU_PARALLEL_LOOP(collapse=3)
do l = ...
    do k = ...
        do j = ...
            $:GPU_LOOP(parallelism='[seq]')
            do i = 1, v_size
                v_rs_weno(j, k, l, i) = v_vf(i)%sf(j, k, l)

The numerical expressions and spatial loop order are preserved. In the three flux-difference loops, the inverse grid spacing is also computed outside the variable loop, avoiding repeated evaluation for each variable.

Why this is faster

With collapse=4, each (variable, cell) pair is a separate GPU iteration, requiring its own collapsed-index decomposition and spatial offset calculations. With collapse=3, each thread processes all variables at one spatial cell, allowing spatial index calculations to be reused across variables.

Nsight Compute profiling of the changed kernels on H200 (3D_performance_test, NVHPC 26.3) shows:

Metric	Change
Executed instructions	−59% to −67%
DRAM bytes read + written	Unchanged (within 0.13%)
Individual kernel duration	−40% to −49%

Nsys profiling further shows that the improvement comes from the modified kernels:

• 3D_performance_test: total GPU kernel time decreased by 10.2%, with 99.6% of the saving attributable to the eight changed kernels.
• 2D_shockdroplet: total GPU kernel time decreased by 11.5%. The six applicable changed kernels each improved by 36–44%, while the other kernels collectively remained nearly unchanged.

Performance

Grind time (ns/gp/eq/rhs), master → this PR. Each entry is the mean of three interleaved runs per variant within the same allocation.

All runs use H200, double precision, and --case-optimization. Two-GPU runs use two MPI ranks on one node.

Example case	NVHPC 26.3, 1 GPU	NVHPC 25.9, 1 GPU	NVHPC 26.3, 2 GPUs
3D_performance_test (201³, WENO5, 30 steps)	0.41600 → 0.37876 (−8.95%)	0.41569 → 0.37808 (−9.05%)	0.25762 → 0.24042 (−6.68%)
3D_shockdroplet (201×101×101, WENO3, 1000 steps)	0.45487 → 0.41869 (−7.96%)	0.45143 → 0.41496 (−8.08%)	0.28420 → 0.26897 (−5.36%)
2D_shockdroplet (4800×1200, WENO3, 565 steps)	0.25092 → 0.22514 (−10.27%)	0.25056 → 0.22550 (−10.00%)	0.13561 → 0.12380 (−8.71%)

Maximum run-to-run spread was 1.24% (NVHPC 26.3, 1 GPU), 0.18% (NVHPC 25.9, 1 GPU), and 3.55% (NVHPC 26.3, 2 GPUs).

The cases are the examples/ cases unchanged except for a shorter t_step_stop (2D_shockdroplet is run with its --res 4 --mach 1.4 --cfl 0.2 option).

Testing

All comparisons use master at 47568a0 as the baseline.

• GPU full suite: NVHPC 26.3, OpenACC, H200 — 725 passed, 0 failed, 39 skipped.
• CPU full suite: NVHPC 26.3, Intel Xeon Platinum 8480+ — 722 passed, 0 failed, 39 skipped.
• Bit-identical outputs: Final restart files match master across the measured configurations, including case-optimized builds.
• No golden files changed.

CPU builds also use the reordered loops. Our shared-node CPU measurements were too noisy to establish a reliable performance difference; upstream CI benchmarks can provide additional coverage.

Contribution Policy

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

[github-actions]

Lines of Code

File	Lines	Diff
src/simulation/m_rhs.fpp	1972	+4
src/simulation/m_riemann_state.fpp	1188	+3
src/simulation/m_weno.fpp	1322	+1

Directory	Lines	Diff
simulation	28534	+8
total	47593	+8

[github-actions]

Claude Code Review

Head SHA: 21c80ba

Files changed:

• 3
• src/simulation/m_rhs.fpp
• src/simulation/m_riemann_state.fpp
• src/simulation/m_weno.fpp

Findings:
No issues found. Checked in step 4: src/simulation/m_rhs.fpp, src/simulation/m_riemann_state.fpp, src/simulation/m_weno.fpp (collapse=4 changed to collapse=3 with the variable-index loop made sequential via GPU_LOOP; bounds, loop-variable privacy, and index orders are unchanged).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.90%. Comparing base (47568a0) to head (21c80ba).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2087   +/-   ##
=======================================
  Coverage   61.90%   61.90%           
=======================================
  Files          86       86           
  Lines       22885    22885           
  Branches     3369     3369           
=======================================
  Hits        14166    14166           
  Misses       6220     6220           
  Partials     2499     2499           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.