Skip to content

Guard against division by zero in VorticityAuxVars.h - #562

Open
andrewdnolan wants to merge 1 commit into
E3SM-Project:developfrom
andrewdnolan:omega/guard-division-by-zero
Open

andrewdnolan wants to merge 1 commit into
E3SM-Project:developfrom
andrewdnolan:omega/guard-division-by-zero

Conversation

@andrewdnolan

Copy link
Copy Markdown

Adds a check to see if the PseudoThickVertex is zero; if true set NormRelVortVertex and NormPlanetVortVertex to zero (avoiding the previous division by zero).

The issue was exposed using SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF, with the gnu compiler, on chrysalis. The _D modifier in the test name resulted in a debug build, which adds the -ffpe-trap=invalid,zero,overflow flags.

This has not been caught previously, because a standalone Debug build of Omega does not use the -ffpe-trap=invalid,zero,overflow flags.

Closes #556.

Checklist

  • Documentation:

  • Linting

  • Building

    • CMake build does not produce any new warnings from changes in this PR
  • Testing

    aurora, oneapi-ifx, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    chrysalis, oneapi-ifx, openmpi

    • CTests Pass
    • Polaris omega_pr Pass

    frontier, craygnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    frontier, craygnu-mphipcc, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    pm-cpu, gnu, mpich

    • CTests Pass
    • Polaris omega_pr Pass

    pm-gpu, gnugpu, mpich

    • CTests Pass
    • Polaris omega_pr Pass
  • Provide relevant details in a comment to the PR titled Testing with the following:

    • Which machines CTest unit tests
      have been run on and indicate that are all passing.
    • The Polaris omega_pr test suite
      has passed, using the Polaris e3sm_submodules/Omega baseline
    • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
    • Indicate "All tests passed" or document failing tests
    • Document testing used to verify the changes including any tests that are added/modified/impacted.
  • Performance related PRs: Please include a relevant PACE experiment link documenting performance before and after.

  • New tests:

    • CTest unit tests for new features have been added per the approved design.
    • Polaris tests for new features have been added per the approved design (and included in a test suite)
  • Stealth Features

    • If any stealth features are included in the PR, please confirm that they have been documented.

@andrewdnolan andrewdnolan added the bug Something isn't working label Sep 17, 2026
@andrewdnolan

Copy link
Copy Markdown
Author

Testing:

A single node, non-threaded, smoke test on chrysalis using debug build:

20260917_140012_rqz30d: 1 test
  SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu (Overall: PASS) details:
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu CREATE_NEWCASE
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu XML
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu SETUP
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu SHAREDLIB_BUILD time=276
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu MODEL_BUILD time=567
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu SUBMIT
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu RUN time=1024
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu MEMLEAK
    PASS SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu SHORT_TERM_ARCHIVER
Prior to this PR, that same test failed with:

29: 
29: Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.
29: 
29: Backtrace for this error:
...
17: #0  0x155549b1a62f in ???
22: #0  0x155549b1a62f in ???
42: #0  0x155549b1a62f in ???
52: #0  0x155549b1a62f in ???
25: #0  0x155549b1a62f in ???
30: #0  0x155549b1a62f in ???
28: #0  0x155549b1a62f in ???
41: #0  0x155549b1a62f in ???
17: #1  0xa9e6c1 in _ZZNK5OMEGA16VorticityAuxVars19computeVarsOnVertexERKN6Kokkos4Impl20HostThreadTeamMemberINS1_6SerialEEEiRKNS1_4ViewIPPdJNS1_11LayoutRightENS1_9HostSpaceEEEESF_ENKUliE2_clEi
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/auxiliaryVars/VorticityAuxVars.h:70
17: #2  0xaa28c0 in _ZN6Kokkos12parallel_forIiZNK5OMEGA16VorticityAuxVars19computeVarsOnVertexERKNS_4Impl20HostThreadTeamMemberINS_6SerialEEEiRKNS_4ViewIPPdJNS_11LayoutRightENS_9HostSpaceEEEESG_EUliE2_S6_EEvRKNS3_31TeamThreadRangeBoundariesStructIT_T1_EERKT0_PPKNSt9enable_ifIXsrNS3_26is_host_thread_team_memberISK_EE5valueEvE4typeE
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/externals/ekat/extern/kokkos/core/src/impl/Kokkos_HostThreadTeam.hpp:788
17: #3  0xaa1352 in _ZN5OMEGA16parallelForInnerIZNKS_16VorticityAuxVars19computeVarsOnVertexERKN6Kokkos4Impl20HostThreadTeamMemberINS2_6SerialEEEiRKNS2_4ViewIPPdJNS2_11LayoutRightENS2_9HostSpaceEEEESG_EUliE2_EEvS8_NS_5RangeEOT_
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/infra/OmegaKokkosHiPar.h:253
17: #4  0xa9f188 in _ZNK5OMEGA16VorticityAuxVars19computeVarsOnVertexERKN6Kokkos4Impl20HostThreadTeamMemberINS1_6SerialEEEiRKNS1_4ViewIPPdJNS1_11LayoutRightENS1_9HostSpaceEEEESF_
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/auxiliaryVars/VorticityAuxVars.h:68
17: #5  0xa89712 in operator()
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/AuxiliaryState.cpp:217
17: #6  0xa8f121 in operator()<>
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/infra/OmegaKokkos.h:241
17: #7  0xa8f121 in operator()
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/infra/OmegaKokkosHiPar.h:142
17: #8  0xa971a7 in exec<void>
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/externals/ekat/extern/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Team.hpp:229
17: #9  0xa94870 in execute
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/externals/ekat/extern/kokkos/core/src/Serial/Kokkos_Serial_Parallel_Team.hpp:265
17: #10  0xa918af in parallel_for<Kokkos::TeamPolicy<Kokkos::Serial>, OMEGA::parallelForOuter<1, OMEGA::AuxiliaryState::computeMomAux(const OMEGA::OceanState*, const Array3DReal&, int, int, OMEGA::TimeInterval) const::<lambda(int, const TeamMember&)> >(const string&, const OMEGA::LaunchConfig<1>&, OMEGA::AuxiliaryState::computeMomAux(const OMEGA::OceanState*, const Array3DRea
l&, int, int, OMEGA::TimeInterval) const::<lambda(int, const TeamMember&)>&&)::<lambda(const TeamMember&)> >
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/externals/ekat/extern/kokkos/core/src/Kokkos_Parallel.hpp:146
17: #11  0xa8f307 in parallelForOuter<1, OMEGA::AuxiliaryState::computeMomAux(const OMEGA::OceanState*, const Array3DReal&, int, int, OMEGA::TimeInterval) const::<lambda(int, const TeamMember&)> >
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/infra/OmegaKokkosHiPar.h:139
17: #12  0xa89f3f in _ZNK5OMEGA14AuxiliaryState13computeMomAuxEPKNS_10OceanStateERKN6Kokkos4ViewIPPPdJNS4_11LayoutRightENS4_9HostSpaceEEEEiiNS_12TimeIntervalE
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/AuxiliaryState.cpp:212
17: #13  0xa8bd74 in _ZNK5OMEGA14AuxiliaryState10computeAllEPKNS_10OceanStateERKN6Kokkos4ViewIPPPdJNS4_11LayoutRightENS4_9HostSpaceEEEEiiNS_12TimeIntervalE
17:     at /gpfs/fs1/home/ac.anolan/repos/Omega/develop/components/omega/src/ocn/AuxiliaryState.cpp:382

Here's a full log file, if you're interested:

/lcrc/group/e3sm/ac.anolan/scratch/chrys/SMS_Vmct_D_P64x1.T62_oQU240.COMEGA-IAF.chrysalis_gnu.20260917_112659_wzv07b/run/e3sm.log.1289521.260917-114035

@andrewdnolan

Copy link
Copy Markdown
Author

Testing

CTest unit tests

  • Machine: chrysalis
  • Compiler: gnu
  • Build type: Debug
  • Result: All tests passed
    • Log: /lcrc/group/e3sm/ac.anolan/Omega_PR562/build/feature/ctest_2026-09-17_16:08.log

Tried running the omega_pr suite, but hit a filesystem error because chrysalis has run out of inodes. I can run the polaris tests once that's resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

debug test fails using gnu compiler on chrysalis

1 participant