Change MPI3 shared memory in Quest so it uses Umpire. #1652

BradWhitlock · 2025-09-06T01:09:50Z

This PR resolves #1058 by migrating direct MPI3 calls to suitable Umpire shared memory replacements. These changes are all in quest and are part of the signed_distance functionality.

Fixed race conditions in quest_signed_distance_interface test that happened with running with multiple MPI ranks
Removed a constraint in the CMake build logic that limited that test to 1 rank, now it should work on 4+ and multiple nodes
Verified that these shared memory changes work on multiple compute nodes. The file is read on 1 rank, shared memory is allocated in 1 rank per node and broadcast to those buffers. Finally, all ranks use the shared memory to create a mint mesh.
Updated release notes.
Submitted a PR to Umpire to fix a memory leak. (merged)
Changed Axom package so it builds Umpire with MPI3 shared memory support when +mpi is given (tpls and host-configs are not updated)

…ionality so it uses Umpire shared memory allocators.

…nce_interface when run on multiple ranks. Remove build logic to limit the test to 1 rank.

…mory via Umpire, as far as I can tell.

…3 shared memory support.

src/axom/quest/interface/internal/QuestHelpers.cpp

…ed with MPI.

kennyweiss

Thanks @BradWhitlock

I had some questions/suggestions about config variables that I'd like to work through before merging, and also possibly about promoting the shared resource allocator logic to core instead of quest, but otherwise this looks great!

Thanks for fixing the umpire memory leak and the issue w/ scaling to more than one rank!

kennyweiss · 2025-09-09T00:42:16Z

scripts/spack/packages/axom/package.py

        depends_on("[email protected]", when="@0.6.0")
        depends_on("umpire@5:5.0.1", when="@:0.5.0")
        depends_on("umpire+openmp", when="+openmp")
+        depends_on("umpire+mpi3_shmem", when="+mpi")


Is mpi3_shmem always available when mpi is available?

Tag: @white238 @davidbeckingsale

I think its something like this:

depends_on("umpire+mpi3_shmem", when="+mpi ^mpi@3:")

@becker33 ?

kennyweiss · 2025-09-09T00:43:04Z

src/axom/quest/tests/CMakeLists.txt

-    if (${test_name} STREQUAL "quest_signed_distance_interface")
-        set(_numMPITasks 1)
-    endif()


src/axom/quest/interface/signed_distance.cpp

RELEASE-NOTES.md

src/axom/quest/interface/internal/QuestHelpers.cpp

kennyweiss · 2025-09-09T01:02:09Z

src/axom/quest/interface/internal/QuestHelpers.cpp

+  MPI_Comm intra_node_comm = MPI_COMM_NULL;
  MPI_Comm inter_node_comm = MPI_COMM_NULL;
  create_communicators(global_comm,
                       intra_node_comm,


Do we still need to create extra communicators w/ the new allocator? Or (hopefully) does the new Umpire shared memory allocator allow us to simplify this?

src/axom/quest/interface/internal/QuestHelpers.cpp

src/axom/quest/interface/signed_distance.cpp

…m:LLNL/axom into feature/whitlock/use_umpire_shared_memory

kennyweiss

Thanks for the updates @BradWhitlock!

My previous reservations related to the build system flags. I think the new #defines are a lot cleaner.

It'd be nice to centralize the shared memory allocation logic to memory_management.*pp in a follow-up, but I don't think it's required for this PR.

kennyweiss · 2025-09-09T20:03:28Z

src/cmake/AxomOptions.cmake

-cmake_dependent_option(AXOM_ENABLE_MPI3 "Enables use of MPI-3 features" OFF "ENABLE_MPI" OFF)
-mark_as_advanced(AXOM_ENABLE_MPI3)
-


kennyweiss · 2025-09-09T20:05:39Z

src/cmake/thirdparty/SetupAxomThirdParty.cmake

+    # Check whether the Umpire defines symbols for shared memory
+    blt_check_code_compiles(CODE_COMPILES UMPIRE_SHARED_MEMORY
+                            VERBOSE_OUTPUT OFF
+                            DEPENDS_ON umpire
+                            SOURCE_STRING "
+    #include <umpire/config.hpp>
+    #if defined(UMPIRE_ENABLE_IPC_SHARED_MEMORY) || defined(UMPIRE_ENABLE_MPI3_SHARED_MEMORY)
+    int main() { return 0; }
+    #else
+    #error Macros not defined
+    #endif
+    ")
+    if (AXOM_ENABLE_MPI AND UMPIRE_SHARED_MEMORY)
+        set(AXOM_USE_UMPIRE_SHARED_MEMORY TRUE)
+    else()
+        set(AXOM_USE_UMPIRE_SHARED_MEMORY FALSE)
+    endif()
+    message(STATUS "Umpire supports shared memory: ${AXOM_USE_UMPIRE_SHARED_MEMORY}")


👍 -- interesting solution!
I thought it might involve checking properties of the umpire target.

kennyweiss

Thanks @BradWhitlock -- even better!

kennyweiss · 2025-09-09T23:17:03Z

src/axom/core/tests/core_shared_memory.hpp

kennyweiss · 2025-09-09T23:18:28Z

src/axom/core/memory_management.cpp

+{
+  int allocator_id = INVALID_ALLOCATOR_ID;
+#if defined(AXOM_USE_UMPIRE_SHARED_MEMORY)
+  const std::string allocatorName("axom_named_allocator");


Would it make sense to call this "axom_shared_allocator" ?

or, looking a few lines down, should it be "axom_node_allocator"?

In the Umpire examples, they first make a "node allocator" but then wrap it as a named allocator. From what I gather from their docs, this is to make the allocator compatible with their IPC shared memory logic. I renamed it to "axom_shared_allocator".

gunney1 · 2025-10-01T18:44:04Z

RELEASE-NOTES.md

Nice functionality @BradWhitlock !

gunney1

Thank you @BradWhitlock

kab163 · 2025-10-07T16:28:28Z

I just want to add a note of caution. Right now, it can be confusing if Umpire's IPC Shared Memory and/or Umpire's MPI3 Shared Memory are turned on since they can be both supported. If IPC is explicitly enabled but MPI3 is not, then MPI3 is turned off and IPC is the default shared memory resource. If MPI3 is explicitly enabled but IPC is not, then MPI3 is the default shared memory resource, but IPC is still turned on (will need to explicitly turn off if you don't want it). If IPC is explicitly enabled and MPI3 is explicitly enabled, then MPI3 is the default shared memory resource. (This all assumes MPI itself is enabled.)

The "default shared memory resource" is what is used when you only specify "SHARED".

To avoid confusion, I would highly recommend using auto traits = umpire::get_default_resource_traits("SHARED::MPI3"); and then auto mpi3_shm_allocator = rm.makeResource("SHARED::MPI3::axom_alloc", traits); (keeping mpi3 in the name) or instead auto traits = umpire::get_default_resource_traits("SHARED::POSIX"); and ...("SHARED::POSIX::axom_alloc", traits); keeping ipc or posix in the name... And/or, you could use a cmake message to tell you what the value of UMPIRE_DEFAULT_SHARED_MEMORY_RESOURCE ... couldn't hurt.

kab163 · 2025-10-07T16:41:32Z

See https://github.com/LLNL/Umpire/blob/ab81a4f22bc2967ee9dd3e2716cafa3f1587e7b9/CMakeLists.txt#L117-L130

kab163 · 2025-10-07T20:11:44Z

Specifically, I discovered today that our recipe_naming_shim example is broken. When I enable IPC and MPI3 shared memory, it keeps trying to use MPI3 even if I do auto traits = umpire::get_default_resource_traits("SHARED::POSIX");. The key is to also do auto node_allocator{rm.makeResource("SHARED::POSIX::node_allocator", traits)}; with emphasis on the SHARED::POSIX prefix.

So again, I encourage caution here and to be as explicit as possible.

…d shared memory allocator. Add cmake status message for value of UMPIRE_DEFAULT_SHARED_MEMORY_RESOURCE.

…m:LLNL/axom into feature/whitlock/use_umpire_shared_memory

BradWhitlock added 4 commits September 4, 2025 18:55

Initial port of read_stl_mesh_shared and quest::signed_distance funct…

b8bd28c

…ionality so it uses Umpire shared memory allocators.

Fix some race conditions that prevented passage of quest_signed_dista…

5fb886e

…nce_interface when run on multiple ranks. Remove build logic to limit the test to 1 rank.

Fix issue in quest_signed_distance_interface test. Complete shared me…

7d4245d

…mory via Umpire, as far as I can tell.

Initial change to Axom spack package.py to select Umpire that has MPI…

4c73b35

…3 shared memory support.

BradWhitlock marked this pull request as draft September 6, 2025 01:09

BradWhitlock added 3 commits September 5, 2025 18:13

make style

55d8f1c

Fix variable name use.

ebab4a6

Change message about Umpire.

6adb053

white238 reviewed Sep 6, 2025

View reviewed changes

src/axom/quest/interface/internal/QuestHelpers.cpp Outdated Show resolved Hide resolved

BradWhitlock added 7 commits September 8, 2025 11:07

Fix old typo

4684659

Fix conditional compilation block.

cb19673

Removed extra dashes.

73ea86a

Avoid direct MPI calls for case when this MPI test is actually NOT us…

f12ff68

…ed with MPI.

Fix mistake

88fa450

Change macro used to select MPI abstraction functions.

5aa1325

Moved a function so it is more like the previous ordering.

f114986

BradWhitlock marked this pull request as ready for review September 9, 2025 00:08

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

e849acc

kennyweiss requested changes Sep 9, 2025

View reviewed changes

BradWhitlock added 4 commits September 9, 2025 12:31

Make AXOM_USE_UMPIRE_SHARED_MEMORY be set in config.hpp

0ddce42

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

ce0f380

Removed MPI build option.

40c11f1

Merge branch 'feature/whitlock/use_umpire_shared_memory' of github.co…

0c2f6c1

…m:LLNL/axom into feature/whitlock/use_umpire_shared_memory

kennyweiss approved these changes Sep 9, 2025

View reviewed changes

kennyweiss added enhancement New feature or request TPL Issues related to Axom's third party libraries Build system Issues related to Axom's build system memory labels Sep 9, 2025

BradWhitlock added 2 commits September 9, 2025 15:04

Moved shared memory allocator to core.

ee8de79

make style

ee8e9ba

BradWhitlock added 4 commits September 9, 2025 15:14

Moved some function declarations.

d367dc7

Change how we get some Umpire includes

4ce036a

Removed some comments related to MPI windows.

cad2b2b

Removed some Umpire includes from signed_distance.

c69d950

BradWhitlock requested review from bmhan12, gunney1, publixsubfan and rhornung67 September 9, 2025 23:18

kennyweiss approved these changes Sep 9, 2025

View reviewed changes

BradWhitlock added 4 commits September 9, 2025 16:45

Renamed axom shared allocator.

7bef6e0

Merged develop into branch

c14f563

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

2118611

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

971165e

kennyweiss added the Reviewed label Sep 15, 2025

rhornung67 approved these changes Sep 23, 2025

View reviewed changes

BradWhitlock added 2 commits September 23, 2025 11:36

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

984cd26

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

12395db

gunney1 reviewed Oct 1, 2025

View reviewed changes

RELEASE-NOTES.md

Copy link

Contributor

gunney1 Oct 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice functionality @BradWhitlock !

gunney1 approved these changes Oct 1, 2025

View reviewed changes

BradWhitlock added 2 commits October 1, 2025 11:47

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

8b74c20

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

9b0dd9c

BradWhitlock added 4 commits October 21, 2025 10:25

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

5475938

Use UMPIRE_DEFAULT_SHARED_MEMORY_RESOURCE in the name of the requeste…

9e8d791

…d shared memory allocator. Add cmake status message for value of UMPIRE_DEFAULT_SHARED_MEMORY_RESOURCE.

Merge branch 'feature/whitlock/use_umpire_shared_memory' of github.co…

a7114d8

…m:LLNL/axom into feature/whitlock/use_umpire_shared_memory

Merge branch 'develop' into feature/whitlock/use_umpire_shared_memory

d3e3ef8

BradWhitlock merged commit 3c96e95 into develop Oct 22, 2025
15 checks passed

BradWhitlock deleted the feature/whitlock/use_umpire_shared_memory branch October 22, 2025 00:23

		cmake_dependent_option(AXOM_ENABLE_MPI3 "Enables use of MPI-3 features" OFF "ENABLE_MPI" OFF)
		mark_as_advanced(AXOM_ENABLE_MPI3)

Change MPI3 shared memory in Quest so it uses Umpire. #1652

Change MPI3 shared memory in Quest so it uses Umpire. #1652

Uh oh!

Conversation

BradWhitlock commented Sep 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kennyweiss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kennyweiss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kennyweiss left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gunney1 left a comment

Choose a reason for hiding this comment

Uh oh!

kab163 commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kab163 commented Oct 7, 2025

Uh oh!

kab163 commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

BradWhitlock commented Sep 6, 2025 •

edited

Loading

kab163 commented Oct 7, 2025 •

edited

Loading

kab163 commented Oct 7, 2025 •

edited

Loading