Subfiling

Very little work on this. These are the changes that I had made at one time. Not sure of status:

diff --git a/TPL/hdf5/runcmake.sh b/TPL/hdf5/runcmake.sh
index 43ad4bb936..5c4559009b 100644
--- a/TPL/hdf5/runcmake.sh
+++ b/TPL/hdf5/runcmake.sh
@@ -66,6 +66,7 @@ cmake .. -DCMAKE_C_COMPILER:FILEPATH=${CC} \
         -DHDF5_ENABLE_PARALLEL:BOOL=${MPI} \
         -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON \
         -DHDF5_BUILD_CPP_LIB:BOOL=OFF \
+        -DHDF5_ENABLE_SUBFILING_VFD:BOOL=YES \
         -DHDF5_BUILD_FORTRAN:BOOL=OFF \
         -DHDF5_BUILD_HL_LIB:BOOL=ON \
         -DHDF5_DISABLE_COMPILER_WARNINGS:BOOL=ON \
diff --git a/packages/seacas/libraries/exodus/src/ex_utils.c b/packages/seacas/libraries/exodus/src/ex_utils.c
index e92cffdea5..7f1f84914b 100644
--- a/packages/seacas/libraries/exodus/src/ex_utils.c
+++ b/packages/seacas/libraries/exodus/src/ex_utils.c
@@ -1764,11 +1764,6 @@ void ex__compress_variable(int exoid, int varid, int type)
 #endif
       }
     }
-#if defined(PARALLEL_AWARE_EXODUS)
-    if (file->is_parallel) {
-      nc_var_par_access(exoid, varid, NC_COLLECTIVE);
-    }
-#endif
   }
 #endif
 }
diff --git a/packages/seacas/libraries/ioss/src/main/io_shell.C b/packages/seacas/libraries/ioss/src/main/io_shell.C
index da07b5c6c9..a9f5d9f820 100644
--- a/packages/seacas/libraries/ioss/src/main/io_shell.C
+++ b/packages/seacas/libraries/ioss/src/main/io_shell.C
@@ -71,7 +71,12 @@ namespace {
 int main(int argc, char *argv[])
 {
 #ifdef SEACAS_HAVE_MPI
-  MPI_Init(&argc, &argv);
+  /* Initialize MPI with threading support */
+  int mpi_thread_required = MPI_THREAD_MULTIPLE;
+  int mpi_thread_provided = 0;
+  MPI_Init_thread(&argc, &argv, mpi_thread_required, &mpi_thread_provided);
+  if (mpi_thread_provided != mpi_thread_required)
+     MPI_Abort(Ioss::ParallelUtils::comm_world(), -1);
   ON_BLOCK_EXIT(MPI_Finalize);
 #endif
   Ioss::ParallelUtils pu{};

CGNS Experience with subfiling

The CGNS "subfiling" branch uses an HDF5 v1.14 feature, which splits an HDF5 file among the nodes. Each node writes a section of the CGNS/HDF5 file. If you have node-local storage, you can write to that and, in the background, merge the subfiles into a single CGNS/HDF5 file if needed. Although that feature shines >10k nodes, it may not improve much for two nodes. There might be another issue if you see a drop-off for two nodes. Does your system create darshan logs, https://wordpress.cels.anl.gov/darshan/? Can you give the Lustre(?) parameters that you are using? I happen to be holding an open questions session this Tuesday if you would rather discuss it in person: https://www.hdfgroup.org/call-the-doctor/

Also, only some of the ranks need to participate in I/O, just those members of the MPI communicator. That way, you can use either independent or collective. Sometimes, the I/O times can vary dramatically between the two.

You can also create the CGNS "skeleton" on one rank, close the file, and open it on all the ranks to write the raw data. You can then have ranks make the write calls but not write any data. What versions of CGNS and HDF5 are you using?