IOSS: add an S3 database component (#725)

* IOSS: add an S3 database component

* update copyright header

* make S3 optional in non-CMake builds

* Ionit: TPL_ENABLE_S3 -> TPL_ENABLE_AWSSDK AND TPL_ENABLE_Cereal
Ioss: TPL_ENABLE_AWSSDK -> TPL_ENABLE_AWSSDK AND TPL_ENABLE_Cereal

* use cstdlib instead of stdlib.h

* use fmt::print instead of std::cout and std::cerr

* don't use ostringstream to create error messages for IOSS_ERROR()

* use default member initialization

* all vectors should be passed by reference

* add back the missing "null" type

* use MAX_LINE_LENGTH for info records instead of MAX_STR_LENGTH

* use fmt::format to assemble keys

* use fmt::format to create subkey search terms

* remove temporary and optional code that was commented out

* rewrite parameter processing
* make a local nonconst copy of the PropertyManager
* populate it with the existing Properties from const PropertyManager
* get any S3 parameters from environment variables and add to the PropertyManager
* populate the S3 parameter struct from the PropertyManager

* put internal methods in the anonymous namespace

* formatting

* make better use of const-ref
use IOSS_ERROR for unrecoverable errors
remove unnecessary error message

* fix copyright header

* remove unused assert header
add a PackedBytes type alias to clarify pack/unpack vector usage

* use IOSS_ERROR for unrecoverable errors

* fix copyright header
use ${CMAKE_SHARED_LIBRARY_SUFFIX} instead of hardcoded .so

* set minimum policy version to 3.5
add "lib" to the list of install dirs to look in for AWS libraries
only enable S3 if both AWSSDK and Cereal are found

* set a default NETCDF_ROOT

* improve the documentation of the IOSS-S3 component and TPLs

* add IOSS-S3 configuration option to cmake-config.
remove cmake-s3 script in favor of cmake-config.

* add AWS TPL to install-tpl.sh

* AWS helpers now return bool instead of int
change result var name from rc to success

* change map_fields result name to num_failed.
change map_properties result name to num_failed.

* envvar to properties: use better variable names
properties to params: respect boolean values not just existence of the property

* update copyright header

* in README.md, refer to README-S3.md for AWSSDK info

* README-S3.md: fix IOSS_S3_DISABLE_EC2_LOOKUP default. should be True.

* Committing clang-format changes

---------

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>

Author: tkordenbrock

README-S3.md

@@ -0,0 +1,59 @@
+# Introduction
+
+This document describes the requirements to build and use the IOSS-S3
+DatabaseIO component.
+
+## Installing the AWS SDK for C++ and Cereal TPLs
+
+In addition to the usual TPLs required for building SEACAS, the
+IOSS-S3 component requires the Amazon Web Services (AWS) Software
+Development Kit (SDK) for C++ and the Cereal serialization
+library.  These TPLs can be installed using the `install-tpl.sh`
+script as described in [README.md](README.md).  There is also a
+description of manual installation in
+[TPL-Manual-Install.md](TPL-Manual-Install.md).
+
+## Building SEACAS with S3
+
+The `cmake-config` script is able to build the IOSS-S3 component using
+the TPLs installed by the `install-tpl.sh` script.  When it finds both
+the AWS SDK for C++ and Cereal, it will enable the IOSS-S3 component.
+A full description of `cmake-config` can be found in
+[README.md](README.md).
+
+## Passing Parameters to the IOSS-S3 component
+
+The IOSS-S3 component has several parameters that control its
+behavior.  Each parameter can be specified either by the user setting
+an environment variable or by the application by setting an IOSS
+Property.  If both are specified, the environment variable takes
+precedence over the Property.
+
+The parameters are descibed in the table below.
+
+| Property or Environment Variable | Values          | Default | Description |
+|----------------------------------|:---------------:|:-------:|-------------|
+| IOSS_S3_ENDPOINT                 | URI of the S3 object store           | None    | This parameter is required and does not have a default. |
+| IOSS_S3_PROFILE                  | name of the S3 credential profile    | default |  |
+| IOSS_S3_CA_FILE                  | path to a certificate authority file | None    | Only required with your S3 object store requires it. |
+| IOSS_S3_USE_CA_FILE              | enable the CA file                   | False   |  |
+| IOSS_S3_USE_TRANSFER_MANAGER     | enable the transfer manager          | False   | Enable to transfer large data. |
+| IOSS_S3_ENABLE_AWS_TRACING       | enable debug in AWSSDK               | False   | Caution: the output is very verbose. |
+| IOSS_S3_DISABLE_EC2_LOOKUP       | disable an EC2 lookup at startup     | True    | Useful for on-prem object stores. |
+
+
+## Example: copy an Exodus file to S3 and back
+
+A simple way to verify your IOSS-S3 installation is to copy an Exodus
+file into your S3 object store, copy it back out to disk and then
+compare them.  The io_shell tool can be used to copy data in and out
+of S3.  In this example, the data is Exodus data so the exodiff tool
+will be used to compare the original file to the copy.
+
+```
+export IOSS_S3_ENDPOINT="https://mys3server"
+export IOSS_S3_PROFILE=my-s3-profile
+io_shell -in_type exodus -out_type s3 my_exodus_file.exo my_exodus_file.exo
+io_shell -in_type s3 -out_type exodus my_exodus_file.exo my_exodus_file_copy.exo
+exodiff my_exodus_file.exo my_exodus_file_copy.exo
+```

README.md

@@ -46,9 +46,9 @@ cd seacas && export ACCESS=`pwd`
 
 There are a few externally developed third-party libraries (TPL) that
 are required (or optional) to build SEACAS: HDF5, NetCDF, CGNS, MatIO,
-Kokkos, and (if MPI set) PnetCDF libraries. You can build the
-libraries using the `install-tpl.sh` script, or you can install them
-manually as detailed in
+Kokkos, (if MPI set) PnetCDF, AWS SDK for C++ and Cereal
+libraries. You can build the libraries using the `install-tpl.sh`
+script, or you can install them manually as detailed in
 [TPL-Manual-Install.md](TPL-Manual-Install.md).
 
 *  The script requires bash and curl, which you may need to install.
@@ -78,6 +78,8 @@ manually as detailed in
 | PARMETIS        | YES, NO | NO  | Should parmetis TPL be built (parallel decomposition). |
 | ADIOS2          | YES, NO | NO  | Should adios2 TPL be built. |
 | CATALYST2       | YES, NO | NO  | Should catalyst 2 TPL be built. |
+| AWSSDK          | YES, NO | NO  | Should AWS SDK for C++ TPL be built. See [README-S3.md](README-S3.md). |
+| CEREAL          | YES, NO | NO  | Should Cereal TPL be built. |
 | KOKKOS          | YES, NO | NO  | Should Kokkos TPL be built. |
 | GNU_PARALLEL    | YES, NO | YES | Should GNU parallel script be built. |
 | FMT             | YES, NO | YES | Should Lib::FMT TPL be built. |

TPL-Manual-Install.md

@@ -11,6 +11,8 @@ described in the previous section.
 *  [GNU Parallel](#gnu-parallel) -- optional
 *  [CGNS](#cgns) -- experimental optional
 *  [Faodel](#faodel) -- optional
+*  [AWS SDK for C++](#aws-sdk-cpp) -- optional
+*  [Cereal](#cereal) -- optional
 
 ## Zoltan
 A snapshot of [zoltan_distrib\_v3.83.tar.gz](http://www.cs.sandia.gov/Zoltan/Zoltan_download.html) is provided in seacas/packages/zoltan.  This will be built automatically as part of the SEACAS build process.
@@ -154,3 +156,72 @@ The IOSS library supports using CGNS for structured and unstructred meshes.  To
 Faodel is a collection of data management tools that Sandia is developing to improve how datasets migrate between memory and storage resources in a distributed system. For SEACAS Faodel support means adding a new backend to IOSS. This enables additional data storage capabilities and the chance to communicate data between execution spaces.
 
 Faodel is available at [Faodel](https://github.com/faodel/faodel). And is built here as a SEACAS TPL.
+
+## AWS SDK for C++
+The AWS SDK for C++ is the official C++ interface to the entire Amazon
+Web Services ecosystem.  This includes packages for transferring data
+to and from S3 (Simple Storage Service) object stores.  For SEACAS,
+AWS support means adding a new backend to IOSS that stores mesh data
+in S3.
+
+The AWS SDK for C++ is available at
+[aws-sdk-cpp](https://github.com/aws/aws-sdk-cpp). And is built here
+as a SEACAS TPL.
+
+The AWS SDK for C++ is in multiple github repositories and composed
+using submodules.  Below is an example of building the tested version
+1.11.77.  Complete build instructions for the AWS SDK for C++ can be
+found here:
+https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/setup-linux.html
+
+
+*  Cloning from github.
+
+  * `cd TPL/aws-sdk-cpp`
+  
+  * `git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp`
+  
+  * `cd aws-sdk-cpp`
+  
+  * `git branch branch/tag-1.11.77 1.11.77`
+  
+  * `git checkout branch/tag-1.11.77`
+  
+  * `git submodule update --recursive`
+
+*  Build using CMake.
+
+  *  Modify `TPL/aws-sdk-cpp/runcmake.sh` to meet your environment
+
+  *  `mkdir build`
+
+  *  `cd build`
+
+  *  `../../runcmake.sh`
+
+  *  `make && make install`
+
+
+## Cereal
+cereal is a serialization library that transforms C++ objects into
+transportable representations that can be transferred between process
+spaces and storage.  For SEACAS, cereal support is required to store
+complex IOSS objects in Faodel and S3.  The Faodel and S3 database
+components require that cereal be installed.
+
+cereal is available at
+[cereal](https://github.com/USCiLab/cereal). And is built here as a
+SEACAS TPL.
+
+cereal is a header-only library so installation is as simple as
+copying the files to your install directory.
+
+  * `mkdir TPL/cereal`
+  
+  * `cd TPL/cereal`
+  
+  * `curl -O -L --insecure https://github.com/USCiLab/cereal/archive/v1.3.2.tar.gz`
+  
+  * `tar xzf v1.3.2.tar.gz`
+  
+  * `cp -R cereal-1.3.2/include/cereal $INSTALL_PATH/include/`

TPL/aws-sdk-cpp/runcmake.sh

@@ -0,0 +1,59 @@
+#! /usr/bin/env bash
+
+### The following assumes you are building in a subdirectory of ACCESS Root
+if [ "X$ACCESS" == "X" ] ; then
+  ACCESS=$(cd ../../../..; pwd)
+  echo "ACCESS set to ${ACCESS}"
+fi
+INSTALL_PATH=${INSTALL_PATH:-${ACCESS}}
+
+DEBUG="${DEBUG:-NO}"
+if [ "$DEBUG" == "YES" ]
+then
+  BUILD_TYPE="Debug"
+else
+  BUILD_TYPE="Release"
+fi
+
+if [ "$CRAY" == "YES" ]
+then
+    SHARED="${SHARED:-NO}"
+else
+    SHARED="${SHARED:-YES}"
+fi
+
+if [[ "$SHARED" == "ON" || "$SHARED" == "YES" ]]
+then
+  OS=$(uname -s)
+  if [ "$OS" = "Darwin" ] ; then
+    LD_EXT="dylib"
+  else
+    LD_EXT="so"
+  fi
+else
+  LD_EXT="a"
+fi
+
+. ${ACCESS}/TPL/compiler.sh
+
+# If using an XLF compiler on an IBM system, may need to add the following:
+# -DCMAKE_Fortran_FLAGS="-qfixed=72" \
+# -DCMAKE_EXE_LINKER_FLAGS:STRING="-lxl -lxlopt"
+
+rm -f config.cache
+
+cmake .. -DCMAKE_C_COMPILER:FILEPATH=${CC} \
+         -DBUILD_SHARED_LIBS:BOOL=${SHARED} \
+         -DCMAKE_POSITION_INDEPENDENT_CODE:BOOL=ON \
+         -DBUILD_TESTING:BOOL=OFF \
+         -DCMAKE_INSTALL_PREFIX=${INSTALL_PATH} \
+	 -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
+         \
+         -DAUTORUN_UNIT_TESTS:BOOL=OFF
+
+echo ""
+echo "         MPI: ${MPI}"
+echo "    COMPILER: ${CC}"
+echo "      ACCESS: ${ACCESS}"
+echo "INSTALL_PATH: ${INSTALL_PATH}"
+echo ""

TPLsList.cmake

@@ -72,6 +72,7 @@ SET( Seacas_TPLS_FINDMODS_CLASSIFICATIONS
   Matio           "cmake/TPLs/"    SS
   CUDA            "cmake/TPLs/"    SS
   Kokkos          "${${PROJECT_NAME}_TRIBITS_DIR}/common_tpls/"  SS
+  AWSSDK          "${PROJECT_SOURCE_DIR}/packages/seacas/cmake/tpls/"    SS
   Faodel          "${PROJECT_SOURCE_DIR}/packages/seacas/cmake/tpls/"    SS
   Cereal          "${PROJECT_SOURCE_DIR}/packages/seacas/cmake/tpls/"    SS
   ADIOS2          "${${PROJECT_NAME}_TRIBITS_DIR}/common_tpls/"  EX

cmake-config

@@ -75,6 +75,8 @@ KOKKOS_PATH=${KOKKOS_PATH:-${INSTALL_PATH}}
 METIS_PATH=${METIS_PATH:-${INSTALL_PATH}}
 PARMETIS_PATH=${PARMETIS_PATH:-${INSTALL_PATH}}
 FMT_PATH=${FMT_PATH:-${INSTALL_PATH}}
+AWSSDK_PATH=${AWSSDK_PATH:-${INSTALL_PATH}}
+CEREAL_PATH=${CEREAL_PATH:-${INSTALL_PATH}}
 
 ### Set to ON for parallel compile; otherwise OFF for serial (default)
 if [ "${MPI}" = "" ]
@@ -330,6 +332,8 @@ HAVE_MATIO=$(check_enable "${MATIO_PATH}/include/matio.h")
 HAVE_CGNS=$(check_enable "${CGNS_PATH}/include/cgnslib.h")
 HAVE_FAODEL=$(check_enable "${FAODEL_PATH}/include/faodel/faodelConfig.h")
 HAVE_ADIOS2=$(check_enable "${ADIOS2_PATH}/include/adios2.h")
+HAVE_AWSSDK=$(check_enable "${AWSSDK_PATH}/include/aws/core/Aws.h")
+HAVE_CEREAL=$(check_enable "${CEREAL_PATH}/include/cereal/cereal.hpp")
 HAVE_CATALYST2=$(check_enable "${CATALYST2_PATH}/include/catalyst-2.0/catalyst.h")
 HAVE_GTEST=$(check_enable "${GTEST_PATH}/include/gtest/gtest.h")
 HAVE_CATCH2=$(check_enable "${CATCH2_PATH}/include/catch2/catch_all.hpp")
@@ -399,6 +403,16 @@ then
                     -DTPL_Kokkos_LIBRARIES=${KOKKOS_PATH}/lib/libkokkoscore.${LD_EXT}"
 fi
 
+if [ "$HAVE_AWSSDK" == "YES" ] && [ "$HAVE_CEREAL" == "YES" ]
+then
+  AWSSDK_SYMBOLS=" -D TPL_ENABLE_AWSSDK:BOOL=${HAVE_AWSSDK}   \
+    -D AWSSDK_ROOT:PATH=${AWSSDK_PATH} \
+    "
+  CEREAL_SYMBOLS=" -D TPL_ENABLE_Cereal:BOOL=${HAVE_CEREAL}   \
+    -D Cereal_INCLUDE_DIRS:PATH=${CEREAL_PATH}/include \
+    "
+fi
+
 if [ "$HAVE_FAODEL" == "YES" ]
 then
   FAODEL_SYMBOLS=" -D TPL_ENABLE_Faodel:BOOL=${HAVE_FAODEL} \
@@ -547,6 +561,8 @@ ${MPI_SYMBOLS} \
 ${DARWIN_OPT} \
 ${FAODEL_SYMBOLS} \
 ${ADIOS2_SYMBOLS} \
+${AWSSDK_SYMBOLS} \
+${CEREAL_SYMBOLS} \
 \
 -D MPI_BIN_DIR:PATH=${MPI_BIN} \
 -D NetCDF_ROOT:PATH=${NETCDF_PATH} \
@@ -589,6 +605,9 @@ echo "  CATALYST2: ${HAVE_CATALYST2}"
 echo "      METIS: ${HAVE_METIS}"
 echo "   PARMETIS: ${HAVE_PARMETIS}"
 echo "     FAODEL: ${HAVE_FAODEL}"
+echo "     AWSSDK: ${HAVE_AWSSDK}"
+echo "AWSSDK_PATH: ${AWSSDK_PATH}"
+echo "CEREAL_PATH: ${CEREAL_PATH}"
 echo "      GTEST: ${HAVE_GTEST}"
 echo "     CATCH2: ${HAVE_CATCH2}"
 echo "    DOXYGEN: ${DOXYGEN}"

install-tpl.sh

@@ -139,6 +139,12 @@ ADIOS2=$(check_valid ADIOS2) || exit
 CATALYST2=${CATALYST2:-NO}
 CATALYST2=$(check_valid CATALYST2) || exit
 
+AWSSDK=${AWSSDK:-NO}
+AWSSDK=$(check_valid AWSSDK) || exit
+
+CEREAL=${CEREAL:-NO}
+CEREAL=$(check_valid CEREAL) || exit
+
 GTEST=${GTEST:-${FAODEL}}
 GTEST=$(check_valid GTEST) || exit
 
@@ -232,6 +238,8 @@ fi
         echo "   FAODEL       = ${FAODEL}"
         echo "   ADIOS2       = ${ADIOS2}"
         echo "   CATALYST2    = ${CATALYST2}"
+        echo "   AWSSDK       = ${AWSSDK}"
+        echo "   CEREAL       = ${CEREAL}"
         echo "   CATCH2       = ${CATCH2}"
         echo "   GTEST        = ${GTEST}"
         echo ""
@@ -1141,32 +1149,77 @@ then
   fi
 fi
 
+# =================== INSTALL aws-sdk-cpp ===============
+if [ "$AWSSDK" == "YES" ]
+then
+  if [ "$FORCE" == "YES" ] || ! [ -e $INSTALL_PATH/include/aws/core/Aws.h ]
+  then
+    echo "${txtgrn}+++ aws-sdk-cpp${txtrst}"
+    cd $ACCESS || exit
+    cd TPL/aws-sdk-cpp || exit
+    if [ "$DOWNLOAD" == "YES" ]
+    then
+      echo "${txtgrn}+++ Downloading...${txtrst}"
+      rm -rf aws-sdk-cpp
+      git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp
+      cd aws-sdk-cpp
+      git branch branch/tag-1.11.77 1.11.77
+      git checkout branch/tag-1.11.77
+      git submodule update --recursive
+      cd ..
+    fi
+
+    if [ "$BUILD" == "YES" ]
+    then
+      echo "${txtgrn}+++ Configuring, Building, and Installing...${txtrst}"
+      cd aws-sdk-cpp || exit
+      rm -rf build
+      mkdir build
+      cd build || exit
+      CRAY=${CRAY} DEBUG=${DEBUG} SHARED=${SHARED} MPI=${MPI} bash -x ../../runcmake.sh
+      if [[ $? != 0 ]]
+      then
+  	echo 1>&2 ${txtred}couldn\'t configure aws-sdk-cpp. exiting.${txtrst}
+  	exit 1
+      fi
+      make -j${JOBS} && ${SUDO} make "V=${VERBOSE}" install
+      if [[ $? != 0 ]]
+      then
+  	echo 1>&2 ${txtred}couldn\'t build aws-sdk-cpp. exiting.${txtrst}
+  	exit 1
+      fi
+    fi
+  else
+    echo "${txtylw}+++ aws-sdk-cpp already installed.  Skipping download and installation.${txtrst}"
+  fi
+fi
+
 # =================== INSTALL CEREAL ===============
-if [ "$FAODEL" == "YES" ]
+if [ "$AWSSDK" == "YES" ] || [ "$FAODEL" == "YES" ]
 then
-  # Currently, the FAODEL backend requires cereal, so if Faodel is enabled, we'll install cereal, too.
-  if [ "$FORCE" == "YES" ] || [ "$FORCE_FAODEL" == "YES" ] || ! [ -e $INSTALL_PATH/include/cereal/archives/portable_binary.hpp ]
+  # Currently, the S3 and FAODEL backends require cereal.  If S3 or Faodel are enabled, we'll install cereal too.
+  if [ "$FORCE" == "YES" ] || [ "$FORCE_AWSSDK" == "YES" ] || [ "$FORCE_FAODEL" == "YES" ] || ! [ -e $INSTALL_PATH/include/cereal/archives/portable_binary.hpp ]
   then
     echo "${txtgrn}+++ Cereal${txtrst}"
     cd $ACCESS || exit
     CEREAL_DIR="TPL/cereal"
     if [ ! -d "${CEREAL_DIR}" ]; then
       mkdir ${CEREAL_DIR}
     fi
-    cd ${CEREAL} || exit
+    cd ${CEREAL_DIR} || exit
     if [ "$DOWNLOAD" == "YES" ]
     then
       echo "${txtgrn}+++ Downloading...${txtrst}"
       rm -rf cereal*
-      curl -O -L --insecure https://github.com/USCiLab/cereal/archive/v1.3.0.tar.gz
-      tar xzf v1.3.0.tar.gz
-      rm -f v1.3.0.tar.gz
-      cp -R cereal-1.3.0/include/cereal $INSTALL_PATH/include/
+      curl -O -L --insecure https://github.com/USCiLab/cereal/archive/v1.3.2.tar.gz
+      tar xzf v1.3.2.tar.gz
+      rm -f v1.3.2.tar.gz
     fi
 
     if [ "$BUILD" == "YES" ]
     then
       echo "${txtgrn}+++ Configuring, Building, and Installing...${txtrst}"
+      cp -R cereal-1.3.2/include/cereal $INSTALL_PATH/include/
     fi
   else
     echo "${txtylw}+++ Cereal already installed.  Skipping download and installation.${txtrst}"