Skip to content

Commit 9a34c1a

Browse files
tkordenbrocktkordenbrock
committed
IOSS: add an S3 database component
1 parent e67472f commit 9a34c1a

24 files changed

+5190
-4
lines changed

‎README-S3.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
2+
# Introduction
3+
4+
This document describes the requirements to build and use the IOSS-S3
5+
DatabaseIO component.
6+
7+
In addition to the usual TPLs required for building SEACAS, the
8+
IOSS-S3 component requires the Amazon Simple Storage Service (S3) SDK
9+
for C++ (AWSSDK) and the Cereal serialization library. Below are
10+
instructions to configure and build AWSSDK and Cereal followed by
11+
instructions to configure and build SEACAS with the IOSS-S3 component
12+
enabled.
13+
14+
## Setup the Environment
15+
16+
First identify a place to build the AWSSDK and Cereal TPLs and SEACAS.
17+
18+
```
19+
SEACAS_S3_ROOT=$HOME/seacas
20+
21+
SEACAS_TPL_SRC_PATH=$SEACAS_S3_ROOT/tpls/src
22+
SEACAS_TPL_INSTALL_PATH=$SEACAS_S3_ROOT/tpls/install
23+
mkdir -p $SEACAS_TPL_SRC_PATH
24+
mkdir -p $SEACAS_TPL_INSTALL_PATH
25+
26+
SEACAS_SRC_PATH=$SEACAS_S3_ROOT/src
27+
SEACAS_INSTALL_PATH=$SEACAS_S3_ROOT/install
28+
mkdir -p $SEACAS_SRC_PATH
29+
mkdir -p $SEACAS_INSTALL_PATH
30+
```
31+
32+
## Installing the AWSSDK TPL
33+
34+
Complete build instructions for the AWSSDK TPL can be found here:
35+
https://docs.aws.amazon.com/sdk-for-cpp/v1/developer-guide/setup-linux.html
36+
37+
The IOSS-S3 component has been tested against AWSSDK tag 1.11.77.
38+
Here is an example of building and installing the tested version.
39+
40+
```
41+
cd $SEACAS_TPL_SRC_PATH
42+
git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp
43+
cd aws-sdk-cpp
44+
git branch branch/tag-1.11.77 1.11.77
45+
git checkout branch/tag-1.11.77
46+
git submodule update --recursive
47+
mkdir build
48+
cd build
49+
50+
cmake -DAUTORUN_UNIT_TESTS:BOOL=OFF -DCMAKE_INSTALL_PREFIX=$SEACAS_TPL_INSTALL_PATH ..
51+
make install
52+
```
53+
54+
## Installing the Cereal TPL
55+
56+
The IOSS-S3 component has been tested against Cereal v1.3.2.
57+
Here is an example of building and installing the tested version.
58+
59+
```
60+
cd $SEACAS_TPL_SRC_PATH
61+
curl -o cereal-1.3.2.tar.gz https://codeload.github.com/USCiLab/cereal/tar.gz/refs/tags/v1.3.2
62+
tar xf cereal-1.3.2.tar.gz
63+
cd cereal-1.3.2
64+
mkdir build
65+
cd build
66+
67+
cmake -DBUILD_SANDBOX=OFF -DBUILD_TESTS=OFF -DSKIP_PERFORMANCE_COMPARISON=true -DCMAKE_INSTALL_PREFIX=$SEACAS_TPL_INSTALL_PATH ..
68+
make install
69+
```
70+
71+
## Building SEACAS TPLs unrelated to the IOSS-S3 component
72+
73+
SEACAS has many optional TPLs that may be required for your build.
74+
75+
For any SEACAS build, you will need libfmt. The latest release can be
76+
found here:
77+
78+
https://github.com/fmtlib/fmt/releases
79+
80+
It should be built using shared libraries, so it should be built like
81+
this:
82+
83+
```
84+
cd $SEACAS_TPL_SRC_PATH
85+
curl -o fmt-11.2.0.tar.gz https://codeload.github.com/fmtlib/fmt/tar.gz/refs/tags/11.2.0
86+
tar xf fmt-11.2.0.tar.gz
87+
cd fmt-11.2.0
88+
mkdir build
89+
cd build
90+
cmake -DBUILD_SHARED_LIBS=TRUE -DCMAKE_INSTALL_PREFIX=$SEACAS_TPL_INSTALL_PATH ..
91+
make install
92+
```
93+
94+
In order to test the IOSS-S3 component, you may want to build the
95+
IOSS-Exodus component. IOSS-Exodus requires the NetCDF TPL. It is
96+
important that the correct NetCDF installation is found so it must be
97+
spcified explicitly by setting the NETCDF_ROOT variable.
98+
99+
There are other optional TPLs depending on the enabled features. The
100+
`cmake-s3` script referenced below expects to find these TPLs in the
101+
directory specified by TPL_INSTALL.
102+
103+
## Building SEACAS with S3
104+
105+
The `cmake-s3` script will discover the available TPLs and execute
106+
cmake to configure SEACAS with the IOSS-S3 and IOSS-Exodus components
107+
enabled.
108+
109+
`cmake-s3` uses the full path to the nc-config utility to ensure that
110+
it finds the correct NetCDF on your system. In the example below, we
111+
assume that NetCDF and all other TPLs not installed for S3 are in
112+
`/usr`, so both NETCDF_ROOT and TPL_INSTALL are set to `/usr`.
113+
114+
```
115+
cd $SEACAS_SRC_PATH
116+
git clone https://github.com/sandialabs/seacas.git
117+
cd seacas
118+
mkdir build
119+
cd build
120+
121+
NETCDF_ROOT=/usr \
122+
TPL_INSTALL=/usr \
123+
CEREAL_ROOT=$SEACAS_TPL_INSTALL_PATH \
124+
FMT_PATH=$SEACAS_TPL_INSTALL_PATH \
125+
AWSSDK_DIR=$SEACAS_TPL_INSTALL_PATH \
126+
INSTALL_PATH=$SEACAS_INSTALL_PATH \
127+
bash ../cmake-s3
128+
make install
129+
```
130+
131+
## Passing Parameters to the IOSS-S3 component
132+
133+
The IOSS-S3 component has several parameters that control its
134+
behavior. They are descibed in the table below.
135+
136+
| Environment Variable | Values | Default | Description |
137+
|----------------------|:---------------:|:-------:|-------------|
138+
| IOSS_S3_ENDPOINT | URI of the S3 object store | None | This parameter is required and does not have a default. |
139+
| IOSS_S3_PROFILE | name of the S3 credential profile | default | |
140+
| IOSS_S3_CA_FILE | path to a certificate authority file | None | Only required with your S3 object store requires it. |
141+
| IOSS_S3_USE_CA_FILE | enable the CA file | False | |
142+
| IOSS_S3_USE_TRANSFER_MANAGER | enable the transfer manager | False | Enable to transfer large data. |
143+
| IOSS_S3_ENABLE_AWS_TRACING | enable debug in AWSSDK | False | Caution: the output is very verbose. |
144+
| IOSS_S3_DISABLE_EC2_LOOKUP | disable an EC2 lookup at startup | False | Useful for on-prem object stores. |
145+
146+
## Example: copy an Exodus file to S3 and back
147+
148+
A simple way to verify your IOSS-S3 installation is to copy an Exodus
149+
file into your S3 object store, copy it back out to disk and then
150+
compare them. The io_shell tool can be used to copy data in and out
151+
of S3. In this example, the data is Exodus data so the exodiff tool
152+
will be used to compare the original file to the copy.
153+
154+
```
155+
export IOSS_S3_ENDPOINT="https://mys3server"
156+
export IOSS_S3_PROFILE=my-s3-profile
157+
$SEACAS_INSTALL_PATH/bin/io_shell -in_type exodus -out_type s3 my_exodus_file.exo my_exodus_file.exo
158+
$SEACAS_INSTALL_PATH/bin/io_shell -in_type s3 -out_type exodus my_exodus_file.exo my_exodus_file_copy.exo
159+
$SEACAS_INSTALL_PATH/bin/exodiff my_exodus_file.exo my_exodus_file_copy.exo
160+
```

‎TPLsList.cmake‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ SET( Seacas_TPLS_FINDMODS_CLASSIFICATIONS
7272
Matio "cmake/TPLs/" SS
7373
CUDA "cmake/TPLs/" SS
7474
Kokkos "${${PROJECT_NAME}_TRIBITS_DIR}/common_tpls/" SS
75+
AWSSDK "${PROJECT_SOURCE_DIR}/packages/seacas/cmake/tpls/" SS
7576
Faodel "${PROJECT_SOURCE_DIR}/packages/seacas/cmake/tpls/" SS
7677
Cereal "${PROJECT_SOURCE_DIR}/packages/seacas/cmake/tpls/" SS
7778
ADIOS2 "${${PROJECT_NAME}_TRIBITS_DIR}/common_tpls/" EX

0 commit comments

Comments
 (0)