Synchronize with develop (#256)

* Add pdc_logger.h to installation (#245)

* sync with gitlab (#248)

* Fix restart issue (#228)

* Fix cache flush (#226)

* Fix a thread race issue that may cause memory error when larger than cache max size data is transferred

* Add a test that writes more data than server cache size

* Fix CI run command

* Fix restart issue

* Update nersc.yml (#238)

* Since PDCinit returns a uint64_t, 0 should indicate failure (#233)

Co-authored-by: Jean Luca Bez <[email protected]>

* Check the return value of `PDC_Client_init` in `PDC_init` (#230)

* Check that return value of PDC_Client_init in PDC_init

* Change return to 0

This will make is simpler when merging #233 (comment)

---------

Co-authored-by: Jean Luca Bez <[email protected]>

* Change `printf` to PDC logger (#232)

* Changed all printf to use pdc logger

Also removed large blocks of comments and chanegd the pdc logger
to print the file name, function, and line number.

* Change typo of LOG_INFO to LOG_ERROR

* Correct grammar from fail -> failed

* update grammer succesfully close -> successfully closed

* switch type of LOG_INFO to LOG_ERROR

* Add logging docs and fix some LOG_INFO->LOG_JUST_PRINT

* update clang formatting

---------

Co-authored-by: Jean Luca Bez <[email protected]>

* Malloc correct size for pdc_obj_metadata_pkg (#237)

Co-authored-by: Jean Luca Bez <[email protected]>

* PDCregion_transfer_create validate client buf, local region, and remote regions (#236)

Co-authored-by: Jean Luca Bez <[email protected]>

---------

Co-authored-by: Jean Luca Bez <[email protected]>
Co-authored-by: Noah Lewis <[email protected]>

* Fix return metadata dtype (#246)

Co-authored-by: Jean Luca Bez <[email protected]>

* Region info transfer struct type and helper functions (#247)

* Fix cache flush (#226)

* Fix a thread race issue that may cause memory error when larger than cache max size data is transferred

* Add a test that writes more data than server cache size

* Fix CI run command

* checkpoint

* Switch variables such as count_0, start_0, and size0... to arrays

This will reduce code duplication, reduce bugs, and make it easier
to switch to support n-dimnesional data.

* clang format

* checkpoint

* created better function names and documentation

* remove

* Committing clang-format changes

* clang format

* remove file

* change for use helper function

* fix bug with incorrect helper function call

---------

Co-authored-by: Houjun Tang <[email protected]>
Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* Fix issues with PDC tools (#249)

* Fix issues with PDC tools

* Correct LOG_ERROR to LOG_INFO

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* Fix printing in `PGOTO_ERROR` and `PGOTO_ERROR_VOID` (#250)

Print new line by default in `PGOTO_ERROR` and `PGOTO_ERROR_VOID`

Co-authored-by: Jean Luca Bez <[email protected]>

* Group Tests Into Folders (#252)

* Fix cache flush (#226)

* Fix a thread race issue that may cause memory error when larger than cache max size data is transferred

* Add a test that writes more data than server cache size

* Fix CI run command

* Grouped commons tests into folders

This commit also changes the src/tests/CmakeLists.txt to build tests
within their new folders

* add deprecated folder remove buf_map folder

* Update run_multiple_mpi_test.sh

* Update dependencies-macos.sh

* Update dependencies-macos.sh

---------

Co-authored-by: Houjun Tang <[email protected]>
Co-authored-by: Jean Luca Bez <[email protected]>
Co-authored-by: Jean Luca Bez <[email protected]>

* Return the same obj_id if the obj is just created or already opened (#254)

* Return the same obj_id if the obj is just created or already opened

* Committing clang-format changes

* Update doc

* Update dependencies-macos.sh

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* add option to choose interface (#255)

* add option to connect to a given network interface
* Committing clang-format changes
* fix conflict
* include header
* enable output on failure

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Fix multithreading compilation (#259)

* fix multhreading compilation

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* Fix segmentation fault of calling `PDCobj_create_mpi` twice with duplicate object name (#262)

* Validate sucess of PDC_obj_create and PDC_find_id in PDCobj_create_mpi

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* Use `PDC_malloc`, `PDC_free`, `PDC_calloc`, and `PDC_realloc` (#260)

* checkpoint

* replace free with PDC_free and calloc with PDC_calloc

* Committing clang-format changes

* fix more mallocs to PDC_malloc

* more PDC_free fixes

* Committing clang-format changes

* Update ubuntu-cache.yml

* remove eno1

* fix realloc

* Committing clang-format changes

* Update ubuntu-no-cache.yaml

* Fix several bugs with error checking with object dim allocation

* Committing clang-format changes

* fix bug

* Committing clang-format changes

* Update ubuntu-no-cache.yaml

* Update ubuntu-cache.yml

* Set default value of ndim to 1 in PDCprop_create when using PDC_OBJ_CREATE

* Committing clang-format changes

* Malloc when defaulting to ndim size 1.
Only free hostname when we PDC_malloc the memory
because pointers returned by getenv are not malloced
and could point to static memory.

* Committing clang-format changes

* Update README.md

minor change to trigger the pipeline

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>
Co-authored-by: Jean Luca Bez <[email protected]>

* Fix Sphinx documentation errors and warnings (#265)

* Fix all sphinx warnings and errors. Removed repeat declarations of functions.

* Committing clang-format changes

* remove def of EXTENSION_MAPPING

* gitignore for docs and fix c structs

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* Replace `docs/README.md` -> steps to build docs (#268)

* Replace docs/README.md -> steps to build docs

* Update README.md

---------

Co-authored-by: Jean Luca Bez <[email protected]>

* Use `FUNC_ENTER` and `FUNC_LEAVE` (#270)

* use func enter and func leave in all functions

* Committing clang-format changes

* fix infinite recursion between memory managment, hash table, and per function timing

* Committing clang-format changes

* add profiling to CI

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* New test macros and code cleanup (#261)

* checkpoint

* Committing clang-format changes

* some tests

* Committing clang-format changes

* checkpoint

* open_obj uses new test macros

* Committing clang-format changes

* read_obj uses TASSERT

* read_obj uses TASSERT

* Committing clang-format changes

* cont_del and cont_getid use test macros

* convert more tests to use macros

* convert more tests to macros

* Committing clang-format changes

* Committing clang-format changes

* clang format

* use test helper in cont_info and cont_add_del

* more tests use macros

* Committing clang-format changes

* use tests macros in more tests

* use PGOTO* macros instead of goto

* clang format

* more log fixes

* logging cleanup and more usage of test macros

* Committing clang-format changes

* clang format and fix CMakeLists for tests

* use tests macros in transfer overlap 2D/3D

* use TASSERT in more tests

* Committing clang-format changes

* use test asserts

* all tests on the CI use TASSERT

* fix printing and newlines in tests

* print time, file name, function name, and line number in debug print

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Tests logging typo fix (#273)

* Fixed logging typos

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Rename pdc_server.exe to pdc_server for consistency (#275)

Co-authored-by: Jean Luca Bez <[email protected]>

* Update vpicio_mts.c (#276)

Co-authored-by: Jean Luca Bez <[email protected]>

* Client Propogate `HG_Finalize` error on `PDCclose` (#263)

* all but 4 close errors are fixed

* Committing clang-format changes

* client side HG_Finalize now passes on serial tests

* Committing clang-format changes

* cleanup

* Committing clang-format changes

* Update pdc_region_transfer.c

* free bulk handles during region transfer close

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Standardize ID Lookup Null Checks and Error Handling (#281)

* cleanup finding id's

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

* Obj open fix (#279)

* Fix seg fault for PDCobj_open on non-existent object

* Committing clang-format changes

* Remove log from NULL check

* Log message when object metadata isn't found.

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* Fix multithread (#274)

* move hash table mutex to hashtable source filse

* Committing clang-format changes

* add multithread compile test

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

* Fix seg fault when mercury initialization fails (#283)

* check for NULL paramterse in hash table

* Committing clang-format changes

---------

Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Luca Bez <[email protected]>

---------

Co-authored-by: Noah Lewis <[email protected]>
Co-authored-by: Houjun Tang <[email protected]>
Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com>

Author: 4 people

.github/workflows/dependencies-macos.sh

@@ -2,9 +2,7 @@
 
 set -eu -o pipefail
 
-export HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK=FALSE
-
-brew install open-mpi automake
+brew install open-mpi automake m4 libtool
 
 # libfabric
 wget https://github.com/ofiwg/libfabric/archive/refs/tags/v1.15.2.tar.gz

.github/workflows/gitlab-develop.yml

@@ -0,0 +1,28 @@
+name: GitLab
+
+on:
+  push:
+    branches:
+      - develop
+
+jobs:
+  sync-with-gitlab:
+    runs-on: ubuntu-latest
+    timeout-minutes: 2
+    
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+          
+      - name: Setup git
+        run: |
+          git config --global user.name "PDC BOT"
+          git config --global user.email "[email protected]"
+    
+      - name: Update branch
+        run: |
+          git fetch origin develop
+          git remote add gitlab https://${{ secrets.GITLAB_TOKEN_NAME }}:${{ secrets.GITLAB_TOKEN }}@${{ secrets.GITLAB_URL }}
+          git checkout develop
+          git push -f gitlab -u develop

.github/workflows/gitlab-stable.yml

@@ -0,0 +1,28 @@
+name: GitLab
+
+on:
+  push:
+    branches:
+      - stable
+
+jobs:
+  sync-with-gitlab:
+    runs-on: ubuntu-latest
+    timeout-minutes: 2
+    
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+          
+      - name: Setup git
+        run: |
+          git config --global user.name "PDC BOT"
+          git config --global user.email "[email protected]"
+    
+      - name: Update branch
+        run: |
+          git fetch stable
+          git remote add gitlab https://${{ secrets.GITLAB_TOKEN_NAME }}:${{ secrets.GITLAB_TOKEN }}@${{ secrets.GITLAB_URL }}
+          git checkout stable
+          git push -f gitlab -u stable

.github/workflows/macos.yml

@@ -33,4 +33,4 @@ jobs:
           sudo sh -c 'echo "`ipconfig getifaddr en0` PDC" >> /etc/hosts'
           sudo scutil --set HostName PDC
           export HG_TRANSPORT="sockets"
-          ctest -L serial
+          ctest -L serial --output-on-failure

.github/workflows/ubuntu-cache-multithread.yaml

@@ -0,0 +1,27 @@
+name: Ubuntu (multithread)
+
+on:
+  pull_request:
+    branches: [ stable, develop ]
+
+  push:
+    branches: [ stable, develop ]
+
+  workflow_dispatch:
+
+jobs:
+  PDC:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Dependencies
+        run: .github/workflows/dependencies-linux.sh
+
+      - name: Build PDC
+        run: |
+          mkdir build && cd build
+          cmake ../ -DBUILD_MPI_TESTING=ON -DBUILD_SHARED_LIBS=ON -DPDC_SERVER_CACHE=ON -DBUILD_TESTING=ON -DPDC_ENABLE_MPI=ON -DPDC_ENABLE_PROFILING=ON -DPDC_ENABLE_MULTITHREAD=ON -DCMAKE_C_COMPILER=mpicc -DCMAKE_POLICY_VERSION_MINIMUM=3.5
+          make -j2

.github/workflows/ubuntu-cache-profiling.yml

@@ -0,0 +1,31 @@
+name: Ubuntu (profiling)
+
+on:
+  pull_request:
+    branches: [ stable, develop ]
+
+  push:
+    branches: [ stable, develop ]
+
+  workflow_dispatch:
+
+jobs:
+  PDC:
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Dependencies
+        run: .github/workflows/dependencies-linux.sh
+
+      - name: Build PDC
+        run: |
+          mkdir build && cd build
+          cmake ../ -DBUILD_MPI_TESTING=ON -DBUILD_SHARED_LIBS=ON -DPDC_SERVER_CACHE=ON -DBUILD_TESTING=ON -DPDC_ENABLE_MPI=ON -DPDC_ENABLE_PROFILING=ON -DCMAKE_C_COMPILER=mpicc -DCMAKE_POLICY_VERSION_MINIMUM=3.5
+          make -j2
+
+      - name: Test PDC
+        working-directory: build
+        run: ctest -L serial --output-on-failure

.github/workflows/ubuntu-cache.yml

@@ -28,4 +28,4 @@ jobs:
 
       - name: Test PDC
         working-directory: build
-        run: ctest -L serial
+        run: ctest -L serial --output-on-failure

.github/workflows/ubuntu-no-cache.yaml

@@ -28,4 +28,4 @@ jobs:
 
       - name: Test PDC
         working-directory: build
-        run: ctest -L serial
+        run: ctest -L serial --output-on-failure

.gitignore

@@ -11,6 +11,9 @@ src/install
 # ignore vscode files
 .vscode
 
+# ignore intellij files
+.idea
+
 # ignore macos files
 .DS_Store
 
@@ -24,4 +27,4 @@ docs/build
 **/*venv
 
 # ignore .zed
-.zed
+.zed

.gitlab-ci.yml

@@ -457,7 +457,7 @@ perlmutter-metrics:
         PDC_TMPDIR: "${CI_BUILDS_DIR}/${CI_PROJECT_NAME}/${CI_JOB_ID}/pdc-tmp-metrics"
         PDC_DATA_LOC: "${CI_BUILDS_DIR}/${CI_PROJECT_NAME}/${CI_JOB_ID}/pdc-data-metrics"
         PDC_CLIENT_LOOKUP: "NONE"
-        PDC_SERVER: "${PDC_BUILD_PATH}/perlmutter/metrics/bin/pdc_server.exe"
+        PDC_SERVER: "${PDC_BUILD_PATH}/perlmutter/metrics/bin/pdc_server"
         PDC_SERVER_CLOSE: "${PDC_BUILD_PATH}/perlmutter/metrics/bin/close_server"
         PDC_CLIENT: "${PDC_BUILD_PATH}/perlmutter/metrics/bin/vpicio_mts"
         PDC_JOB_OUTPUT: "pdc-metrics.log"