COMPSs/PyCOMPSs Framework

Website • Documentation • Releases • Slack • ✉

COMP Superscalar (COMPSs) is a programming model which aims to ease the development of applications for distributed infrastructures, such as Clusters, Grids and Clouds. COMP Superscalar also features a runtime system that exploits the inherent parallelism of applications at execution time.

Repository Structure

• builders: Packages, scripts for local installations, scripts for supercomputers installation and package building scripts
• compss : Programming Model, Bindings and Runtime source code
• dependencies : Embedded dependencies
• files : Dependency files (i.e. PARAVER configurations)
• tests : Integration tests
• utils : Misc utils (i.e. OVA scripts, Docker generation, Storage implementations)

Supported Systems

COMPSs/PyCOMPSs fully supports Linux systems for amd64, ppc64, arm64 and riscv64 architectures. macOS systems are also supported with some limitations.

Building From Sources

Follow the next steps to build COMPSs in your current machine.

1. Install dependencies

For an updated list of COMPSs dependencies and how to install them for different systems visit the dependencies section of the COMPSs documentation website.

2. Get GIT submodules

Before installing COMPSs you need to download the git submodules that contain its dependencies. To do that execute the following two commands at the root of the repository.

./submodules_get.sh

3. Build COMPSs

Note: Remember to install the COMPSs dependencies and to get the GIT submodules before trying to build COMPSs from sources.

• Building COMPSs for all users (not supported in macOS)

cd builders/
INSTALL_DIR=/opt/COMPSs/
sudo -E ./buildlocal [options] ${INSTALL_DIR}

• Building COMPSs for current user

cd builders/

INSTALL_DIR=$HOME/opt/COMPSs/
./buildlocal [options] ${INSTALL_DIR}

For macOS visit the Building from sources section of the COMPSs documentation website.

Many COMPSs modules can be activated/deactivated during the build using different options in the buildlocal command. You may check the available options by running the following command:

cd builders
./buildlocal -h

Running docker tests

1. Install Docker and docker-py

Follow these instructions

• Docker for Mac. Or, if you prefer to use Homebrew.
• Docker for Ubuntu.
• Docker for Arch Linux.

Add user to docker group to run docker as non-root user.

• Instructions.

2. Build the docker image

Run the following command at the root of the project to build the image that will be used for testing. The command create an image named compss and install the current branch into the image.

docker build --target=ci -t compss .

3. Run the tests

To run the tests inside the docker image use the ./tests/scripts/docker_test script. For example, you can run the first test without retrials as follows:

./tests/scripts/docker_test -R -t 1

Tip: for viewing all options use: ./tests/scripts/docker_test -h.

The docker main command creates a new docker container, copies the current framework inside it and runs its tests.

Note: you can change the COMPSs image used and the name of the container setting the following environment variables (showing defaults):

export COMPSS_IMAGE=compss
export COMPSS_CONTAINER=compss_test

Please be aware that:

• Code changes affecting the tests sources, config files (e.g. local.cfg, and scripts (like ./local) will be visible inside the newly created container.
• Code changes affecting the installation will not be visible in the installation because framework is not reinstalled. To do that rebuild the docker image as explained in step 2.
• If you run the command once, the container will be available for manual inspection (such as logs). You can log into in issuing docker exec --user jenkins -it compss_test bash and use the CLI as usual.
• You must remove the container before running again docker tests: docker rm -fv compss_test.
• To delete the created image issue docker rmi compss

4. Run the tests locally on macOS

In order to run the tests locally on macOS, GNU sed (gsed) is needed. To install it, use:

brew install gsed

This will install gsed at /opt/homebrew/bin/gsed.

Finally, the NIO_mac.cfg file needs to be updated with any specific features of the local macOS environment, commonly variables such as the java_home, compss_home and runcompss_opts parameters should be updated.

Support

For support please send and e-mail to [email protected]

(c) Workflows and Distributed Computing Group (WDC) - Department of Computer Science (CS) - Barcelona Supercomputing Center (BSC)