Containers have become a transformative technology in HPC, offering significant advantages in portability, reproducibility, performance, isolation, and security. They are rapidly gaining adoption in scientific computing and are poised to play a pivotal role in the future of HPC.
In this training, you’ll learn the workflow Tufts RT used to build, manage, pull and run containers on Tufts HPC.
Topics include:
- What are containers and why should we use them?
- Container registry - Docker Hub.
- How to use GitHub action to build docker containers?
- How to use singularity/apptainer to pull containers?
- How to use
container-modto pull containers, generate modules, and jupyter kernels? - How to build containers using your personal computers?
- To ensure that you're not wasting your time building your own containers, it's recommended to check if there are any publicly available containers that can serve your purpose.
- If you plan to use GPU containers, ensure compatibility between the CUDA version that was used to build the target application inside container and our GPU’s CUDA driver version.
- When running containers that require GPU, make sure to include the
--nvflag. - Remember to use
ncdu, a helpful tool, to regularly check and clean up$HOME/.apptainerand$HOME/.singularitydirectories.
- Load modules
- singularity pull
- singularity shell
- singularity run
- docker build
- github action
- container-mod
| Usage | Name | Link |
|---|---|---|
| Operation system | ubuntu | https://hub.docker.com/_/ubuntu |
| Operation system | rockylinux | https://hub.docker.com/_/rockylinux |
| Python | python | https://hub.docker.com/_/python |
| R/Rstudio | rocker/tidyverse | https://hub.docker.com/r/rocker/tidyverse |
| R/Rstudio | rocker/geospatial | https://hub.docker.com/r/rocker/geospatial |
| R/Rstudio | rocker/cuda | https://hub.docker.com/r/rocker/cuda |
| R/Rstudio | rocker/ml | https://hub.docker.com/r/rocker/ml |
| GPU | nvidia/cuda | https://hub.docker.com/r/nvidia/cuda |
| Data Science | pytorch | https://hub.docker.com/r/pytorch/pytorch |
| Data Science | tensorflow | https://hub.docker.com/r/tensorflow/tensorflow |
| Our organization account | tuftsttsrt | https://hub.docker.com/u/tuftsttsrt |