Configuring, compiling and running

Configuring, compiling and running

Prerequisites #

  • MPI (either OpenMPI or Intel-MPI)
  • GCC or Intel Fortran compiler
  • (optional) Parallel HDF5 (compiled with either OpenMPI or Intel-MPI)

On clusters typically all you need to do is to load the specific modules see here. If you are, however, running on a local machine make sure to install the following prerequisites (assuming non-Intel compiler and apt package manager): sudo apt install build-essential libopenmpi-dev (for gcc and openmpi), sudo apt install libhdf5-openmpi-dev hdf5-tools (for hdf5).

Compiling with configure.py + GNU Make #

# to view all configuration options
python3 configure.py -h
# to configure the code (example)
python3 configure.py -mpi08 --user=user_2d_rec -2d
# compile and link (-j compiles in parallel which is much faster)
make -j
# executable will be in the `bin/` directory

List of all the available flags can be found by running python3 configure.py -h.

When changing configure options make sure to make clean before recompiling the code.

Compiling with cmake #

Since v2.8 we also support cmake for the code configuration.

# to configure the code (example)
cmake -B build -D mpi08=ON -D hdf5=ON -D user=user_2d_rec -D dim=2
# compile
cmake --build build -j
# executable will be in the `build/src/` directory (*.xc extension)

To clean-build the code, simply remove the build/ directory and rerun the cmake commands.

Running: #

# run the code (on clusters need to do `srun`, or `irun` etc. depending on the cluster)
mpirun -np <NCORES> ./<EXECUTABLE> -input <INPUTFILE> -output <OUTPUTDIR>

Docker #

Another way to avoid the tedium of installing libraries (especially for local development) is to use Docker containers. This approach allows to quickly create an isolated linux environment with all the necessary packages already preinstalled (similar to a VM, but much lighter). The best thing is that VSCode can natively attach to a running container, and all the development can be done there (make sure to install the appropriate extension).

To get started with this approach, make sure to install the Docker (as well as the Docker-compose) then simply follow these steps.

# from tristan root diretory
cd docker
# launch the container in the background (first-time run might take a few mins)
docker-compose up -d
# if you are attempting to also force rebuild the container, use the `--build` flag
docker-compose up -d --build
# ensure the container is running
docker ps

Then you can attach to the container via VSCode, or if you prefer the terminal, simply attach to the running container by doing:

docker exec -it trv2 zsh
# then the code will be in the `/home/$USER/tristan-v2` directory
cd /home/$USER/tristan-v2

To stop the container run docker-compose stop. To stop and delete the container simply run docker-compose down from the same docker/ directory.

Each container has in principle its own isolated filesystem, aside from the shared /root/tristan-v2 directory. Any changes to the rest of the container’s filesystem are discarded when the container is deleted (either via docker-compose down or directly docker rm <CONTAINER>).

Cluster specific customization #

Code can be configured for a specific cluster saved into the configure.py file. For example, to enable all the Perseus-specific configurations (Princeton University) include the --cluster=perseus flag when calling the python configure.py. This will automatically enable the new MPI version, ifport library, intel compilers and specific vectorization flags.

Most clusters use the so-called slurm scheduling system where jobs are submitted for scheduling using a submit script. Here is a best practice example of such a script:

#!/bin/bash
#SBATCH -J myjobname
#SBATCH -n 64
#SBATCH -t 01:00:00

# specify all the variables
EXECUTABLE=tristan-mp2d
INPUT=input
OUTPUT_DIR=output
SLICE_DIR=slices
RESTART_DIR=restart
REPORT_FILE=report
ERROR_FILE=error

# here you'll need to include cluster specific modules ...
# ... these are the ones used on `perseus`
module load intel-mkl/2019.3/3/64
module load intel/19.0/64/19.0.3.199
module load intel-mpi/intel/2018.3/64
module load hdf5/intel-16.0/intel-mpi/1.8.16

# create the output directory
mkdir $OUTPUT_DIR
# backup the executable and the input file
cp $EXECUTABLE $OUTPUT_DIR
cp $INPUT $OUTPUT_DIR
srun $EXECUTABLE -i $INPUT -o $OUTPUT_DIR -s $SLICE_DIR -r $RESTART_DIR > $OUTPUT_DIR/$REPORT_FILE 2> $OUTPUT_DIR/$ERROR_FILE

Stellar (PU) #

Modules to load:

1) intel-rt/2021.1.2                                                          
2) intel-tbb/2021.1.1
3) intel-mkl/2021.1.1
4) intel-debugger/10.0.0
5) intel-dpl/2021.1.2
6) /opt/intel/oneapi/compiler/2021.1.2/linux/lib/oclfpga/modulefiles/oclfpga
7) intel/2021.1.2
8) ucx/1.9.0
9) intel-mpi/intel/2021.1.1
10) hdf5/intel-2021.1/intel-mpi/1.10.6
11) anaconda3/2020.11

Simply loading the following modules will automatically load all the others:

intel/2021.1.2
intel-mpi/intel/2021.1.1
hdf5/intel-2021.1/intel-mpi/1.10.6
anaconda3/2020.11 # <- used when configuring with python

You can make an alias for simplicity and put in your .zshrc or .bashrc:

alias tristan_modules='module purge; module load intel/2021.1.2 intel-mpi/intel/2021.1.1 hdf5/intel-2021.1/intel-mpi/1.10.6 anaconda3/2020.11'

Frontera #

Modules:

  1) intel/18.0.5   2) impi/18.0.5   3) phdf5/1.10.4   4) python3/3.7.0

Useful alias:

alias tristan_modules="module purge; module load intel/18.0.5; module load impi/18.0.5; module load phdf5/1.10.4; module load python3/3.7.0"

Helios (IAS) #

Before running the code do the following:

# if running on a single node:
export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so

# if running on multiple nodes:
export I_MPI_PMI_LIBRARY=/usr/lib64/libpmi2.so
export UCX_TLS=all