A Scientist's Guide to Cloud-HPC: Example with AWS ParallelCluster, Slurm, Spack, and WRF

3.1 A minimum config file for production HPC

The cluster infrastructure is fully specified by ~/.parallelcluster/config. A minimum, recommended config file would look like:

[aws]
aws_region_name = xxx

[cluster your-cluster-section-name]
key_name = xxx
base_os = centos7
master_instance_type = c5n.large
compute_instance_type = c5n.18xlarge
cluster_type = spot
initial_queue_size = 2
scheduler = slurm
placement_group = DYNAMIC
vpc_settings = your-vpc-section-name
ebs_settings = your-ebs-section-name

[vpc your-vpc-section-name]
vpc_id = vpc-xxxxxxxx
master_subnet_id = subnet-xxxxxxxx

[ebs your-ebs-section-name]
shared_dir = shared
volume_type = st1
volume_size = 500

[global]
cluster_template = your-cluster-section-name
update_check = true
sanity_check = true

A brief comment on what are set:

• aws_region_name should be set at initial pcluster configure. I use us-east-1.

• key_name is your EC2 key-pair name, for ssh to master instance.

• base_os = centos7 should be a good choice for HPC, because CentOS is particularly tolerant of legacy HPC code. Some code that doesn't build on Ubuntu can actually pass on CentOS. Without build problems, any OS choice should be fine -- you shouldn't observe visible performance difference across different OS, as long as the compilers are the same.

• Use the biggest compute node c5n.18xlarge to minimize communication. Master node is less critical for performance and is totally up to you.

• cluster_type = spot will save you a lot of money by using spot instances for compute nodes.

• initial_queue_size = 2 spins up two compute nodes at initial launch. This is default but worth emphasizing. Sometimes there is not enough compute capacity in a zone, and with initial_queue_size = 0 you won't be able to detect that at cluster creation.

• Set scheduler = slurm as we are going to use it in later sections.

• placement_group = DYNAMIC creates a placement group 5 on the fly so you don't need to create one yourself. Simply put, a cluster placement group improves inter-node connection.

• vpc_id and master_subnet_id should be set at initial pcluster configure. Because a subnet id is tied to an avail zone 6, the subnet option implicitly specifies which avail zone your cluster will be launched into. You may want to change it because the spot pricing and capacity vary across avail zones.

• volume_type = st1 specifies throughput-optimized HDD 7 as shared disk. The minimum size is 500 GB. It will be mounted to a directory /shared (which is also default) and will be visible to all nodes.

• cluster_template allows you to put multiple cluster configurations in a single config file and easily switch between them.

Credential information like aws_access_key_id can be omitted, as it will default to awscli credentials stored in ~/.aws/credentials.

The full list of parameters are available in the official docs 8. Other useful parameters you may consider changing are:

• Set placement = cluster to also put your master node in the placement group.

• Specify s3_read_write_resource so you can access that S3 bucket without configuring AWS credentials on the cluster. Useful for archiving data.

• Increase master_root_volume_size and compute_root_volume_size, if your code involves heavy local disk I/O.

• max_queue_size and maintain_initial_size are less critical as they can be easily changed later.

I have omitted the FSx section, which is left to the next post.

One last thing: Many HPC code runs faster with hyperthreading disabled 9. To achieve this at launch, you can write a custom script and execute it via the post_install option in pcluster's config file. This is a bit involved though. Hopefully there can be a simple option in future versions of pcluster. (Update in Nov 2019: AWS ParallelCluster v2.5.0 release adds a new option disable_hyperthreading = true that greatly simplify things!)

With the config file in place, run pcluster create your-cluster-name to launch a cluster.

3.2 What are actually deployed

(This part is not required for first-time users. It just helps understanding.)

AWS ParallelCluster (or other third-party cluster tools) glues many AWS services together. While not required, a bit more understanding of the underlying components would be helpful -- especially when debugging and customizing things.

The official doc provides a conceptual overview 10. Here I give a more hands-on introduction by actually walking through the AWS console. When a cluster is running, you will see the following components in the console:

• CloudFormation Stack. Displayed under "Services" - "CloudFormation". This is the top-level framework that controls the rest. You shouldn't need to touch it, but its output can be useful for debugging.

The rest of services are all displayed under the main EC2 console.

• EC2 Placement Group. It is created automatically because of the line placement_group = DYNAMIC in the config file.

• EC2 Instances. Here, there are one master node and two compute nodes running, as specified by the config file. You can directly ssh to the master node, but the compute nodes are only accessible from the master node, not from the Internet.

• EC2 Auto Scaling Group. Your compute instances belong to an Auto Scaling group 11, which can quickly adjust the number of instances with minimum human operation. The number under the "Instances" column shows the current number of compute nodes; the "Desired" column shows the target number of nodes, and this number can be adjusted automatically by the Slurm scheduler; the "Min" column specifies the lower bound of nodes, which cannot be changed by the scheduler; the "Max" column corresponds to max_queue_size in the config file. You can manually change the number of compute nodes here (more on this later).

The launch event is recored in the "Activity History"; if a node fails to launch, the error message will go here.

• EC2 Launch Template. It specifies the EC2 instance configuration (like instance type and AMI) for the above Auto Scaling Group.

• EC2 Spot Request. With cluster_type = spot, each compute node is associated with a spot request.

• EBS Volume. You will see 3 kinds of volumes. A standalone volume specified in the ebs section, a volume for master node, and a few volumes for compute nodes.

• Auxiliary Services. They are not directly related to the computation, but help gluing the major computing services together. For example, the cluster uses DynamoDB (Amazon's noSQL database) for storing some metadata. The cluster also relies on Amazon SNS and SQS for interaction between the Slurm scheduler and the AutoScaling group. We will see this in action later.

Imagine the workload involved if you launch all the above resources by hand and glue them together. Fortunately, as a user, there is no need to implement those from scratch. But it is good to know a bit about the underlying components.

In most cases, you should not manually modify those individual resources. For example, if you terminate a compute instance, a new one will be automatically launched to match the current autoscaling requirement. Let the high-level pcluster command handle the cluster operation. Some exceptions will be mentioned in the "tricks" section later.

4.1 Using Slurm

After login to the master node with pcluster ssh, you will use Slurm to interact with compute nodes. Here I summarize commonly-used commands. For general reference, see Slurm's documentation: https://www.schedmd.com/.

Slurm is pre-installed at /opt/slurm/ :

$ which sinfo
/opt/slurm/bin/sinfo

Check compute node status:

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      2   idle ip-172-31-3-187,ip-172-31-7-245

The 172-31-xxx-xxx is the Private IP 12 of the compute instances. The address range falls in your AWS VPC subnet. On EC2, hostname prints the private IP:

$ hostname  # private ip of master node
ip-172-31-7-214

To execute commands on compute nodes, use srun:

$ srun -N 2 -n 2 hostname  # private ip of compute nodes
ip-172-31-3-187
ip-172-31-7-245

The printed IP should match the output of sinfo.

You can go to a compute node with the standard Slurm command:

$ srun -N 1 -n 72 --pty bash  # Slurm thinks a c5n.18xlarge node has 72 cores due to hyperthreading
$ sinfo  # one node is fully allocated
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      1  alloc ip-172-31-3-187
compute*     up   infinite      1   idle ip-172-31-7-245

Or simply via ssh:

$ ssh ip-172-31-3-187
$ sinfo  # still idle
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
compute*     up   infinite      2   idle ip-172-31-3-187,ip-172-31-7-245

In this case, the scheduler is not aware of such activity.

The $HOME directory is exported to all nodes via NFS by default, so you can still see the same files from compute nodes. However, system directories like /usr are specific to each node. Software libraries should generally be installed to a shared disk, otherwise they will not be accessible from compute nodes.

4.2 Check system & hardware

A natural thing is to check CPU info with lscpu and file system structure with df -h. Do this on both master and compute nodes to see the differences.

A serious HPC user should also check the network interface:

$ ifconfig  # display network interface names and details
ens5: ...

lo: ...

Here, the ens5 section is the network interface for inter-node commnunication. Its driver should be ena:

$ ethtool -i ens5
driver: ena
version: 1.5.0K

This means that "Enhanced Networking" is enabled 13. This should be the default on most modern AMIs, so you shouldn't need to change anything.

4.3 Cluster management tricks

AWS ParallelCluster is able to auto-scale 14, meaning that new compute nodes will be launched automatically when there are pending jobs in Slurm's queue, and idle nodes will be terminated automatically.

While this generally works fine, such automatic update takes a while and feels a bit black-boxy. A more straightforward & transparent way is to modify the autoscaling group directly in the console. Right-click on your AutoScaling Group, and select "Edit":

• Modifying "Desired Capacity" will immediately cause the cluster to adjust to that size. Either to request more nodes or to kill redundant nodes.

• Increase "Min" to match "Desired Capacity" if you want the compute nodes to keep running even if they are idle. Or keep "Min" as zero, so idle nodes will be killed after some time period (a few minutes, roughly match the "Default Cooldown" section in the Auto Scaling Group).

• "Max" must be at least the same as "Desired Capacity". This is the hard-limit that the scheduler cannot violate.

After compute nodes are launched or killed, Slurm should be aware of such change in ~1 minute. Check it with sinfo.

To further change the type (not just the number) of the compute nodes, you can modify the config file, and run pcluster update your-cluster-name 15.

5.1 MPI libraries (OpenMPI with Slurm support)

Spack can install many MPI implementations, for example:

$ spack info mpich
$ spack info mvapich2
$ spack info openmpi

In this example I will use OpenMPI. It has a super-informative documentation at https://www.open-mpi.org/faq/

$ spack install openmpi  # not what we will use here

$ spack install openmpi@3.1.3  # not what we will use here

$ which sinfo  # comes with AWS ParallelCluster
/opt/slurm/bin/sinfo
$ sinfo -V
slurm 16.05.3

packages:
  slurm:
    paths:
      slurm@16.05.3: /opt/slurm/
    buildable: False

$ spack install openmpi+pmi schedulers=slurm  # use this

$ spack find -p openmpi

$ export PATH=$(spack location -i openmpi)/bin:$PATH

$ ompi_info --param btl all
  MCA btl: self (MCA v2.1.0, API v3.0.0, Component v3.1.3)
  MCA btl: tcp (MCA v2.1.0, API v3.0.0, Component v3.1.3)
  MCA btl: vader (MCA v2.1.0, API v3.0.0, Component v3.1.3)
  ...

#include <mpi.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
    int rank, size;
    char hostname[32];
    MPI_Init(&argc, &argv);

    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    gethostname(hostname, 31);

    printf("I am %d of %d, on host %s\n", rank, size, hostname);

    MPI_Finalize();
    return 0;
}

$ mpicc -o hello_mpi.x hello_mpi.c
$ mpirun -np 1 ./hello_mpi.x  # runs on master node
I am 0 of 1, on host ip-172-31-7-214

$ mpirun -np 2 --host ip-172-31-5-150,ip-172-31-14-243 ./hello_mpi.x
I am 0 of 2, on host ip-172-31-5-150
I am 1 of 2, on host ip-172-31-14-243

$ srun -N 2 --ntasks-per-node 2 ./hello_mpi.x
I am 1 of 4, on host ip-172-31-5-150
I am 0 of 4, on host ip-172-31-5-150
I am 3 of 4, on host ip-172-31-14-243
I am 2 of 4, on host ip-172-31-14-243

5.2 HDF5 and NetCDF libraries

HDF5 and NetCDF are very common I/O libraries for HPC, widely used in Earth science and many other fields.

In principle, installing HDF5 is simply:

$ spack install hdf5  # not what we will use here

Many HPC code (like WRF) needs the full HDF5 suite (use spack info to check all the variants):

$ spack install hdf5+fortran+hl  # not what we will use here

Further specify MPI dependencies:

$ spack install hdf5+fortran+hl ^openmpi+pmi schedulers=slurm  # use this

Similarly, for NetCDF C & Fortran, in principle it is simply:

$ spack install netcdf-fortran  # not what we will use here

To specify the full dependency, we end up having:

$ spack install netcdf-fortran ^hdf5+fortran+hl ^openmpi+pmi schedulers=slurm  # use this

5.3 Further reading on advanced package management

For HPC development you generally need to test many combinations of libraries. To better organize multiple environments, check out:

• spack env and spack.yaml at: https://spack.readthedocs.io/en/latest/tutorial_environments.html. For Python users, this is like virtualenv or conda env.

• Integration with module at: https://spack.readthedocs.io/en/latest/tutorial_modules.html. This should be a familiar utility for existing HPC users.

5.4 Note on reusing software installation

For a single EC2 instance, it is easy to save the environment - create an AMI, or just build a Docker image. Things get quite cumbersome with a multi-node cluster environment. From official docs, "Building a custom AMI is not the recommended approach for customizing AWS ParallelCluster." 23.

Fortunately, Spack installs everything to a single, non-root directory (similar to Anaconda), so you can simply tar-ball the entire directory and then upload to S3 or other persistent storage:

$ spack clean --all  # clean all kinds of caches
$ tar zcvf spack.tar.gz spack  # compression
$ aws s3 mb [your-bucket-name]  # create a new bucket. might need to configure AWS credentials for permission
$ aws s3 cp spack.tar.gz s3://[your-bucket-name]/   # upload to S3 bucket

Also remember to save (and later recover) your custom settings in ~/.spack/packages.yaml, ~/.spack/linux/compilers.yaml and .bashrc.

Then you can safely delete the cluster. For the next time, simply pull the tar-ball from S3 and decompress it. The environment would look exactly the same as the last time. You should use the same base_os to minimize binary-compatibility errors.

A minor issue is regarding dynamic linking. When re-creating the cluster environment, make sure that the spack/ directory is located at the same location where the package was installed last time. For example, if it was at /shared/spack/, then the new location should also be exactly /shared/spack/.

The underlying reason is that Spack uses RPATH for library dependencies, to avoid messing around $LD_LIBRARY_PATH 24. Simply put, it hard-codes the dependencies into the binary. You can check the hard-coded paths by, for example:

$ readelf -d $(spack location -i openmpi)/bin/mpicc | grep RPATH

If the new shared EBS volume is mounted to a new location like /shared_new, a quick-and-dirty fix would be:

$ sudo ln -s /shared_new /shared/

5.5 Special note on Intel compilers

Although I'd like to stick with open-source software, sometimes there is a solid reason to use proprietary ones like the Intel compiler -- WRF being a well-known example that runs much faster with ifort than with gfortran 32. Note that Intel is generous enough to provide student licenses for free.

Although Spack can install Intel compilers by itself, a more robust approach is to install it externally and add as an external package 25. Intel has a dedicated guide for installation on EC2 26 so I won't repeat the steps here.

Once you have a working icc/ifort in $PATH, just running spack compiler add should discover the new compilers 27.

Then, you should also add something like

extra_rpaths: ['/shared/intel/lib/intel64']

to ~/.spack/linux/compilers.yaml under the Intel compiler section. Otherwise you will see interesting linking errors when later building libraries with the Intel compiler 28.

After those are all set, simple add %intel to all spack install commands to build new libraries with Intel.

6.1 Environment setup

Add those to your ~/.bashrc (adapted from the WRF compile tutorial):

# Let WRF discover necessary executables
export PATH=$(spack location -i gcc)/bin:$PATH  # only needed if you installed a new gcc
export PATH=$(spack location -i openmpi)/bin:$PATH
export PATH=$(spack location -i netcdf)/bin:$PATH
export PATH=$(spack location -i netcdf-fortran)/bin:$PATH

# Environment variables required by WRF
export HDF5=$(spack location -i hdf5)
export NETCDF=$(spack location -i netcdf-fortran)

# run-time linking
export LD_LIBRARY_PATH=$HDF5/lib:$NETCDF/lib:$LD_LIBRARY_PATH

# this prevents segmentation fault when running the model
ulimit -s unlimited

# WRF-specific settings
export WRF_EM_CORE=1
export WRFIO_NCD_NO_LARGE_FILE_SUPPORT=0

WRF also requires NetCDF-C and NetCDF-Fortran to be located in the same directory 29. A quick-and-dirty fix is to copy NetCDF-C libraries and headers to NetCDF-Fortran's directory:

$ NETCDF_C=$(spack location -i netcdf)
$ ln -sf $NETCDF_C/include/*  $NETCDF/include/
$ ln -sf $NETCDF_C/lib/*  $NETCDF/lib/

6.2 Compile WRF

$ cd WRF-4.0.3
$ ./configure

• For the first question, select 34, which uses GNU compilers and pure MPI ("dmpar" -- Distributed Memory PARallelization).

• For the second question, select 1, which uses basic nesting.

You should get this successful message:

(omitting many lines...)
------------------------------------------------------------------------
Settings listed above are written to configure.wrf.
If you wish to change settings, please edit that file.
If you wish to change the default options, edit the file:
    arch/configure.defaults


Testing for NetCDF, C and Fortran compiler

This installation of NetCDF is 64-bit
                C compiler is 64-bit
        Fortran compiler is 64-bit
            It will build in 64-bit

*****************************************************************************
This build of WRF will use NETCDF4 with HDF5 compression
*****************************************************************************

To fix a minor issue regarding WRF + GNU + OpenMPI 30, modify the generated configure.wrf so that:

DM_CC = mpicc -DMPI2_SUPPORT

Then build the WRF executable for the commonly used em_real case:

$ ./compile em_real 2>&1 | tee wrf_compile.log

You might also use a bigger master node (or go to a compute node) and add something like -j 8 for parallel build.

It should finally succeed:

(omitting many lines...)
==========================================================================
build started:   Mon Mar  4 01:32:52 UTC 2019
build completed: Mon Mar 4 01:41:36 UTC 2019

--->                  Executables successfully built                  <---

-rwxrwxr-x 1 centos centos 41979152 Mar  4 01:41 main/ndown.exe
-rwxrwxr-x 1 centos centos 41852072 Mar  4 01:41 main/real.exe
-rwxrwxr-x 1 centos centos 41381488 Mar  4 01:41 main/tc.exe
-rwxrwxr-x 1 centos centos 45549368 Mar  4 01:41 main/wrf.exe

==========================================================================

Now you have the WRF executables. This is a good first step, considering that so many people are stuck at simply getting the code compiled 31. Actually using WRF for research or operational purposes requires a lot more steps and domain expertise, which is way beyond this guide. You will also need to build the WRF Preprocessing System (WPS), obtain the geographical data and the boundary/initial conditions for your specific problem, choose the proper model parameters and numerical schemes, and interpret the model output in a scientific way.

In the future, you might be able to install WRF with one-click by Spack 33. For WRF specifically, you might also be interested in EasyBuild for one-click install. A fun fact is that Spack can also install EasyBuild (see spack info easybuild), despite their similar purposes.

That's the end of this guide, which I believe has covered the common patterns for cloud-HPC.