1.
Shifter and Singularity on Blue Waters
Maxim Belkin
June 7, 2018

2.
A simplistic view of a scientific application
My Application
DATA RESULTS

3.
Your Computer
My Application
DATA RESULTS i
Blue Waters
My Application
DATA RESULTS
Received an allocation on Blue Waters!

4.
A more realistic view of a scientific application
My Application
LIBRARY X
DATA RESULTS
EXECUTABLE Y HARDWARE Z
Your Computer

5.
My Application
DATA
LIBRARY A EXECUTABLE B HARDWARE C
RESULTS
Blue Waters
A more realistic view of a scientific application

6.
My Application
LIBRARY X
DATA RESULTS
EXECUTABLE Y HARDWARE C
Software
Containers

7.
https://github.com/NERSC/shifter https://github.com/singularityware/singularityhttps://github.com/docker
DockerShifter Singularity
Not suitable
for HPC
Designed
for HPC
Can be used in
HPC environment

8.
Docker
FROM centos:7
RUN yum update && ... wget …
COPY file.ext inside/container/
RUN ./configure && make && …
WORKDIR /home/user
1) Build: docker build -t repo/name:tag .
2) Upload: docker push repo/name:tag
3) Download: docker pull repo/name:tag
4) Run: docker run repo/name:tag
4a) Execute: docker exec <container> command
Dockerfile

9.
https://github.com/NERSC/shifter https://github.com/singularityware/singularity

10.
Docker
ssh qsub aprun
Login nodes
User
MOM nodes
Compute nodes
-l gres=shifter16

11.
Docker
qsub
Login nodes
MOM nodes
-l gres=shifter16
-v UDI=repo/name:tag
-l gres=shifter16
-v UDI=repo/name:tag
export CRAY_ROOTFS=SHIFTER
aprun -b ... --
app-in-container --options
Compute nodes
application
application
application
application
application
application
application
application
UDI: User-Defined Image

12.
Docker
qsub
Login nodes
MOM nodes
-l gres=shifter16
-l gres=shifter16
module load shifter
aprun -b ... --
shifter --image=repo/name:tag --
app-in-container --options
Compute nodes
application
application
application
application
application
application
application
application

13.
Shifter module
shifter
shifterimg
images
lookup
pull
login
returns info about available UDIs
returns an ID of an image (if available)
launches an app from UDI on compute nodes
downloads Docker image and converts it to UDI
authenticates to DockerHub (for private images)

14.
Examples
shifterimg images
shifterimg lookup ubuntu:zesty
shifterimg pull centos:latest
shifterimg login
shifterimg pull --user bw_user prvimg1:latest
shifterimg pull --group bw_grp prvimg2:latest
image on DockerHub
image on Blue Waters

15.
My Application
LIBRARY X
DATA RESULTS
EXECUTABLE Y HARDWARE Z

16.
MPI in Shifter: requirements
1. glibc 2.17+ (CentOS 7, Ubuntu 14)
2. Use one of the following MPIs (see MPICH-ABI compatibility)
3. Do not use package manager to install MPI*
In a Container:
• MPICH v3.1+
• Intel® MPI Library v5.0+
• Cray MPT v7.0.0+
• MVAPICH2 2.0+
• Parastation MPI 5.1.7-1+
• IBM MPI v2.1+

17.
MPI in Shifter: requirements
On Blue Waters:
1. Use Intel or GNU Programming Environment
2. Load MPICH ABI-compatible MPI (on BW: cray-mpich-abi)
3. Set LD_LIBRARY_PATH to the location of MPICH-ABI libraries

18.
MPI in Shifter: in Dockerfile
FROM centos:7
RUN yum -y install file gcc make gcc-gfortran gcc-c++ wget curl
RUN cd /usr/local/src/ &&
wget http://www.mpich.org/static/downloads/3.2/mpich-3.2.tar.gz &&
tar xf mpich-3.2.tar.gz &&
rm mpich-3.2.tar.gz &&
cd mpich-3.2 &&
./configure &&
make && make install &&
cd /usr/local/src &&
rm -rf mpich-3.2*

19.
module unload PrgEnv-cray
module unload cce
module load PrgEnv-gnu
module unload cray-mpich
module load cray-mpich-abi
LIBS=$CRAY_LD_LIBRARY_PATH:/opt/cray/wlm_detect/default/lib64
CACHE=$PWD/cache.$PBS_JOBID
mkdir -p $CACHE
for dir in $( echo $LIBS | tr ":" " " ); do cp -L -r $dir $CACHE; done
export LD_LIBRARY_PATH=$CACHE/lib:$CACHE/lib64
19
1
2
3
4
5
6
7
8
9
10
11
12
MPI in Shifter: job batch script snippet

20.
qsub -l gres=shifter16 -l nodes=2:ppn=32:xe
module load shifter
aprun -b -n 64 -N 32 -- shifter --image=repo/image:tag -- app
aprun -b -n 64 -N 32 -- shifter --image=repo/image2:tag -- app2
20
MPI in Shifter
qsub -l gres=shifter16 -v UDI=repo/image:tag -l nodes=2:ppn=32:xe
export CRAY_ROOTFS=SHIFTER
aprun -b -n 64 -N 32 -- app
1
2

21.
21
MPI in Shifter
2 4 8 16 32 64 128 256 512 1024 2048 4096
Nodes
2
4
8
16
32
64
128
256
Jobstart-uptime,s
1.7 GB
CLI
36 MB
prologue
ppn = 1
1 2 4 8 16
processes per node (ppn)
4
8
16
32
64
Jobstart-uptime,s
1.7 GB
CLI
36 MB
prologue
80 nodes
a
b
T(N, ppn=1)
T(N=80, ppn)
1
2
DOI: 10.1145/3219104.3219145
Hon Wai Leong

22.
22
Performance of MPI applications in Shifter
DOI: 10.1145/3219104.3219145
Time,µs
Message size, bytes
107
106
105
104
103
102
10 106
105
104
103
102
1
a CLE
MPI_Alltoall
MPI_Alltoallv
Shifter
0
1
2
3
4
0 200 400 600 800 1000 1200
b
Time,ms
Message size, kilobytes
CLE
MPI_Bcast
MPI_Reduce
Shifter
7.5
8.0
8.5
IORRead,GB/s
Benchmark number
0 10 20 30
2.5
3.0
3.5
IORWrite,GB/s
Cray Linux Environment
Shifter
Cray Linux Environment
Shifter
Galen Arnold

23.
23
Using GPUs in Shifter
1. Use Shifter CLI:
shifter --image ...
2. Set CUDA_VISIBLE_DEVICES:
export CUDA_VISIBLE_DEVICES=0
aprun -b ... -- shifter --image=centos:latest -- nvidia-smi

24.
24
Using MPI + GPUs in Shifter
Caveat: shifter command removes LD_LIBRARY_PATH
Solution: use a script to pass LD_LIBRARY_PATH
aprun -b -n ... -N 1 -e VAR="$LD_LIBRARY_PATH” ...
shifter --image=repo/name:tag ... -- script.sh
script.sh:
export LD_LIBRARY_PATH=“$VAR”
application

25.
Directory mapping between BW and Shifter UDI
shifter ... --volume=/path/on/BW:/path/in/UDI
Important: /path/in/UDI must exist in UDI
qsub -v UDI="centos:latest -v /u/sciteam/$USER:/home"
/projects, /scratch, and “Home” folders are mapped between Blue Waters and UDI!

26.
https://github.com/NERSC/shifter https://github.com/singularityware/singularity

27.
Docker
ssh qsub aprun
Login nodes
User
MOM nodes
Compute nodes
-l gres=singularity

28.
Docker
qsub
Login nodes
MOM nodes
-l gres=singularity
-l gres=singularity
module load singularity
singularity build IMAGE.simg docker://centos:latest
singularity exec -H /u:/home IMAGE.simg -- app
Compute nodes
application
application
application
application
application
application
application
application

29.
Directory mapping between BW and Singularity
singularity exec -B /path/on/BW:/path/in/singularity
And... /path/in/singularity does not have to exist.
/projects on BW is /mnt/abc/projects in SingularityNote: /projects on BW is /mnt/abc/projects in Singularity
/scratch on BW is /mnt/abc/scratch in Singularity

30.
GPUs in Singularity
singularity exec --nv ...
If you’re using BW directories, make sure they are mapped into Singularity image
Example:
aprun -n 16 -N 8 -b -- singularity exec -B
/dsl/opt/cray/nvidia/default/bin:/home/staff/mbelkin
/home/staff/mbelkin/nvidia-smi

31.
38
Thank you!