The document discusses containerizing MPI workloads using Docker and QNIBTerminal. It provides an overview of Docker, describes the QNIBTerminal testbed which runs an HPCG benchmark on multiple Linux distributions within Docker containers, and presents results showing a low performance overhead for containerized workloads compared to bare metal. Future work is discussed around optimizing containers for HPC and benchmarking real-world applications.

1. QNIBTerminal plus InfiniBand
Containerized MPI Workloads
Christian Kniep
2014-11-05

2. Agenda
• About Me
• Docker in a Nutshell
• QNIBTerminal
• Study
• Testbed + HPCG
• Results
• Future Work
• Conclusion
2

3. About Me
• 1/2013 -> 10/2014 R&D @Bull SAS
• since then independent R&D / Freelancing
• Hot topics
• Containerization
• GO-Lang
• Log / Performance Management
• HPC Cluster Software Stack / Interconnect
3
• Iteration through L1/2/3 SysOps
• Mostly german automotive sector

4. Docker in a Nutshell
4
• (chroot on steroids)2

5. Docker in a Nutshell
• Builds on-top LinuX Containers (LXC)
• Kernel namespaces (isolation)
• cgroups (resource mgmt)
5
• (chroot on steroids)2

6. Docker in a Nutshell
• intuitive build system
6
• (chroot on steroids)2
• Builds on-top LinuX Containers (LXC)
• Kernel namespaces (isolation)
• cgroups (resource mgmt)

7. Docker in a Nutshell
• intuitive build system
• public repositories
• RedHat backing
7
• (chroot on steroids)2
• Builds on-top LinuX Containers (LXC)
• Kernel namespaces (isolation)
• cgroups (resource mgmt)

8. Traditional vs. Lightweight
Layers
8
SERVICE SERVICE
Userland (OS) Userland (OS) Userland (OS)
KERNEL KERNEL
HYPERVISOR
HOST KERNEL
SERVER
SERVICE
KERNEL
Userland (OS)
SERVICE SERVICE
Userland (OS) Userland (OS) Userland (OS)
HOST KERNEL
SERVER
SERVICE
Userland (OS)
Traditional Virtualisation Containerisation
IB
IB

9. Rocket ‚Docker‘
• 2013-01-18 First commit
• 2013-02-01 First online demo
• 2013-03-21 Demo at PyCon US
• 2013-03-23 Version 0.1
• 2013-03-26 Repository on github.com created
• 2013-04-23 Version 0.2
• 2013-05-06 Version 0.3
• 2013-06-03 Version 0.4
• 2013-06-25 Joining Linux Foundation
• 2013-07-18 Version 0.5 (top, mount)
• 2013-08-23 Version 0.6 (-privileged, LXC conf)
• 2013-09-09 Collaboration with Red Hat
• 2013-10-29 Foundation of Docker Inc.
• 2013-11-26 Version 0.7
9
• 2014-01-21 Docker Inc. raised $15M
• 2014-02-04 Version 0.8 (MacOSX, btrfs exp., ONBUILD)
• 2014-03-10 Version 0.9 (exec driver, libcontainer)
• 2014-06-09 Version 1.0 (PROD, pause, XFS, COPY)
• 2014-06-09 official repos, public repository
• 2014-06-09 enterprise support, training, consulting
• 2014-07-03 Version 1.1 (.dockerignore, mount „/“)
• 2014-08-22 Version 1.2 (-restart, caps , rw /etc/hosts)
• 2014-09-16 Docker Inc. raised $40M
• 2014-09-16 Microsoft teams up with Docker Inc.
• 2014-10-16 Version 1.3 (signed img, proc injection, …)

10. QNIBTerminal
Motivation
10
Plain Metrics

11. QNIBTerminal
Motivation
11
Plain Log Events

12. QNIBTerminal
Motivation
12
Overlap Metrics/Log Events

13. QNIBTerminal
Overview
13
One Node Setup
• All network traffic over bridge
• Crippled MPI workload
elk
dns
elasticsearch
logstash
kibana
helixdns
etcd
haproxy haproxy
Compute
slurmd compute<N>
slurmd compute0
slurmctld slurmctld
grafana grafana
graphite-api graphite-api
graphite-web graphite-web
carbon carbon
Log/Events
Services Performance

14. • 3 containers on top (CentOS 6, CentOS 7, Ubuntu 12)
• SLURM Resource Scheduler
• 1 native partition
• 3 containers partitions
• Multiple Open MPI version installed
• gcc versions
Testbed
14
• 8 nodes (CentOS 7, 2x 4core XEON, 32GB, Mellanox ConnectX-2)

15. HPCG Benchmark
• mimics thermodynamic application workload
• Linpack corrective / successor in the long-term?
15

16. Results
partition’s performance
16
6
5,25
GFLOP/s 3
4,5
3,75
native cos7 cos6 u12
CentOS 7.0
oMPI 1.6.4
gcc 4.8.2

17. Results
partition’s performance
17
6
5,25
GFLOP/s 3
4,5
3,75
native cos7 cos6 u12
CentOS 7.0
oMPI 1.6.4
gcc 4.8.2
CentOS 6.5
oMPI 1.5.4
gcc 4.4.7
Ubuntu12.04
oMPI 1.5.4
gcc 4.6.3

18. Results
multiple MPI versions
6
5,25
s
GFLOP/4,5
3,75
3
distribution
18 native
cos7
cos6
u12
oMPI 1.6.4
oMPI 1.6.4
oMPI 1.5.4
oMPI 1.5.4

19. Results
multiple MPI versions
6
5,25
s
GFLOP/4,5
3,75
3
distribution 1.6.4 1.8.4
19 native
cos7
cos6
u12
oMPI 1.6.4
oMPI 1.6.4
oMPI 1.5.4
oMPI 1.5.4
gcc 4.8.2
gcc 4.8.2
gcc 4.4.7
gcc 4.6.3

20. Results
multiple MPI versions
6
5,25
s
GFLOP/4,5
3,75
3
distribution 1.5.4 1.6.4 1.8.4
20 native
cos7
cos6
u12
oMPI 1.6.4
oMPI 1.6.4
oMPI 1.5.4
oMPI 1.5.4
gcc 4.8.2
gcc 4.8.2
gcc 4.4.7
gcc 4.6.3

21. Future Work
• Compare with tuned bare-metal
• Tune docker installation
• Use of SV-IOR (Keynote earlier today)
• Compare different frameworks to orchestrate
• Security evaluations
21
• Benchmark real-world applications

22. Conclusion
• Abstraction bare-metal / application works fine
• Bare-metal kernel provides access to IB
• Container in charge from MPI upwards
• Out-of-the-box: container beats bare-metal
• Continuous testing/deployment of containerized workloads
22
• Bunch of tooling within docker ecosystem
• Low performance overhead

23. La Fin
• Contact
• @CQnib / @qnibinc
• christian@qnib.org
• http://qnib.org
23
https://www.flickr.com/photos/dharmabum1964/3108162671

24. La Fin
• Contact
• @CQnib / @qnibinc
• christian@qnib.org
• http://qnib.org
• Paper: http://doc.qnib.org/
24
https://www.flickr.com/photos/dharmabum1964/3108162671

25. La Fin
25
https://www.flickr.com/photos/dharmabum1964/3108162671
• Contact
• @CQnib / @qnibinc
• christian@qnib.org
• http://qnib.org
• Paper: http://doc.qnib.org/
• Interested?
• Docker Pitch today
• Internal Evaluations
• Workshops / Talks

26. La Fin
26
https://www.flickr.com/photos/dharmabum1964/3108162671
• Contact
• @CQnib / @qnibinc
• christian@qnib.org
• http://qnib.org
• Paper: http://doc.qnib.org/
• Interested?
• Docker Pitch today
• Internal Evaluations
• Workshops / Talks
• Questions?