"Bare-Metal Docker" at Lighting Talk of Meetup Docker Tokyo#11 20170427

1. Bare-Metal Container
1
National Institute of Advanced Industrial
Science and Technology(AIST)
Kuniyasu Suzaki

2. Background of BMC 1/3
Drawback of Container
• Container technology (Docker) becomes popular.
– Docker offers an environment to customize an application easily.
– It looks like to be good for an application, but it is a server centric.
• It does not allow to change the kernel.
– Kernel options passed through /sys are not effective because Docker
uses union file system (AUFS or DeviceMapper), which prevents
options from reaching to the kernel.
• Some applications cannot run on Docker.
– DPDK on Docker does not work on some machines, because it
depends on “igb_uio” and “rte_kni” kernel modules.
• Some provider offers the kernel which can treat DPDK on Docker,
but it is case by case solution. It is not fundamental solution.
2

3. Background of BMC 1/3
Drawback of Container
• Container technology (Docker) becomes popular.
– Docker offers an environment to customize an application easily.
– It looks like to be good for an application, but it is a server centric.
• It does not allow to change the kernel.
– Kernel options passed through /sys are not effective because Docker
uses union file system (AUFS or DeviceMapper), which prevents
options from reaching to the kernel.
• Some applications cannot run on Docker.
– DPDK on Docker does not work on some machines, because it
depends on “igb_uio” and “rte_kni” kernel modules.
• Some provider offers the kernel which can treat DPDK on Docker,
but it is case by case solution. It is not fundamental solution.
3
Container is a jail for a kernel optimizer.

4. Background of BMC 1/3
Drawback of Container
• Container technology (Docker) becomes popular.
– Docker offers an environment to customize an application easily.
– It looks like to be good for an application, but it is a server centric.
• It does not allow to change the kernel.
– Kernel options passed through /sys are not effective because Docker
uses union file system (AUFS or DeviceMapper), which prevents
options from reaching to the kernel.
• Some applications cannot run on Docker.
– DPDK on Docker does not work on some machines, because it
depends on “igb_uio” and “rte_kni” kernel modules.
• Some provider offers the kernel which can treat DPDK on Docker,
but it is case by case solution. It is not fundamental solution.
4
Container is a jail for a kernel optimizer.
HPC users want to optimize the kernel for their
applications. Kernel is a servant.
Container way is not fit for them.

5. Background of BMC 2/3
General kernel leads weak performance
• Arrakis[OSDI’14] showed that nearly 70% of network latency
was spent in the network stack in a Linux kernel.
• Many DB applications (e.g., Oracle, MongoDB) reduce the
performance by THP (Transparent Huge Pages) which is
enabled on most Linux distributions.
5

6. Background of BMC 2/3
General kernel leads weak performance
• Arrakis[OSDI’14] showed that nearly 70% of network latency
was spent in the network stack in a Linux kernel.
• Many DB applications (e.g., Oracle, MongoDB) reduce the
performance by THP (Transparent Huge Pages) which is
enabled on most Linux distributions.
6
It is not fundamental solution.
HPC users want to optimize the kernel for their
applications. Kernel is a servant.

7. Background of BMC 3/3
Power consumption for each application
• Current power measurement is coarse.
– PUE only shows usage of data-center scale.
– Current power consumption is theme for vender and
administrators
• Users have no incentive for low power.
– Current accounting is based on time consumption
7

8. Background of BMC 3/3
Power consumption for each application
• Current power measurement is coarse.
– PUE only shows usage of data-center scale.
– Current power consumption is theme for vender and
administrators
• Users have no incentive for low power.
– Current accounting is based on time consumption
8
There is no good method to measure power
consumption “for an application”.
No accounting which considers power consumption.

9. What is BMC?
• BMC(Bare-Metal Container) runs a container
(Docker) image with a suitable Linux kernel on a
remote physical machine.
– Application on Container can change kernel settings and
machine which fit for application and extract the full
performance.
– It means the power is almost used for an application.
9
Application Centric Architecture

10. machine
kernel
container manager
Server Centric Architecture
Traditional Style
(Ex: container)
Invoke app.
Power always up
Admin’s
Space
User’s
Space
app
container
app
container
app
container
Comparison
Pros:
• Multi Tenant
•Quick Response (No Rebooting)
Cons:
• Kernel is not replaced.
Pros:
• Apps can select a kernel & hardware.
• Apps occupy the machine and extract the
performance.
Cons:
• Set up overhead (Rebooting)
Boot the kernel & app.
BMC
machine machine machine
kernel
app
container
kernel kernel
Application Centric Architecture
Select a kernel
Select a physical
machine
BMC manager
Remote Machine
management
(WOL, AMT, IPMI)
network
bootloader
network
bootloader
network
bootloader
Power frequently up/down
app
container
app
container

11. Current BMC target
• BMC can use Linux kernels for Docker.
–CoreOS
–RancherOS
–Snappy Ubuntu Core
–RedHat Project Atomic
–Mesosphere DCOS
–VMware Photon
11

12. Related works
• vSphere Integrated Containers [VMware’s product]
– Integrate vSphere, Linux, and Docker Engine.
• Triton [Joyent’s product] acquired by Samsung 2016
– Triton = Docker + SmartOS.
• In order to optimize, user needs to customize SmarOS.
• LinuxBIOS/BProc Cluster[HPCS’02]
– Testbed for kernel test. It is not so easy to implement because it
requires to replace BIOS.
• Shifter[Cray User Group’15]
– Translate Docker image for their environment.
– It does not allow to change the kernel.
• SLURM[ICDCN’14]
– Measure power consumption for an application. It depends on function
to measure power (Intel RAPL: Running Average Power Limit, or
CRAY machine).
12

13. • Join Open Source Summit Japan 2017.
– May/30 – June/2, Toyo Conference Center Ariake
– “Bare Metal Container”, 16:50 June/1
– http://events.linuxfoundation.jp/events/open-source-summit-japan
• Official HP: http://www.itri.aist.go.jp/cpc/research/bmc/
• Docker Image for BMC manager:
https://hub.docker.com/r/baremetalcontainer/
• Source Code: https://github.com/baremetalcontainer
• K.Suzaki, H.Koie, and R.Takano, Bare-Metal Container --- Direct
execution of a container image on a remote machine with an optimized
kernel ---, the 18th International Conferences on High Performance
Computing and Communications (HPCC) December/2016
13