Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1NF22uW. Ben Corrie discusses Project Bonneville, how to create a shared Linux kernel for privileged containers, running containers without Linux, and VMware's dynamic resource constraints of a container host. Filmed at qconsf.com. Ben Corrie is currently a Principal Investigator at VMware and is the lead inventor of Project Bonneville, which runs Docker containers directly on ESX. Ben has had a 17 year engineering career, principally focussed on solving complex resource management problems and developing optimizations in managed runtimes, such as the JVM.

1. Exploding the
Linux Container Host
Presenter: Ben Corrie (@bensdoings)

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/bonneville

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. 2
Containers vs VMs
• Google Wisdom:
– VMs and Containers are similar but different
– Try running containers in VMs for security
– Containers are best for scale-out density
– VMs are better for legacy apps

5. 3
Docker HubDocker
Client
LINUX HOST
Container
Control & TTY
/ REST
Pull
ImageDocker
Daemon
Docker
Daemon
Layered file
system (AUFS)
S
D
N
Docker
ImagesNET TRAFFIC
TO
CONTAINERS
Linux Kernel
1. A executable process
2. Resource constraints / private namespace
3. Binary dependencies: Application, runtime, OS
4. A shared Linux kernel for running the executable
STATELESS
PORTABLE
FAST
SECURE
What is a Container?

6. 4
What is a Container Host?
Docker
Client
LINUX HOST
LXC
Container
Docker
Daemon Container
Container Layered file
system (AUFS)
S
D
N
Docker
ImagesNET TRAFFIC
TO
CONTAINERS
Linux Kernel
Control & TTY
/ REST
Docker Hub
Pull
Image
1. Control plane & lifecycle management for containers
2. Resource scheduling and a container abstraction
3. Infrastructure abstractions: Storage, networking etc
4. A Linux kernel
STATEFUL
LONG-RUNNING
SINGLE USER
SINGLE USE

7. 5
My Demo Container Hosts
Derek
Clive

8. 6
Docker HubDocker
Client
ESX HOST / HYPERVISOR
ContainerVM
Control & TTY
/ REST
Pull
ImageDocker
Daemon
Docker
Daemon VM
Layered file
system (VMDK)
S
D
N
Docker
ImagesNET TRAFFIC
TO
CONTAINERS
ESX Kernel
1. A executable process
2. Resource constraints / private namespace
3. Binary dependencies: Application, runtime, OS
4. A “shared” Linux kernel for running the executable
STATELESS
PORTABLE
FAST
SECURE
What is a ContainerVM?
Linux Kernel

9. 7
Why?????
• Simple answer: The Container Host
• Linux container host limitations
– Single Docker daemon = single user
– Long running – slow and disruptive to refresh
– Stateful – images, volumes, containers, patch levels
– Static size – only resource efficient if well-packed
– Kernel is a single point of failure
• When virtualized
– Limited access to virtual infrastructure
– Limited monitoring of containers without 3rd party agents
– Duplicated infrastructure layer
STATEFUL
LONG-RUNNING
MULTI USER
MULTI USE

10. 8
Differences between Derek & Clive
1. Multi-tenancy
2. Dynamic resource boundaries
3. Disposable nested container hosts
– Control plane performance
– Statelessness – container hosts as cattle!
– Eg. Docker in Jenkins Slaves
• Dependencies on slaves are contained
• Slaves themselves need to be “garbage collected”
– Eg. Pre-populated container cache for Docker build -> push -> dispose
– Eg. Save /var/lib/docker in a volume – state persists, host does not
4. Multi-OS support

11. The Docker ecosystem you love on the Hypervisor you trust
• Provision Docker containers direct to vSphere
– No need for a Linux container host
– Vanilla Docker client connects to Docker Daemon appliance
• Hardware-virtualized “containerVM” abstraction
– Containers are provisioned as VMs, not in VMs
– Hardware virtualization provides unprecedented security and isolation
– x86 abstraction allows for more than just Linux
• “Instant Clone” delivers container speed and efficiency
– Container start in 2 seconds with a “shared” Linux Kernel
9
What is Bonneville?

12. 10
Limitations Virtualizing Docker As-Is
ESX Host/Cluster
C1 C2
Docker
API + Daemon
Images
Multi-user
guest OS
Tenant wasting ESX memory
when containers stopped
Other App
Tenant at capacity
Duplicated
Image caches
Limited isolation
Only Linux kernels that
support Docker

13. Exploding the Linux Container Host – in detail
ESX Host/Cluster
Other App
Modified Docker
API + Daemon
Multi-user guest OS
Images
C
1
C
2
ESX Host/Cluster
C
1
C
2
Docker
API + Daemon
Images
Multi-user
guest OS
Tenant
wasting ESX
memory when
containers
stopped
Other App
Tenant
at capacity
Duplicated
Image caches
Limited
isolation
Shared
Image Cache
Robust
Isolation
Tenant not
at capacity
From earlier… To this…
Tenant consuming
minimal memory

14. 12
What’s inside? Instant Clone and the “shared” Linux Kernel
ESX Host
Bonneville Appliance
Photon Pico (25MB)
C1 C2 C3
Provisions
Shared
Kernel
Photon Pico Kernel
Container Image A
Container Image B
Container Image C
Read/Write Layer
Volume Volume

15. • Early concerns about efficiency of 1:1 container / VM mapping
• Container efficiency typically measured in terms of start time and memory consumption
• Start Time
– Start time not inherent limitation of VMs, simply the need to boot an OS
– Instant Clone removes the need for OS boot
– Docker appeal more than just container start time – pull image, run image, delete image flow
– Developers want instant container start, less critical when provisioning apps
• Memory consumption
– Misleading “Hello World” comparisons often made. Real apps use memory regardless
– Bonneville memory efficiencies achieved through Instant Clone + Photon Pico
– Instant Clone raises the potential for sharing much more than just the base OS
13
Bonneville Efficiency

16. • Goal for Bonneville is complete transparency to the client / user
• Some concepts have to be a little different
• Container privileged access
– In Docker, flag gives a container privileged access to both the host kernel and the host itself
– In Bonneville, privileged access is the default with zero access to the host
• Host mounted volumes
– In Docker, you can mount a volume on the host into a container
• Useful for certain things, but means that the container is not idempotent
– In Bonneville, the host and container don’t share a filesystem
• Default container size
– In Docker if no constraints are specified, container has access to all the hosts resources
– In Bonneville this wouldn’t make sense, so a default size is used
14
Docker Feature Parity: Can you even tell?

17. • What is a “Container Host”?
– A finite amount of compute resource with the necessary capability to host containers
• A container host does not have to be bound to an OS or physical machine
15
vSphere Integrated Containers: The Virtual Container Host
Concept Linux ESX VCH
Container host
boundaries
A VM or physical box An ESX server A vSphere resource pool
Grow container host Shut down VM / N/A N/A Reconfigure the pool
Clustering Docker Swarm Docker Swarm vSphere cluster
Nested hosts Docker-in-Docker Resource pool / Photon Resource pool / Photon

18. • Various takes on the “containerVM” concept have recently emerged
– “Clear Containers” from Intel
• Similar to Bonneville in concept, but different in execution – more of an OSS POC
• KVM without x86 QEMU layer or BIOS initializes Intel “Clear Linux” very fast
– “Hyper”
• Startup based in China with a very similar concept to Bonneville
• Supports KVM and Xen with a custom Linux kernel. Intended as Container-as-a-Service infrastructure
• Security and Isolation at the heart of these solutions
– Hypervisor hardware isolation is well proven and battle-hardened. Linux kernel exploits keep emerging
– Need to be able to secure and verify provenance of container images
• Bonneville delivers best of all worlds
– Robust security and isolation of a VM
– Full privileged access to a kernel – load kernel modules, loopback mount etc.
16
Isolation and Security

19. • Docker is a platform
• Bonneville is the Docker platform for vSphere
• Bonneville gives you best of both worlds
– Speed, efficiency and workflow of containers
– Security, isolation and flexibility of VMs
• Don’t let your container hosts become pets!
@bensdoings
17
Summary

20. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/
bonneville