A talk presented by Phil Estes & Shaun Murakami, IBM Cloud Open Technologies, at the Barcelona OpenStack Summit on October 25, 2016. This talk covers a new feature that will be available in the Docker 1.13 engine for using the CRIU project to checkpoint and restore container processes on Linux. Phil & Shaun present details of this new capability and then demonstrate a proof-of-concept "live migration" of containers across nova compute hosts.

1. Live Container Migration
Phil Estes, STSM Open Containers, IBM Cloud, @estesp
Shaun Murakami, Senior Engineer, IBM Cloud, @stmuraka

2. Background
• Live migration supported by most hypervisors in OpenStack
• Container use across OpenStack ecosystem growing
MAGNUM, MURANO, KOLLA, KURYR, KUBERNETES
• Why can’t I do live migration of containers?
http://docs.openstack.org/developer/nova/support-matrix.html

3. Why?
• A hypervisor is controlling an emulated (complete) machine
• System suspend/resume is a solved problem
• Hypervisor-based live migration can take advantage of this
• Containers are processes
• Can the Linux kernel “migrate” a process?
• Well, not as easily…
• Until CRIU
• “Checkpoint Restore In Userspace”
• Work began in 2011/2012 in the Linux kernel
• 3.11 kernel + CRIU in late 2013 had working release
• 2016: CRIU 2.0 released

4. CRIU: A Brief Overview
• Proposed and led by Pavel Emelyanov from OpenVZ team
• Key difference from other checkpoint/restore attempts in Linux
was that the C/R work was performed from userspace
• All major kernel changes required were completed by the 3.11 Linux
kernel
• CRIU handles “freezing” all the process state:
• process/thread info+caps+uid/gid info, application memory, open files,
pipes/fifos, UNIX sockets, network sockets [magic!], IPC, timers, signals,
terminals, events/inotify state
See https://criu.org/Main_Page for more information.

5. CRIU & Docker
• The Docker engine is bringing CRIU
support to the Docker API/client UI
• Support has been merged into the
engine as of September 2016
• Will be part of the Docker 1.13 release
• Hard work (and perseverance) by Ross
Boucher
• Support already existed in
libcontainer (and the runc) codebase
• Popular demo from DockerCon SF 2015
migrating Quake game backend
between hosts

6. Docker Implementation Overview
• Docker 1.13 use will require the experimental build to allow for
testing/validation of capabilities and use cases
• Simplest model implemented allows for checkpoint associated
with a specific container; restore re-starts same container at the
named checkpoint
• Live migration not an intended feature. Don’t try this at home!
https://criu.org/Live_migration
# docker checkpoint [OPTIONS] container checkpoint_id
OPTIONS: [--keep-running]
# docker start --checkpoint checkpoint_id [OPTIONS] container

7. Implementation Details
Two potential paths for live migration of container data:
memory
process state
filesystem
contents
memory
process state
filesystem
contents
1. Only in-memory
changes to process
state
2. Filesystem is
untouched (e.g. a
read-only root
filesystem)
1
1. In-memory changes
to process state
AND
2. Changes to root
filesystem
underneath
container process
2

8. In-memory Only Sequence
• Checkpoint the container
• Migrate checkpoint metadata to target
• Pull (or pre-pull) exact image on target
• Create container from same image at target
• Extract checkpoint metadata into new container path
• Start container with docker start --checkpoint
1
memory
process state
filesystem
contents

9. In-memory + Filesystem Sequence
• Checkpoint the container
• Migrate checkpoint metadata to target
• docker export stopped container, migrate to target
(docker import on target to re-create target image)
• Read container metadata from source container
• Create container from new image at target
• Replicate container metadata into new container
• Extract checkpoint metadata into new container path
• Start container with docker start --checkpoint
2
memory
process state
filesystem
contents

10. Other Challenges
> Handling Container Metadata
• docker export -> docker import only addresses filesystem
migration; does not save container runtime metadata (e.g.
docker inspect information)
> Networking across Docker Hosts
• IP conflict with getting same IP assignment across hosts
• Requires load balancer solution fronting migrating services
> Volume Support
• External volume data migration would require a solution that
supported volume drivers/data export (use clustered FS)

11. Demo time!
Demonstrate basic checkpoint restore on a single host1
2
3
In-memory database migration across hosts
Video streaming container migration

12. Demo 1
A counter on the same host with checkpoint -> restore
busybox
HOST A
busybox’
HOST A
# docker run busybox
1
2
3
4
<checkpoint>
# docker start --ch...
<restore>
5
6
7
8
9
10
.
.

13. Demo 2
A tweet producer/consumer application centered around
an in-memory database, redis.
redis
tweet2redis
load balancer
(nginx)
HOST A HOST B
redis’
tweetviewer
(node.js)
HOST C

14. Demo 3
A video streaming back-end migration
Video
Stream
load balancer
(nginx)
HOST A HOST B
Video
Stream’
HOST C
video
client

15. Integration with OpenStack
• Magnum community integration of migration “steps”
• Preconfigured nova instances with criu support/capabilities?
• Normalize networking aspect via Kuryr/overlay capabilities
• Use of SwarmKit/Docker with Swarm mode features for service
discovery (remove requirement on self-managed load balancer)

16. Thank You!