KubeCon 2017 session with @JasonMcGee: Do you want to increase the density of your Kubernetes workers? We have run some experiments with Kubernetes workers in LXC containers managed by LXD. Come to hear our experience and challenges during the experiment!

1. Leverage LXC/LXD with Kubernetes
Jason McGee, IBM Fellow and VP, IBM
Lin Sun, Senior Software Engineer, IBM

2. Agenda
• Background
• Why are we looking at this?
• Experiments
• Demo
• Summary

3. Beta available March 20th.
Combining Docker and Kubernetes to deliver powerful tools,
an intuitive user experience, and built-in security and
isolation to enable rapid delivery of applications - all while
leveraging IBM Cloud Services including cognitive
capabilities from Watson.
www.ibm.com/cloud-computing/bluemix/containers
IBM Bluemix Container Service

4. Intelligent Scheduling Automated rollouts and rollbacks Container Security & IsolationDesign Your Own Cluster
Self-healing Horizontal scaling Leverages IBM Cloud & Watson Integrated Operational Tools
S
M
L
Service discovery & load balancing Secret & configuration management Simplified Cluster Management Native Kubernetes Experience
IBM Bluemix Container Service

5. IBM Bluemix | IBM Confidential | ©2017 IBM Corporation
Architecture
• Free tier worker is
deployed in our account
• One free tier worker per
account
• Paid tier workers are
deployed in customer’s
account
• Carrier-Cruiser model
• Hub-Spoke model

6. Free tier of IBM Bluemix Container Service

7. Requirements for free tier
• Each tenant has only 1 kubernetes worker (2 CPU, 4 GB memory)
• Isolation between each tenant
• Fast launch and destroy clusters
• Minimum cost yet providing a lightweight native kubernetes
experience
• Easy migration to paid tier

8. Why are we looking at this?
• Increase density for free tier
• Reduce cost for free tier
• Fast deployment for free tier worker
• Quick tear down for free tier worker
• Many free tier clusters are idle

9. Experiments we explored
• Run kubernetes worker in docker containers
• Run kubernetes in LXC container
• Run kubernetes worker in LXC Container

10. Run Kubernetes worker in Docker containers
• We started with running kubernetes worker in Docker
• It works but requires Docker container in privileged mode

11. Introduction of LXD
• LXD is a container hypervisor and a new user experience for LXC
• Not a rewrite of LXC, led by Canonical, Ltd
• 2 Key components
• A system-wide deamon (lxd)
• A command line client (lxc)
• Docker vs LXD
• Docker specializes in deploying applications
• LXD specializes in deploying (Linux) Virtual Machines

12. Run kubernetes in LXC containers
• Kubernetes (master + worker) in non privileged LXC container
docker profile
• Can’t run Docker privileged container
• Kubernetes processes directly run in LXC
• A few kubernetes containers require privileged access

13. Run kubernetes worker in LXC containers
• Kubernetes worker in
non privileged LXC
container docker
profile
• Kubernetes worker
processes directly run
in LXC
• Easy migration to paid
tier

14. Run kubernetes worker in LXC containers

15. Run kubernetes worker in LXC containers

16. Run kubernetes worker in LXC containers
• Demo!

17. Density with LXC & Kubernetes
• Current Free tier: 2 Core, 4 GB memory
• With our LXC Experiment
• 8 Core, 8GB memory LXD host
• Each LXC with idle k8s worker running: 140MB peak, 100MB average
• Each LXC with k8s worker and guestbook example: 1.5GB peak, 800MB average
• LXC supports hard memory limit by default but allows for soft limit
• Can run 10+ LXC Kubernetes workers, assume 20% workers are highly used while
rest are idle

18. List of Issues we opened
• Privileged Docker containers in LXD: https://github.com/lxc/lxd/
issues/2825
• Skip OOM score adjust in unprivileged containers
• https://github.com/kubernetes/kubernetes/pull/43079
• https://github.com/opencontainers/runc/pull/1386

19. Summary of the experiment
• LXC/LXD provides fast deployment, much higher density thus lower cost
• Easy migration to paid tier
• Wish lists:
• Explore cpu/mem limits options
• Explore copy/snapshot features
• Explore DNS
• Explore Kubernetes keys and certs

20. Thank you!