This document provides an overview of containers, orchestration, and security as it relates to deploying container applications in production using Kubernetes. It discusses what Kubernetes is and its key design elements. It then outlines the reference layers needed for Kubernetes cluster operations including prerequisites, control services, worker nodes, cluster add-ons, and user applications. Finally, it discusses some of the challenges of operating Kubernetes in production including networking complexity, ensuring high availability, and integrating security.

1. Containers,
Orchestration and
Security, oh my!
A Journey to Container Apps in Production

2. Rob Hirschfeld (aka Zehicle online)
In Community: OpenStack Board Member (4 years)
Co-Chair of Kubernetes Cluster Ops SIG
Founder of Digital Rebar & Crowbar Projects
Professional: CEO of RackN - hybrid automation software
Executive at Dell - scale data center ops
Cloud Data Center Ops going back to 1999

3. What is Kubernetes?
Container Orchestration / Container Scheduler
API driven to provide restart, placement, network routing and life-cycle
For Applications designed for Kubernetes
Key Design Elements: Immutable Infrastructure (stateless ops)
12 Factor Configuration
Service Oriented

4. Reference Layers for K8s Cluster Ops
Ready State
0
Ready
Prerequisites
1
Prereq
Cluster API &
Control Services
2
Control
Worker Nodes
3
Nodes
Cluster Add-ons
4
Add-Ons
User Applications
5
Apps
Watcher
DNS (if not layer 1)
Kubernetes Dashboard
Heapster, Logs, etc
Container Service (e.g. Docker)
Kubelet
Proxy
Ancillary: SDN, Log, Security, etc
API
Scheduler & Controller Mmgr
For static pod approach: Kubelet
Cluster database (etdc)
Certificate Sharing (trust)
SDN, Storage, & DNS
Base nodes ready for installation
Operating System, Storage & Net
Trusted access to systems
SystemWideOperationsConcerns
Delivered via Containers
And “Sidecars”

5. Kubernetes Cluster Services
Client
0
Ready
1
Prereq
2
Control
3
Nodes
etcd
(cluster)
etcd
(cluster)
etcd
(cluster)
API
(cluster)
API
(cluster)
API
(cluster)
Kubelet
KubeCtl
Container Manager
5
Apps
Network CNI
Host
Network
Host
Storage
Host
Init
Pod Pod Pod Pod
4
Add-Ons
Certificate
Authority
Scheduler
(leader)
Heapster
Infrastructure
APIs
Routers,
Storage,
LBs...
Proxy
...
Controller
(leader)
DNS Watcher ...

6. Together 4ever: API server + Kubelet
Client
0
Ready
1
Prereq
2
Control
3
Nodes
etcd
(cluster)
etcd
(cluster)
etcd
(cluster)
API
(cluster)
API
(cluster)
API
(cluster)
Kubelet
KubeCtl
Container Manager
5
Apps
Network CNI
Host
Network
Host
Storage
Host
Init
Pod Pod Pod Pod
4
Add-Ons
Certificate
Authority
Scheduler
(leader)
Heapster
Infrastructure
APIs
Routers,
Storage,
LBs...
Proxy
...
Controller
(leader)
DNS Watcher ...

9. Kubernetes Networking….
Is simple! Everything talks to everything!
Kube Proxy service manages iptables to redirect traffic
between worker hosts.
Or maybe it’s not that simple…. Services, load balancers and CNI.
Multi-tenant isolation requires adding a SDN infrastructure

10. Worker
Nodes
Worker
Nodes
Building in Security: TLS and PKI
Master
Node 1
Master
Node 2+
etcd
Kublet Proxy
API
Server
Controller
Controller
etcd
API
Server
User!
Scheduler
Scheduler

11. Worker
Nodes
What about HA? We need to add Load Balancers
Master
Node 1
Master
Node 2+
Worker
Nodes
etcd
Kublet Proxy
Controller
Controller
etcd
User!
Scheduler
Scheduler
Load
Balancer
API
Server
API
Server

12. Yikes! Can we make that simpler? Compromises...
Worker
Nodes
Master
Node 1
Master
Node 2+
Worker
Nodes
etcd
Kublet Proxy
Controller
Controller
etcd
User!
Scheduler
Scheduler
Load
Balancer
API
Server
API
Server

13. Worker
Nodes
Oh… Apps need Load Balancers too!
Master
Node 1
Master
Node 2+
Worker
Nodes
etcd
Kublet Proxy
Controller
Controller
etcd
User!
Scheduler
Scheduler
Load
Balancer
API
Server
API
Server
End User
Load
Balancer
App
Containers
Let’s
Encrypt

14. So… Operating Kubernetes?
Good News:
● Well designed and active project (quarterly releases!?!)
● Solves real problems for managing container applications
● Great ecosystem building above project
Mixed News:
● Primarily focused on AWS & Google infrastructure
● Scale, upgrades, security and integration still in progress
● Networking and Storage around containers still maturing