Control Center is an open source application management and orchestration system. Control Center was developed by Zenoss; however, apart from working with the Zenoss platform Control Center can be used to manage other Docker applications, from a simple web application to a multi-tiered stateful application stack. Control Center is based on a service oriented architecture which enables applications to run as a set of distributed services that can span hosts, datacenters, and geographic regions. A simple declarative application template file is used to describe how an application is configured, deployed, managed, and monitored.

1. Zenoss Control Center
Open Source Application Management
& Orchestration System for Docker
controlcenter.io

2. Overview
• History
• Brief introduction to Docker
• Why Docker?
• What is Control Center?
• Demo
• Questions

3. History
• Zenoss founded in 2005 as Open Source
Enterprise IT Management Solution
• Docker was chosen to support the scale
requirements for monitoring the modern
datacenter
• Need was created for managing Zenoss within
Docker
• Birth of Control Center
• Zenoss Control Center Open Sourced in
August 2014

4. What is Docker? docker.io
Docker is an open-source project to easily create
lightweight, portable, self-sufficient
containers from any application.
http://slides.com/stevenborrelli/docker

5. ● Based on Linux containers
● Minimal overhead (cpu/io/network)
● Can run on any Linux system that supports LXC, nsinit
● Docker container contains everything it needs to run
● A docker container should be able to run anywhere that
Docker can run.
What is Docker?
http://slides.com/stevenborrelli/docker

6. Why Docker?
• Scalability
– Ability to spin up new daemons on the fly
• Easy Installs
– Just pull image and deploy
– Dependencies exist in the container
• Reliable History Tracking
– Improves upgrades
– Easy to roll back

7. What Docker doesn’t provide (yet)
● service discovery
● a good log interface
● cross host volumes
● cross host scheduling
… and anything else you may find in a cluster
manager

8. Where Docker Ends, Control Center Begins
App Containers
Control Center
Container Manager
Host OS (UOS)
Physical/Virtual
Server
CC
Control Center (CC) is an
application written in Go!
(aka golang).
CC takes care of all
interactions with Docker
and the host system.
You install CC on every
computer that you intend
to run Zenoss on.
CC is open source software
(Apache 2 license).

9. Terms
● serviced: Control Center command-line client
● Service: An entity that manages a container and/or other services.
● Service Definition: Describes how a service is run and configured
● Service Template: A document of service definitions relating to a platform.
We tell Control Center how to execute the
application inside of a container image via a
Service Template.
The same container may act completely
differently depending on the service definition
used to start it.

10. CONTROL CENTER - SYSTEMS
The Control Center has 3 important systems that divide the work of
orchestrating and monitoring of services.
Master - This subsystem houses several subservices and is generally
responsible of the overall orchestration of the system. There will only be one
Control Center Master.
Node Controller - This controller is responsible for all docker interaction,
starting/stopping services, mounting of service resources and metric collection.
There is one of these per host.
Container Controller - This program is the first program that gets launched
inside a container. It is responsible for starting the actual service, providing
network proxies, forwarding logs/events, and forwarding service specific stats.

11. CONTROL CENTER - SYSTEM OVERVIEW
Master
Logstash
Zookeeper
Metric Svc
Docker
Registry
UI/Rest API
RPC API
Node
Controller
RPC API
Docker
Container
Container
controller
Container
controller
controller
logstash-forwarder
logstash-forwarder
Your Service
Your Service
Your Service
logstash-forwarder
TCP Mux
Shared FS Host Boundary
Node Controller RPC TCP Mux

12. CONTROL CENTER MASTER
Elasticsearch
Database for control center
Zookeeper
Coordination service
Logstash
Log consolidation
Uses its own copy of Elasticsearch
Metrics -- CC centric metric system
OpenTSDB
Standalone Hbase
Consumer
Query service
Celery
Schedule tasks
Registry
docker-registry
Shared Storage (NFS)
Web server for CC UI
Rest API / RPC API

13. CC NODE CONTROLLER
● 1 Node Controller per host
● Configured how to talk to master
● Watches the master’s zookeeper for services that has been scheduled
to start on its hosts
● Pulls down images from Master’s Docker Registry
● Starts/Stops/Manages services as needed
● Mux distributes all network traffic to this box to the appropriate
containers

14. CONTAINER CONTROLLER
● Injected into each container
● Serviced binary is bind mounted into every container that it runs
automatically
● The Serviced container controller is what runs and then executes the
actual service command
● Imports endpoints from other services
● transparently proxies connections to remote endpoints and makes them
appear local
● Bind mounts Logstash forwarder

15. CC - CORE CONCEPTS
Pool - a collection of compute, network and storage
resources.
● IP Addresses
● Hosts
● Storage Systems (future)

16. CC - CORE CONCEPTS
Host - a compute node (physical or virtual) that is available to
the control center.
○ Hostname
○ Pool ID
○ Kernel Version
○ OS Distribution
○ Docker Version
○ ServiceD Version
○ Memory Size
○ CPU Count
○ Interface (NICs) Names & Addresses
○ Monitoring Profile
https://github.com/zenoss/serviced/blob/develop/d
omain/host/host.go#L19

17. CC - CORE CONCEPTS
Service Template - A hierarchical description of a services that deliver an
Application
Resource Manager
Zproxy (Nginx)
Zope
Query OpenTSDB HBase
MariaDB
ZEP
RabbitMQ
MariaDB 2
Zookeeper
Not Limited to Tree Dependencies

18. CC - CORE CONCEPTS
Service - Is a description of an individual process.
● ImageID - what container image does this service use
● Startup - what command must run to start the service
● Config Files - what configuration files to inject into the container
before startup
● Log Files - what log files the service writes to inside the container
and what is the format of those log files
● Memory Limit - what is the maximum allowable amount of memory
the service can use
● Volumes - what persistent disk storage is needed by the service and
where that storage should be mounted
● Endpoints - what network endpoints does this service depend on or
does this service expose
● Health Checks - what scripts can be run to determine the health of a
service
● Metrics - what datapoints the service exposes
● Thresholds - what thresholds can be applied to the exposed metrics
● Graphs - what graphs are relevant to the service

19. CC - CORE CONCEPTS
Schema - the relationship between the various domain objects in the control
center
Service Control Center
Template
Pool [default]
Host [app0.example.com]
Host [app0.example.com]
Host [app0.example.com]
Service A
Service A
Service A
ServiceA @T0

20. CC - NETWORK CONNECTIONS
Imagine your service needs to talk to Redis.
It thinks it can access redis via 127.0.0.1:6379
The container controller actually accepts the connection
Your Service
Container
Controller
localhost interface (lo)
127.0.0.1
Zookeeper
The controller finds where redis is actually running via ZK.
Mux
The controller makes an encrypted connection to the mux.
Redis Container
The mux contacts the exposed port of the redis container.

21. MariaDB Sample Service Template
{
"Command": "/usr/bin/mysqld_safe",
"Endpoints": [
{
"Application": "mysql",
"PortNumber": 3306,
"Protocol": "tcp",
"Purpose": "export"
}
],
"ImageId": "zenoss/zenoss5x",
"Instances": {
"min": 1
},
"Launch": "auto",
"LogConfigs": [
{
"path": "/var/log/mysqld.log",
"type": "mysql"
}
],
"Name": "MySQL",
"Snapshot": {
"Pause": "PAUSE_CHECK_TIMEOUT=60 LOCK_HOLD_DURATION=600 $ZENHOME/bin/quiesce-mysql.sh pause",
"Resume": "$ZENHOME/bin/quiesce-mysql.sh resume"
},
"Tags": [
"daemon"
]
}

22. Persistent Volumes
For those services that have to maintain state between
execution, the Control Center will mount a folder
from the Shared File System into the container.
The SFS is used so that even if the service starts up on a
different machine it can still access the same files.
Service 1
Service 2
Control Center
Docker
SFS
Operating System
Service 3
Service 4
Service 6
Control Center
Service 5
Docker
SFS
Operating System

23. Install Control Center on Ubuntu
14.04
# install docker repo
wget -O - http://get.docker.io | sh
# install zenoss repo
sudo apt-key adv --keyserver keys.gnupg.net --recv-keys AA5A1AD7
sudo sh -c 'echo "deb [ arch=amd64 ] http://get.zenoss.io/apt/ubuntu trusty universe"
> /etc/apt/sources.list.d/zenoss.list’
# update repo database
sudo apt-get update
# install Control Center
sudo apt-get -y install serviced
# then start services
sudo start serviced

24. Demo Time

25. Questions?