Presentation on MaestroNG, an orchestration and management tool for multi-host container deployments with Docker.
#lspe meetup, February 20th, 2014 at Yahoo!'s URL café.
12. Maestro is actually MaestroNG,
a re-invention of Kimbro Staken’s Maestro
(formerly, dockermix)
13. Takes in a definition of services, their dependencies ,
configuration and target host…
!
…and automates the deployment (and control) of their
corresponding containers on these hosts.
14. Classic use case: a pool of “dumb” workers on your
favorite cloud/hosting provider that just run Docker.
!
No need to (ma)ssh into anything,
no need to pre-configure anything.
!
Everything is remote controlled.
15. Other typical use case: running all the components of
your stack in a single, local virtual machine.
!
Useful for development, integration testing, etc.
16. Philosophy: lightweight application/service containers.
!
Represent and control your software stack
and its dependencies.
!
Docker images are the output of your CI process
(automation!).
!
Start fast, fail faster.
Not for heavyweight, complex container “VMs”.
17. Each service instance (container) defines where it runs
and which ports it exposes, among other things.
!
Like Docker links, Maestro works by injecting this
information in the container’s environment about each
container’s service’s dependencies.
20. Using this information, you can configure your
application at container start time.
!
If you like Python, Maestro helps you by providing a set
of guest helper functions in maestro.guestutils to easily
extract and use this data.
21. #!/usr/bin/env python
!
# This is my cool container’s “init script”
!
import os
from maestro.guestutils import *
!
os.execl(‘java’, ‘java’,
‘-jar’, ‘my-app.jar’,
‘-DlistenPort={}’.format(get_port(‘service’)),
‘-DzkServers={}’.format(
get_node_list(‘zookeeper’, ports=[‘peer’])))
22. Dependency order is respected on start;
inverse order on stop.
!
Can be overridden to stop individual services or
containers.
24. So how do you wield
this power?
A bit clunkily, with YAML (and a bit of Jinja2).
!
!
!
(sorry)
25. # Yay, YAML!
name: lspe
!
registries:
# Define custom image registries for
# private registries, with credentials.
!
ships:
# Declare each target host.
# (Docker daemon locations)
!
services:
# Declare each service, their
# instances, dependencies and
# configuration
26. registries:
# Quay.io with Maestro robot account
quay.io:
registry: https://quay.io/v1/
email: maestro@signalfuse.com
username: signalfuse+maestro
password: {{ env.SUPER_SECRET }}
When starting a container, Maestro will automatically
login and pull the image from the right place if the image
name matches a configured registry.
27. ships:
# Local virtual machine
vm:
ip: 192.168.10.2
docker_port: 4243
timeout: 10
# Slow VM is slow
# A shorter form…
vm2: {ip: 192.168.10.3, timeout: 5}
Ships carry containers and are referred to by name in the
configuration.
28. services:
# ZooKeeper
zookeeper:
image: quay.io/signalfuse/zookeeper:3.4.5
!
# Our zoo isn’t too wild,
# only one keeper is enough.
zk-node-1:
ship: vm
ports:
client: 2181
peer: 2888/tcp
leader_election: “3888/tcp:3888/tcp”
# Keep persistent data on the host.
volumes:
/var/lib/zookeeper: /data/zookeeper
# Environment can be passed-in too.
env:
JVM_FLAGS: “-Xmx1g”
29. # Kafka
kafka:
image: quay.io/signalfuse/kafka:0.8.0
requires: [ zookeeper ]
env:
ZOOKEEPER_BASE: /lspe/kafka
RETENTION_HOURS: 48
broker-1:
ship: vm
ports: {broker: 9092, jmx: “7199:17199”}
# Keep persistent data on the host.
volumes:
/var/lib/kafka: /data/kafka
env:
BROKER_ID: 0
More flexibility in port mappings, volume bindings, and
environment variables definition not shown here.
30. See README.md for full
syntax details and features
https://github.com/signalfuse/maestro-ng/blob/master/README.md
32. What’s next?
More flexible service status detection (not only port pinging)
Soft and hard service dependencies
Parallel startup of independent services and instances of a service
33. That’s it!
Thanks for listening! :)
github.com/dotcloud/docker-py
github.com/signalfuse/maestro-ng