Docker is an amazing technology. In particular, its build-once-run-anywhere model unlocks the world of cluster schedulers like Mesos and Kubernetes. These solve many of the problems of running high-scale websites, but introduce new challenges that need addressing.
In this talk, Evan will describe PaaSTA, a PaaS built on top of open source tools including Docker, Mesos, Marathon, and Chronos. PaaSTA provides tooling for developers to quickly turn their microservice into a monitored, highly available application spanning multiple datacenters and cloud regions. Evan will give an overview of the open-source technologies that power PaaSTA, discuss how Yelp has glued these together to give developers control without burdening them with the complexities of the infrastructure, and show the workflow used by developers to update and maintain their services on PaaSTA.
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
The Glue is the Hard Part: Making a Production-Ready PaaS
1. The Glue is the Hard Part:
Making a Production-Ready PaaS
Evan Krall
Site Reliability Engineer @ Yelp
2. Agenda
PaaSTA
What parts does PaaSTA
have?
How did we glue them
together?
Wrap-up
Intro
Context: Yelp before
PaaSTA
What's in a PaaS?
Production-Ready
What makes a PaaS
production-ready?
Lessons learned
Next steps
8. Dependency Hell
As services gain adoption, shared libraries
become difficult to upgrade. Not all services
are Python anymore.
8
9. Too Many Services
We can no longer fit all services on each
service host. How do we split them up?
9
10. “I wonder how many
organizations that say they're
"doing DevOps" are actually
building a bespoke PaaS. And
how many of those realize it.”
— @markimbriaco
1
0
30. ● Mesos is an "SDK for distributed
systems", batteries not included.
● Requires a framework
○ Marathon (like ASG for Mesos)
○ Chronos (Periodic tasks)
● Supports Docker as task executor
Scheduling in PaaSTA:
Mesos and Marathon
30
31. Marathon
● Run N copies of Docker image
● Works with Mesos to find space on
cluster
● Replaces dead instances
31
43. Aside: Declarative Control
43
● Describe end goal, not path
● Helps achieve fault tolerance
"Deploy 12abcd34 to prod"
vs.
"Commit 12abcd34 should be running in prod"
Gas pedal vs. Cruise Control
44. Configuring Marathon
44
● Need a wall around Marathon: it has root
on your entire cluster.
● Cron job
● Combines per-service config and
currently-blessed docker image
48. Discovery in PaaSTA:
SmartStack
● Registration agent on each box
writes to ZooKeeper
● Discovery agent on each box reads
from ZK, configures HAProxy
48
50. Registering with SmartStack
50
● configure_nerve.py queries local
mesos-slave API
● Keeping it local means registration
works even if Mesos master or
Marathon is down.
● We can register non-PaaSTA
services as well
52. Nerve registers service instance in ZooKeeper:
/nerve/region:myregion
├── service_1
│ └── server_1_0000013614
├── service_2
│ └── server_1_0000000959
├── service_3
│ ├── server_1_0000002468
│ └── server_2_0000002467
[...]
from http://mesos.apache.org/documentation/latest/mesos-architecture/
{
"host":"10.0.0.123",
"port":31337,
"name":"server_1",
"weight":10,
}
ZooKeeper Data
52
53. Normally hacheck acts as a
transparent proxy for healthchecks:
$ curl -s yocalhost:6666/http/service_1/1234/status
{
"uptime": 5693819.315988064,
"pid": 2595160,
"host": "server_1",
"version": "b6309e09d71da8f1e28213d251f7c",
}
$
hacheck
53
54. Can also force healthchecks to fail
before we shut down a service
$ hadown service_1
$ curl -s yocalhost:6666/http/service_1/1234/status
Service service_1 in down state since 1443217910: krall
$
hacheck
54
57. HAProxy
● By default, bind to 0.0.0.0
● Bind only to yocalhost on public-
facing servers
● Gives us goodies for all clients:
○ Redispatch on conn failure
○ Easy request logging
○ Rock-solid load balancing
57
58. yocalhost
58
● One HAProxy per host
● What address to bind HAProxy to?
● 127.0.0.1 is per-container
● Add loopback address to host:
169.254.255.254
● This also works on servers without
Docker
59. docker container 2
lo 127.0.0.1
eth0 169.254.14.18
docker container 1
yocalhost
59
lo 127.0.0.1
eth0 169.254.14.17
docker0 169.254.1.1
eth0 10.1.2.3
haproxy
lo 127.0.0.1
lo:0 169.254.255.244
63. Monitoring a PaaS is different
63
● Things can change frequently
○ Which boxes run which services?
○ What services even exist?
● Traditional "host X runs service Y"
checks don't work anymore.
64. Monitor the invariants
64
● N copies of a service are running
● Marathon running on X,Y,Z
● All nodes are running mesos-slave,
synapse, nerve, docker
● Cron jobs have succeeded recently
65. Sensu monitoring
65
● Decentralized checking
● Client executes checks, puts
results on a message queue
● Sensu servers handle results from
the queue, route them to email,
PagerDuty, JIRA, etc.
66. try:
something that might fail
except:
send failure event
else:
send success event
We can send our own events
66
70. The right abstractions can save you a lot of
work if you need to swap components
Between infra components
70
71. Iterative improvements find
local optima
Sometimes you need to take bigger risks to
get bigger rewards
"Evolution versus Revolution"
71
72. ● It's open source now!
● More polish, docs, examples
● Support more technologies
○ Chronos in-progress
○ Docker Swarm?
○ Kubernetes?
What's next for PaaSTA?
72
