Pacemaker: OpenStack's Pid 1

David Vossel <dvossel@redhat.com>
PACEMAKER
OpenStack Summit
May 21,2015
OpenStack's PID 1

Story Time

And how Pacemaker Saved the Day
The Future of HA

There once was a database

DB

Not like the other databases
DB

DB
A distributed self replicating database

A distributed self replicating database
Active/Active Replicated Database
DB DB DB DB

DB DB DB DB
Everyone: HOORAY!

DB DB DB DB
Everyone: Load Balance?
Clients
????????????????????

HAProxy enters the scene.

DB DB DB DB
proxy
HA Proxy: No Problem
Clients

DB DB DB DB
proxy
Clients
Everyone:Hooray!

DB DB DB DB
proxy
Clients
* Everyone: ummmmm

Everyone: What if a node dies?
DB DB DB DB
proxy
Clients

DB DB DB DB
Clients
proxy
Everyone: What if a node dies?

DB DB DB DB
Clients
proxy
* Everyone: ummmmm

DB DB DB DB
Clients
proxy
Everyone: But...What if proxy dies?

DB DB DB DB
Clients
proxy
HA Proxy:...
Everyone: But...What if proxy dies?

DB DB DB DB
Clients
proxy
Everyone:What?????
HA Proxy:...

DB DB DB DB
Clients
proxy
Everyone:Uhhh???!!!!??
HA Proxy:...
????????????????????

DB DB DB DB
Clients
Everyone:Hello????
HA Proxy:...
????????????????????

Clients
Everyone:Anyone?
????????????????????

Everyone:I knew this cloud thing wouldn't work...

KeepaliveD: Wait! Guys...I've got an idea!!!
Everyone:I knew this cloud thing wouldn't work...

KeepaliveD enters the scene.

keepaliveD
DB DB DB DB
proxy proxy proxy proxy
VIP VIP VIP VIP
Clients
Everyone:Whoa, Proxy Failover!

keepaliveD
keepaliveD
DB DB DB DB
VIP VIP VIP VIP
Clients
Everyone:Awesome!!

keepaliveD
keepaliveD
DB DB DB DB
VIP VIP VIP VIP
Clients
Everyone:Awesome!!
Keepalived:: I know
right?!!

keepaliveD
Clients
Everyone: No Way!!!
Keepalived: I Rock!
keepaliveD
Active/Active Replicated DatabaseActive/Active Replicated Database
DB DB DB DB
VIP VIP VIP VIP

keepaliveD
Clients
keepaliveD
DB DB DB DB
VIP VIP VIP VIP
Everyone:Wait...Hold up

keepaliveD
Clients
keepaliveD
DB DB DB DB
VIP VIP VIP VIP
Everyone: But....What actually
happens to the “other”nodes?

Peripeteia - per·i·pe·tei·a
a sudden reversal of fortune or change
in circumstances, especially in
reference to fictional narrative.

keepaliveDkeepaliveD keepaliveD
DB DB DB DB
VIP VIP VIP VIP
Clients
Keepalived:What Other Nodes? Keepalived:What Other Nodes?

keepaliveDkeepaliveD
DB DB DB DB
VIP VIP VIP VIP
Clients
Clients: HEY?! How come the thing
I just wrote isn't in the database?

DB DB DB DB
VIP VIP VIP VIP
Clients
Clients: HEY?! How come the thing
I just wrote isn't in the database?
Clients:Yeah,what's going on?!

DB DB DB DB
VIP VIP VIP VIP
Clients
* Everyone: I've made a huge mistake....

The Missing Piece?

Pacemaker: System Level HA
●
System level HA is holistic.
Pacemaker
DB DB DB DB
VIP VIP VIP VIP
Pacemaker
DB DB DB DB
VIP VIP VIP VIP

Pacemaker
DB DB DB DB
VIP VIP VIP VIP
●
●
Defines the policy of how to recover a set of applications

Pacemaker
DB DB DB DB
VIP VIP VIP VIP
●
●
Defines the policy of how to recover a set of applications
●
Enforces the policy to achieve system wide deterministic behavior.

●
We don't question what happened to the other nodes...We know.
Pacemaker Pacemaker
DB DB DB DB
VIP VIP VIP VIP
Back to the Story... How does Pacemaker Help?

●
We don't question what happened to the other nodes...We know.
●
And how do we know this?
Pacemaker Pacemaker
DB DB DB DB
VIP VIP VIP VIP
Back to the Story... How does Pacemaker Help?

Introducing STONITH

Introducing STONITH
Shoot the Other Node in the Head

STONITH = Pacemaker's Fencing Daemon.
●
Pacemaker knows the state of lost/misbehaving nodes
Pacemaker Pacemaker
DB DB DB DB
VIP VIP VIP VIP

STONITH = Pacemaker's Fencing Daemon.
●
Pacemaker knows the state of lost/misbehaving nodes
●
Because with fencing via STONITH... that state is dead.
Pacemaker Pacemaker
DB DB DB DB
VIP VIP VIP VIP

Quick Recap...

Without Pacemaker+STONITH
keepaliveDkeepaliveD keepaliveD
DB DB DB DB
VIP VIP VIP VIP
Clients
Keepalived:I dunno.Who cares? Keepalived:I dunno.Who cares?

With Pacemaker+STONITH...
Pacemaker Pacemaker
DB DB DB DB
VIP VIP VIP VIP
Pacemaker:They are dead
because I killed them..
Clients

The Takeaway.
●
Pacemaker and load balancing are NOT mutually exclusive.

The Takeaway.
●
●
Pacemaker and HAProxy are meant for one another.

The Takeaway.
●
●
Pacemaker and HAProxy are meant for one another.
HAProxy
Pacemaker

Pacemaker
The Distributed PID 1

Modern PID 1's role.
●
SystemD:
●
Launch services parallel
●
yet observe strict ordering between dependent services.
●
Monitor/recover failed resources.
systemd
galerarabbitmqStart
Order Unrelated dependencies
start in parallel.
Ordering is enforced.
These services can start in
parallel only after their
dependencies start.Nova
redis
ceilometer

The Problem
●
OpenStack services are not isolated to a local machine.
systemd
Galera Cluster
Form
Galera
cluster
Then
Start
Nova.
Nova
NODE1 NODE2 NODE3 NODE4
systemd systemd systemd

The Problem
●
OpenStack services are not isolated to a local machine.
●
SystemD Can't coordinate this.
systemd
systemd systemd systemd
Galera Cluster
Form
Galera
cluster
Then
Start
Nova.
Nova

The Fix.
●
But Pacemaker can
●
because Pacemaker is distributed.
Pacemaker
Galera Cluster
Form
Galera
cluster
Then
Start
Nova.
Nova

The Fix.
●
Pacemaker,just like systemd,can...
●
Launch services parallel
●
yet observe strict ordering between dependent services.
●
Monitor/recover failed resources.
Pacemaker
galerarabbitmqStart
Order Unrelated dependencies
start in parallel.
Ordering is enforced.
These services can start in
parallel only after their
dependencies start.Nova
redis
ceilometer

The Fix.
●
Except pacemaker can coordinate this across any number of nodes.
Pacemaker
RabbitMQ Cluster
NODE5 NODE6 NODE7 NODE8 NODE9
Galera Cluster Redis Cluster
Ceilometer ClusterNova Cluster

The Fix.
●
Except pacemaker can coordinate this across any number of nodes.
●
With any number of resources
Pacemaker
NODE1 NODE2 NODE3 NODE4 NODE5 NODE6 NODE7 NODE8 NODE9
HAProxy HAProxy HAProxy
VIP-Galera VIP-RedisVIP-Rabbit VIP-Nova
HAProxy
VIP-keys
HAProxy
VIP-cinder VIP-celio
Keystone ClusterCeilometer Cluster
Nova Cluster
Cinder Cluster
VIP-glance
HAProxy
VIP-neutron
HAProxy
Glance Cluster
Horizon Cluster
HAProxy HAProxy
RabbitMQ ClusterGalera Cluster Redis Cluster
Neutron Cluster
Swift Cluster
Heat Cluster

Resource Constraints
●
Pacemaker has unique capabilities for managing resources and
modeling complex resource dependencies.
●
Examples:
●
Start resource X then start resource Y
●
Colocate resource X with resource Y
●
Resource X prefers node A over node B
●
Resource X prefers node A between 8am-5pm

Architecture
HA OpenStack Controller Nodes

How it works... in one sentence
Pacemaker managing Virtual IPs + Load Balancers + Controller
services to maximize the availability of OpenStackAPIs

Pacemaker
HA-proxy
VIP
Service Service Service
NODE1 NODE2 NODE3
Each distributed service has a front end
Virtual IP tied to a Load Balancer.
HA-proxyHA-proxy

Each Load Balancer routes VIP traffic to its
respective Active/Active service instances
Pacemaker
HA-proxy
VIP
NODE1 NODE2 NODE3
HA-proxyHA-proxy

Pacemaker
HA-proxy
VIP
NODE1 NODE2 NODE3
HA-proxy
HA-proxy

Active/Active OpenStack Controller
service... like Nova, Glance, Keystone,
Galera, Redis, Rabbitmq, ect...
Pacemaker
HA-proxy
VIP
NODE1 NODE2 NODE3
HA-proxyHA-proxy

PROXY CLONE
SERVICE CLONE
Pacemaker
HA-proxy
VIP
NODE1 NODE2 NODE3
Scaling with Resource Clones
●
Pacemaker's ability to clone services makes scaling trivial.
●
Want more instance?
HA-proxy HA-proxy

PROXY CLONE
SERVICE CLONE
Pacemaker
HA-proxy
VIP
NODE1 NODE2 NODE3
Scaling with Resource Clones
NODE4 NODE5 NODE6
●
Increment the number of clone instances pacemaker is allowed to run
for a service to scale service instances.
HA-proxy HA-proxy HA-proxy HA-proxy HA-proxy

Pacemaker
HA-proxy
Galera VIP
Galera Galera Galera
NODE1 NODE2 NODE3
How it works, continued...
●
Services interact with one another using each service's Virtual IP
●
Example: Both Glance and Nova need access to Galera...Galera is
accessed via the front end Virtual IP and those requests are
distributed to the backend galera cluster.
Glance Nova
DB Request DB Request
HA-proxyHA-proxy

Deployment Strategies
HA OpenStack Controller Nodes

Collapsed Architecture
Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
HA Proxy HA Proxy HA Proxy
Separate VIP per service
●
All controller nodes run the same services.
●
VIP+load balancers distribute access to APIs
across cloned nodes.

Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NOVA VIP
Clients

Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
Glance VIP
Clients

Collapsed Architecture Scaling
Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE4
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE5
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE6
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S

Startup Ordering Revisited.
Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
Bootstrap and start Start Galera Cluster
across multiple nodes.

Startup Ordering Revisited.
Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
Bootstrap and start Start Galera Cluster
across multiple nodes.
Then Start Nova instances, which depend
on an active Galera cluster.

Stop ordering Ordering Revisited.
Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
If we're shutting down galera cluster

Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
Stop everything that depends on Galera.
Like Nova...

Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
Then Shutdown Galera cluster.

Complex Startup Ordering
Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Slave
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Slave
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Slave
Neutron N
Swift
Heat
Neutron S
Start Redis Clone.

Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Slave
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Slave
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Master
Neutron N
Swift
Heat
Neutron S
Promote one instance of Redis
clone to be Master Instance.

Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Slave
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Slave
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis Master
Neutron N
Swift
Heat
Neutron S
Then start Ceilometer cluster which
depends on Redis

Segregated Architecture
Pacemaker
●
Each service runs on its own dedicated hardware.
●
Scales much further
●
Add capacity where capacity makes sense.
●
Requires lots and lots of nodes.

Pacemaker
●
Take a closer look.

Pacemaker
NODE1
Galera
NODE2
Galera
NODE3
Galera
●
Possible to have have an entire set of nodes just for Load balancing
●
Dedicated cluster for galera
NODE4
Proxy
NODE5
Proxy
NODE6
Proxy
NODE7
Nova
NODE8
Nova
NODE9
Nova
VIP VIPVIP
More services that way.
Glanc
NOD
VIP VIPVIP
VIP VIPVIP
VIP VIPVIP
VIP VIPVIP

Mixed Architecture
●
Mixture of collapsed and segregated.
●
Break some components into separate hardware
●
Most of cluster is collapsed
Pacemaker
NODE1
Galera
NODE2
Galera
NODE3
Galera
NODE4
Proxy
NODE5
Proxy
NODE6
Proxy
NODE7 NODE8 NODE9
VIP VIPVIP
VIP VIPVIP
VIP VIPVIP
VIP VIPVIP
VIP VIPVIP
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Redis
Neutron N
Swift
Heat
Neutron S
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Redis
Neutron N
Swift
Heat
Neutron S
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Redis
Neutron N
Swift
Heat
Neutron S

Pacemaker Advantages
●
Automates bootstrap of services that previously required hand holding.
●
Example: Automate Galera bootstrap
1.Find out which galera instance is most up-to-date
2.Bootstrap most current galera instance first.
3.Then sync other galera instances

Pacemaker Advantages. Continued...
●
start/stop distributed services in a graceful ordered manner.
●
Gracefully and controller node into standby for maintenance.
●
Dynamically grow capacity by adding more pacemaker nodes
●
Centralized view of distributed service state.

Architecture
HA OpenStack Compute Nodes

HA for Cattle
●
Both Pets and Cattle need High Availability.
●
Recognize the techniques used for each are different.

Pacemaker for Pets and small Herds.
●
No limits in the number of resources.
●
Pacemaker supports“n-node”clusters.
●
Cluster are limited by the Corosync messaging layer to 16 nodes.
Pacemaker

Pacemaker Remote for the Cattle.
●
Pacemaker Remote allows clusters to scale beyond corosync
membership layer limitations.
●
Pacemaker Remote can scale clusters to 100s possibly 1000s of nodes.
Pacemaker + Pacemaker Remote

The Solution: Pacemaker Remote
●
Pacemaker Remote is a single daemon,pacemaker_remoted
Pacemaker Remote
Remote Node

Pacemaker
Node 2 Node 3Node 1
Pacemaker Remote
Node 5 Node 6Node 4
Node 8 Node 9Node 7
Node 11 Node 12Node 10
Node 16
Remote Node
●
Pacemaker Remote is a single daemon, pacemaker_remoted
●
This daemon is a lightweight way of integrating nodes into the cluster.

Pacemaker
Node 2 Node 3Node 1
Pacemaker Remote
Node 5 Node 6Node 4
Node 8 Node 9Node 7
Node 16
Remote Node
●
Pacemaker Remote is a single daemon, pacemaker_remoted
●
This daemon is a lightweight way of integrating nodes into the cluster.
●
Cluster services spread out across pacemaker and pacemaker_remote
nodes as a single cluster partition.
Cluster Services

Pacemaker Remote use-case
Pacemaker
Node 2 Node 3Node 1
Pacemaker Remote
Node 5 Node 6Node 4
Node 8 Node 9Node 7
Node 16
Remote Node
●
Management of services on Pacemaker Remote can work just like Pacemaker
●
But thrives in the cattle use case where every remote instance is identical.
Cluster Services
Cloned service
Pacemaker Remote
Remote Node
Cloned service
Pacemaker Remote
Remote Node
Cloned service

Pacemaker Remote use case
Pacemaker
Node 2 Node 3Node 1
●
Cloned services scale quite well on pacemaker remote
Cluster Services
Cloned services
Remote Node
Pacemaker Remote

Compute Node HA Strategy.
Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S

Compute Node HA Strategy.
Compute Service Group
Remote Node
Pacemaker Remote
Pacemaker
NODE1
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE2
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
NODE3
Keystone
Ceilometer
Nova
Cinder
Glance
Horizon
RabbitMQ
Galera
Redis
Neutron N
Swift
Heat
Neutron S
LibvirtdD
Neutron Open vSwitch
Ceilometer Compute
Nova Compute

Why HA Compute Nodes?
●
Maximize the Availability of Compute Instances.
●
detection of dead Cattle instances
●
Automate recovery of Cattle instances

Why HA Compute Nodes?
●
Maximize the Availability of Compute Instances.
●
detection of dead Cattle instances
●
Automate recovery of Cattle instances
●
Pacemaker Remote also has a secret weapon.

STONITH + Pacemaker Remote.

The Future

Limits?

How many
services?

How many
services?
How many
nodes?

Questions?
Visit us at
clusterlabs.org

Pacemaker: OpenStack's Pid 1

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Semelhante a Pacemaker: OpenStack's Pid 1

Semelhante a Pacemaker: OpenStack's Pid 1 (20)

Último

Último (20)

Pacemaker: OpenStack's Pid 1