5. High Availability
● Minimize downtime by avoiding SPOF
● Create service redundancy
○ Active-Active When possible
■ Stateless services or HA internal support
■ Active-Passive if nothing else is applicable
● Scale out Architecture
8. ● Cluster Resource Manager
● Uses Corosync for cluster communication
● Monitor and Control Resources:
○ Floating Virtual IP Address (VIP)
○ SystemD/LSB/OCF Services
○ Cloned Services(Active/Active)
● STONITH - Fencing with Power Management
○ Important for ensuring data consistency
Pacemaker
9. ● Virtual IP(VIP)
● SystemD Cloned Resource
● STONITH Fencing
Pacemaker OpenStack Service
Node 2 - 192.168.1.2Node 1 - 192.168.1.1
pcsd pcsd
Cloned
STONITH STONITH
Service Service
Service
Virtual IP
10.0.0.1
10. HAProxy Load Balancer
Load Balancing and Proxy for HTTP/TCP
● Mature and popular with web applications
● Health Checking
● Load Distribution
11. ● Load Distribution
○ Round Robin,
○ Stick-Table
● API Isolation
● Failure Detection
Node 1
Node 2 Node 3
HAProxy Load Balancer
Service Service
HAProxy
37. Nova
Nova
Compute
★ Nova-API
○ API
★ Nova-Scheduler
○ VM placement
★ Nova-Conductor
○ Updates DB on
Compute’s behalf
★ Nova-Compute
○ Runs VM Instances
SQL
Nova-API
Nova
Scheduler
RabbitMQ
Nova
Conductor
libvirt/KVM
VMVM
39. Controller Services
● Nova-API configured
with LB and VIP
● Nova-API,
Nova-Scheduler and
Nova-Conductor
are Stateless A/A
Cloned services
Node 2Node 1
ClonedConductor Conductor
ClonedHAProxy HAProxy
pcsd pcsd
ClonedNova-API Nova-API
ClonedScheduler Scheduler
Nova-API
VIP
Nova
SQL RabbitMQ
STONITH STONITH
40. Compute Service
● Each host is independent
● Nova-compute watched locally
by SystemD
● VM HA not supported(yet),
Probably Liberty
Nova
Compute2
Nova
Compute
libvirt/KVM
VM
Compute1
VMVM
Nova
Compute
libvirt/KVM
41. Compute Service
● Probably supported in Liberty
● Each host is independent
● Nova-compute watched locally
by SystemD
● Liberty Blueprint: Mark Host Down
Nova VM HA
Compute1
VMVM
Nova
Compute
libvirt/KVM
STONITH
pacemaker_remote
Compute1
VMVM
Nova
Compute
libvirt/KVM
STONITH
pacemaker_remote
49. Horizon
● Cloned Stateless
HTTPd Service
● Same SSL Certs
on all nodes
● Cache is local
on each host
Node 2Node 1
Cloned
Cloned
HTTPd/
Horizon
HTTPd/
Horzon
HAProxy HAProxy
pcsd pcsd
Horizon
VIP
STONITH STONITH
56. Resources
RDO HA Ref Arch
https://github.com/beekhof/osp-ha-deploy
Layer 3 High Availability - VRRP DVR DHCP
http://assafmuller.com/2014/08/16/layer-3-high-availability/
DVR
http://assafmuller.com/2015/04/15/distributed-virtual-routing-
overview-and-eastwest-routing/
Creating a Highly Available Red Hat OpenStack Platform
Configuration (OSP5 and RHEL 7)
https://access.redhat.com/articles/1150463
About High Availability with OpenStack Platform
https://access.redhat.com/articles/1274203
New nova API call to mark nova-compute down
https://review.openstack.org/#/c/169836/
The Different Facets of OpenStack HA
http://blog.russellbryant.net/2015/03/10/the-different-facets-of-
openstack-ha/
Implementation of Pacemaker Managed OpenStack VM Recovery
http://blog.russellbryant.net/2015/04/08/implementation-of-
pacemaker-managed-openstack-vm-recovery/
57. HA Talks during Summit
HA Infrastructure Talks
Pacemaker: OpenStack’s PID 1
MariaDB Galera cluster : Best practices
High Availability Architecture
Deep Dive Into a Highly Available OpenStack Architecture
Real World Practices
Highly Available OpenStack: From Theory to Reality
Lessons learned on upgrades: the importance of HA and
automation
Providing OpenStack Service High-Availability Through
Anycast Routing
HA Storage Talks
Keeping OpenStack storage trendy with Ceph and containers
DRBD9 for OpenStack
The Road to Enterprise-Ready OpenStack Storage as Service
Dude, where is my volume
HA Networking Talks
Highly Available, Performant, VXLAN Service Node
IPv6 impact on Neutron L3 High Availability
High Availability and Resiliency Testing Strategies for OpenStack
Clouds