Tutorial presented at Percona Live UK 2011

1. MySQL High Availability Sprint:
Launch the Pacemaker!
Florian Haas
hastexo

2. MySQL HA Sprint: Get Prepared!
● Please grab a DVD from your table!
● We have 12 in total – please share if needed!
● Follow instructions in README.pdf.
● If your laptop does not have a DVD drive, please see
me for a USB stick.
● For any questions, I'm the guy in the blue shirt who
answers to “Florian”. When in doubt, just holler.

3. Florian Haas
CEO & Principal Consultant, hastexo
http://www.hastexo.com/who/florian
florian@hastexo.com

4. Visual Cues & Clues
How to follow this talk easily

5. This layout means:
Would you treat me
to the courtesy of your kind attention?

6. This layout means:
● I'd ask you to do this,
● and this,
● and this.

7. Our virtual environment

8. An Overview
of the Linux HA stack
(for MySQL)

9. Application Interface MySQL RA
Cluster Resource Mgmt Pacemaker
Cluster Messaging Corosync
Storage Replication DRBD
(optional)

10. Physical DRBD Physical
Host Host
MySQL
Master
MySQL MySQL MySQL App App
Slave Slave Slave Server Server

11. Physical DRBD Physical
Host Host
MySQL
Master
MySQL MySQL MySQL App App
Slave Slave Slave Server Server

12. DRBD
http://www.hastexo.com/knowledge/drbd
http://www.drbd.org
DRBD & the DRBD logo are registered trademarks of Linbit Information Technologies

13. Create a DRBD resource!
● Name: mysql
● Device name: /dev/drbd0
● Backing disk: /dev/sdb1
● Meta data: internal
● Replication: 192.168.133.111 ↔ 112
Port 7788
Hint: use “legacy” (pre-8.4) DRBD User's Guide

14. Corosync
http://www.hastexo.com/knowledge/corosync
http://www.corosync.org

15. Create a Corosync Cluster!
● Enable secauth
● 2 rings, RRP mode: active
● Multicast addresses: 239.255.42.0 and .1
● Network addresses: 192.168.122.0 and .133.0
● No logging to files, syslog only
● Start Corosync
● Check connectivity (corosync-cfgtool -s)

16. Pacemaker
http://www.hastexo.com/knowledge/pacemaker
http://www.clusterlabs.org

17. Create a Pacemaker Cluster
● Add the “pacemaker” service to /etc/corosync.d
● Use ver: 1
● Restart corosync
● Start pacemakerd
● Verify both nodes have joined the cluster
(crm_mon)

18. STONITH/Fencing

19. Set up your cluster
● Open the shell with crm configure
● Set no-quorum-policy to ignore
● Disable STONITH
● Kids, don't do this at home!
● Commit the configuration

20. Pacemaker Resources

21. Set up a floating IP address
● Add an IPaddr2 resource
● Name: p_ip_mysql
● IP: 192.168.122.110
● CIDR Netmask: 24
● NIC: eth0
● Commit the configuration

22. Resource monitoring

23. Add resource monitoring
● Add a monitor operation to p_ip_mysql
● Interval: 10s
● Commit the configuration
● Manually delete the IP address
● Watch with crm_mon -rf
● Observe resource recovery

24. Master/Slave Sets

25. Started
start stop
Stopped

26. Started Started
start stop start stop
Stopped Stopped

27. Master
promote demote
Slave Slave
start stop start stop
Stopped Stopped

28. Add a DRBD Master/Slave Set
● Create the DRBD primitive p_drbd_mysql
● Use the ocf:linbit:drbd RA
● Add 2 monitor operations (role=Master, role=Slave)
with non-equal intervals
● Wrap it in a master/slave set
● notify=true, clone-max=2, master-max=1
● Commit the configuration
● Watch the cluster promote

29. Constraints

30. Create a Filesystem
● Run mkfs -t ext3 on whichever node is the Master
● Add the Filesystem resource
● Name: p_fs_mysql
● Device: /dev/drbd0
● Directory: /var/lib/mysql
● File system type: ext3
● Add required order & colocation constraints

31. Groups

32. Add a Group
● Create the g_mysql resource group
● Add p_fs_mysql and p_ip_mysql
● Watch the constraints update automagically

33. And Finally: MySQL!

34. Add the MySQL resource
● Run mysql_install_db
● Create an ocf:heartbeat:mysql primitive p_mysql
● Set additional_parameters:
--bind-address 192.168.122.110
● Add resource level monitoring
● Add it to the g_mysql group
● Commit, Rejoice!

35. One more thing...

36. Physical DRBD Physical
Host Host
MySQL
Master
MySQL MySQL MySQL App App
Slave Slave Slave Server Server

37. Physical DRBD Physical
Host Host
MySQL
Master
MySQL MySQL MySQL App App
Slave Slave Slave Server Server

38. Physical Physical
Host Host
MySQL
InnoDB Recovery
Master
Umm.. WTF? Hurry up!
MySQL MySQL MySQL App App
Slave Slave Slave Server Server
Booooring

39. Physical Physical
Host Host
MySQL
Master
MySQL MySQL MySQL App App
Slave Slave Slave Server Server

40. Further Information
http://www.hastexo.com
http://twitter.com/hastexo

41. Liked this talk?
● Go to www.hastexo.com/shoutbox
● Log in
(you can just use your OpenID from Wordpress,
Blogger, Yahoo, Google Apps, Google Profile)
● Leave us a message!