A presentation about how to make MySQL highly available, presented at the San Francisco MySQL Meetup (http://www.sfmysql.org/events/15760472/) on January 26th, 2011. A video recording of this presentation is available from Ustream: http://ustre.am/fyLk

1. MySQL High Availability Solutions
Lenz Grimmer <lenz@grimmer.com>
<
http://lenzg.net/ | Twitter: @lenzgr
2011-01-26 | San Francisco MySQL Meetup | USA

2. $ whoami
1998 2002
2008 2010

3. Agenda
● High Availability: Concepts &
Considerations
● MySQL Replication
● DRBD / Pacemaker
● MySQL Cluster
● Other HA tools/applications

4. Single MySQL Server
MySQL Client
SQL (SELECT, INSERT,
UPDATE, DELETE)
MySQL Storage Engines
(InnoDB, MyISAM, PBXT...)
Disk Storage
(XFS, ReiserFS, JFS, ext3...)

5. Why HA?
● Something can and will fail
● Service Maintenance
● Downtime is expensive
● Adding HA to an existing system is complex

6. Elimination of the SPOF
● Identify what will fail eventually
● Hard disks
● Fans
● Consider what might fail
● Application crashes
● OOM-Situations, Kernel-Panics
● Network connections, Cables
● Power supply

7. What is HA Clustering?
● Redundancy
● One system or service goes down → others
take over
● IP address takeover, service takeover
● Ensuring data availability & integrity
● Not designed for high-performance

8. High Availability Levels

9. Rules of High Availability
● Prepare for failure
● Aim to ensure that no important data is
lost
● Keep it simple, stupid (KISS)
● Complexity is the enemy of reliability
● Automate it
● Test your setup frequently!

10. HA Components
● Heartbeat
● Checks that services that are monitored, are alive.
● Can check individual servers, software services,
networking etc.
● HA Monitor
● Configuration of the services
● Ensures proper shutdown and startup
● Allows manual control
● Shared storage / Replication

11. Split-Brain
● Communications failures can lead to
separated partitions of the cluster
● If those partitions each try and take control
of the cluster, then it's called a split-brain
condition
● If this happens, then bad things will happen
● Use Fencing or Moderation/Arbitration to
avoid it

12. Redundancy Using MySQL Replication
MySQL Replication

13. MySQL Replication
● Unidirectional
● Statement- or row-based (MySQL 5.1)
● Built into MySQL
● Easy to use and set up
● One Master, many Slaves
● Asynchronous – Slaves can lag behind
● New in MySQL 5.5: Semisync Replication

14. MySQL Replication (2)
● Master maintains Binary logs & index
● Replication on Slave is single-threaded
http://forge.mysql.com/wiki/ReplicationFeatures/ParallelSlave
● No automated fail-over
● No heartbeat, no monitoring
● New in 5.5: replication heartbeat

15. MySQL Replication - Overview
Read & Write
Web/App
Server
Write
Relay
Log
mysqld SQL
I/O
Threa
Thread
d
Index &
Binlogs
Data
Replication Binlog Data
mysqld
MySQL Master MySQL Slave

16. Statement-based Replication
● Pro
✔ Proven (around since MySQL 3.23)
✔ Smaller log files
✔ Auditing of actual SQL statements
✔ No primary key requirement for replicated tables
● Con
✗ Non-deterministic functions and UDFs
✗ LOAD_FILE(), UUID(), CURRENT_USER(),
FOUND_ROWS()
(but RAND() and NOW() work)

17. Row-based Replication
● Pro
✔ All changes can be replicated
✔ Similar technology used by other RDBMSes
✔ Fewer locks required for some INSERT, UPDATE or
DELETE statements
● Con
✗ More data to be logged
✗ Log file size increases (backup/restore implications)
✗ Replicated tables require explicit primary keys
✗ Possible different result sets on bulk INSERTs

18. Replication Topologies
Master > Slave Master > Slaves
Master > Slave > Slaves Masters > Slave (Multi-Source)
Ring (Multi-Master)
Master < > Master (Multi-Master)

19. Master-Master Replication
● Two nodes are both master and slave to
each other
● Useful for easier failover
● Not suitable for (write) load-balancing
● Don't write to both masters
simultaneously!
● Use Sharding or Partitioning instead
(e.g. MySQL Proxy)

20. MySQL Replication as a HA Solution
● What happens if the Master fails?
● What happens if the Slave fails?
● This doesn’t sound like High Availability!
● Yes!
● Replication is only part of a HA
configuration

21. Pacemaker (Linux-HA)
● Supports 2 or more Nodes (v2)
● Resource monitoring (Apps and HW)
● Active fencing mechanism (STONITH)
● Node failure detection in seconds
● Supports many applications (incl. MySQL)
● http://clusterlabs.org/
● http://www.clusterlabs.org/wiki/Load_Balanced_MySQL_Replicated_Cluster
● http://www.clusterlabs.org/wiki/DRBD_MySQL_HowTo

22. Replication & HA
● Combined with Pacemaker
● Virtual IP takeover
● Slave gets promoted to Master
● Side benefits: load balancing & backup
● Can be tricky to fail back
● No automatic conflict resolution
● Proper failover needs to be scripted

23. Redundancy With Disk Replication
Disk Replication

24. DRBD
● Distributed Replicated Block Device
● “RAID-1 over network”
● Synchronous/asynchronous block replication
● Automatic resynchronisation
● Application-agnostic
● Can mask local I/O-Errors
● Active/passive configuration
● Dual-primary mode (requires cluster file sytem
like GFS or OCFS2)
● http://drbd.org/

25. DRBD in Detail
● DRBD replicates data blocks between to
block devices
● DRBD can be combined with Linux-HA and
other HA solutions
● MySQL runs normally Applications
on primary node
● MySQL is not active on Active Node Virtual IP Passive Node
the secondary node
● DRBD is Linux only DRBD

26. Redundancy Using Shared Storage

27. Replication vs. SAN
● Data Consistency / Integrity
● Synchronous vs. asynchronous
● SAN can become the SPOF
● Cold caches
● “Split brain”-Situations
● SAN/NAS I/O Overhead

28. Redundancy with MySQL Cluster

29. MySQL Cluster
● “Shared-nothing”-Architecture
● Automatic partitioning
● Distributed fragments
● Synchronous replication
● Fast, automatic fail-over of data nodes
● Automatic resynchronisation
● Transparent to MySQL applicationen
● Supports transactions
● http://mysql.com/products/database/cluster/

30. MySQL Cluster
● In-memory indexes
● Not suitable for all query patterns
(cross-table JOINs, range scans)
● No support for foreign keys
● Not suitable for long running transactions
● Network latency crucial
● Can be combined with MySQL replication
(RBR)

32. MySQL Cluster & Replikation
● MySQL Cluster
● Easy failover from one MySQL node to another
● Scaling write load using multiple SQL nodes
● Asynchronous replication from Cluster to
regular MySQL slaves
● Slaves take read load (InnoDB/MyISAM)
● Quick setup of new slaves (Cluster Online
Backup)
● Easy failover and fast recovery

33. http://johanandersson.blogspot.com/2009/05/ha-mysql-write-scaling-using-cluster-to.html

34. Galera Replication
● Patch for InnoDB plus external library
● Synchronous replication
● Single- or multi-master
● Multicast-Replication
● HA plus load sharing possible
● Certifikate-based replikation
(instead of 2PC)
● http://codership.com/products/mysql_galera

35. MMM
● MySQL Master-Master Replication
Manager
● Perl-Script for monitoring, failover and
management
● Master-master replication configuration
(one active writable master)
● Failover by moving virtual IP-Address
● Also supports read balancing
● http://mysql-mmm.org/

36. Continuent Tungsten Replicator
● Database-external solution
● Asynchronous, Master-Slave, Fan-out &
Fan-in
● Java
● Log-based
● Events are stored in the Transaction History
Log (THL)
● Modular architekture (Pipelines, Stages)
● http://continuent.com/community/tungsten-replicator

37. Red Hat Cluster Suite
● HA and load balancing
● Supports up to 16 nodes
● Shared storage
● Monitors services, file systems and network
status
● Fencing (STONITH) or distributed lock
manager
● Configuration synchronization
● http://www.redhat.com/cluster_suite/

38. Solaris Cluster / OpenHA Cluster
● Provides failover and scalability services
● Solaris / OpenSolaris (Project Colorado)
● Kernel-level components plus userland
● Agents monitor applications
● Geo Edition to provide Disaster Recovery
using Replication
● Open Source since June 2007
● http://hub.opensolaris.org/bin/view/Community+Group+ha-clusters/WebHome
● http://hub.opensolaris.org/bin/view/Project+ha-mysql/WebHome

39. Flipper
● Perl Script managing pairs of MySQL
servers replicating to each other.
● Automatic switchover, triggered manually
● Enables one half of the pair to be taken
offline for maintenance work while other
half carries on
● Moves IP addresses based on a role ("read-
only", "writable")
● http://provenscaling.com/software/flipper/

40. Q&A
Questions, Comments?

41. Thank you!
Lenz Grimmer <lenz@grimmer.com>
Twitter: @lenzgr
http://lenzg.net/