O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

MySQL Replication Evolution -- Confoo Montreal 2017

300 visualizações

Publicada em

MySQL Replication has evolved since the early days with simple async master/slave replication with better security, high availability, and now InnoDB Cluster

Publicada em: Internet
  • Hey guys! Who wants to chat with me? More photos with me here 👉 http://www.bit.ly/katekoxx
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

MySQL Replication Evolution -- Confoo Montreal 2017

  1. 1. MySQL Replication Evolution ConFoo Montreal 2017 Dave Stokes -- MySQL Community Manager david.stokes@oracle.com @stoker slideshare.net/davidmstokes Elephantanddolphin.blogger.com & opensourcedba.wordpress.com 1
  2. 2. Safe Harbor 2 THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. THE DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES OR FUNCTIONALITY DESCRIBED FOR ORACLE'S PRODUCTS REMAINS AT THE SOLE DISCRETION OF ORACLE.
  3. 3. 21 Years Old MySQL has been part of Oracle’s family of databases for six years. MySQL 8 MySQl 5.7 is the current release but the next version will be MySQL 8. Big feature is real time data dictionary Group Replication Active master-master replication. JSON A new native JSON datatype to store documents in a column of a table Document Store Programmers not know SQL but need a database? X Devapi allows them to use RDMS from language of choice Encryption Use Oracle Key Vault to encrypt your data at rest. 3
  4. 4. Replication Past Present Future 4
  5. 5. Basic Premise Behind Replication 1. Copy all the data on one server and copy onto another. 2. Setup Replication. 3. Log any changes on the first server to a log file, apply that logfile to the second server. 5
  6. 6. Imagine Your Data as a Painting: Make copy of Mona Lisa 6
  7. 7. Make copy of changes to Mona Lisa: Apply changes to copy 7
  8. 8. Keep making copy of changes to Mona Lisa: Apply changes made on master to slave 8
  9. 9. Architecture 10,000’ Binary Log on master Slave Attaches I/O Thread Grabs Data Data copied to slave Applier thread changes slave’s data 9
  10. 10. Graphic overview of replication 10
  11. 11. But what are we replicating? 11
  12. 12. Replicating changes to tables and data SBR: When using statement-based binary logging, the master writes SQL statements to the binary log. Replication of the master to the slave works by executing the SQL statements on the slave. This is called statement-based replication (often abbreviated as SBR), which corresponds to the standard MySQL statement-based binary logging format. Replication capabilities in MySQL version 5.1.4 and earlier used this format exclusively. What is sent: Structured Query Language (SQL), your query RBR: When using row-based logging, the master writes events to the binary log that indicate how individual table rows are changed. Replication of the master to the slave works by copying the events representing the changes to the table rows to the slave. This is called row-based replication (often abbreviated as RBR). What is sent: The Delta, the results of your query 12
  13. 13. OrBoth MBR: You can also configure MySQL to use a mix of both statement-based and row-based logging, depending on which is most appropriate for the change to be logged. This is called mixed-format logging. When using mixed-format logging, a statement-based log is used by default. Depending on certain statements, and also the storage engine being used, the log is automatically switched to row-based in particular cases. Replication using the mixed format is often referred to as mixed-based replication or mixed-format replication. 13
  14. 14. Notes for the future With MySQL.5.6 RBR started sending over only the primary key and the changed data (not sending unchanged data) which can drastically cut the amount of data sent to slave servers. This can be huge!! Many future products will work better with RBR as it more deterministic. So plan accordingly. 14
  15. 15. Threading Before 5.6 MySQL Replication is SINGLE threaded – Airline boarding example MySQL 5.6 is multi-threaded at the database level MySQL 5.7 is multi-threaded at the table level 15
  16. 16. Synchronicity 16
  17. 17. Async and Semi Synchronous Semi Synchronous replication -- a commit performed on the master blocks before returning to the session that performed the transaction until at least one slave acknowledges that it has received and logged the events for the transaction (Note not written to table, just recorded for future). Asynchronous replication -- slave servers retrieve data, master unaware of slave’s consumption. 17
  18. 18. 18
  19. 19. Topologies -- Before 5.7 19
  20. 20. Common - Read / Write Split 20
  21. 21. Multi-source Replication -- MySQL 5.7 21
  22. 22. Group Repilcation -- Labs.MySQL.Com for evaluation 22
  23. 23. Multi-Master Lots of folks want TWO active masters and this can be done but not recommended, You need to have a sharp crew and plan for conflicts. Not generally recommended before Group Replication 23
  24. 24. Multi-Master MySQL Cluster You can run active-active master-master with MySQL Cluster, even between data centers. This can be very expensive and MySQl Cluster is not a full featured version of the MySQL Server. 24
  25. 25. Galera Cluster Layer separate from MySQL that is mainly for high availability (not high performance). Claims to have snapshot isolation on transactions but watch out for ‘first committer wins’ and prepare for rollbacks. Not low latency 25
  26. 26. How to setup MySQL Replication 26
  27. 27. Two types of replication w/ & w/o GTIDs A global transaction identifier (GTID) is a unique identifier created and associated with each transaction committed on the server of origin (master). This identifier is unique not only to the server on which it originated, but is unique across all servers in a given replication setup. There is a 1- to-1 mapping between all transactions and all GTIDs. 3E11FA47-71CA-11E1-9E33-C80AA9429562:1-5 Note the 1-5 is a group of transactions 27
  28. 28. Binlog & ID Enable Binary Log on Master, Unique ID number Create User Create user for replication Binlog Offset Get Master’s Binary Log coordinates Snapshot Snapshot data, copy onto slave SLAVE running CHANGE MASTER command and START SLAVE Before GTIDs (MySQL 5.5 and before) 28
  29. 29. Enable Binary Log & Unique ID on Master Edit my.cnf file [mysqld] log-bin=mysql-bin server-id=1 29
  30. 30. Create replication user mysql> CREATE USER 'repl'@'%.mydomain.com' IDENTIFIED BY 'slavepass'; mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%.mydomain.com'; 30
  31. 31. Get binary log position mysql> FLUSH TABLES WITH READ LOCK; mysql > SHOW MASTER STATUS; +------------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +------------------+----------+--------------+------------------+ | mysql-bin.000003 | 73 | test | manual,mysql | +------------------+----------+--------------+------------------+ 31
  32. 32. Unlock tables mysql> UNLOCK TABLES; 32
  33. 33. Config slave & load data No config thee slave server. Remember server-id must be unique [mysqld] server-id=2 shell> mysql -h master < fulldb.dump 33
  34. 34. Change Master & Start Slave mysql> CHANGE MASTER TO -> MASTER_HOST='master_host_name', -> MASTER_USER='replication_user_name', -> MASTER_PASSWORD='replication_password', -> MASTER_LOG_FILE='recorded_log_file_name', -> MASTER_LOG_POS=recorded_log_position; mysql> START SLAVE; 34
  35. 35. Binlog & ID Enable Binary Log on Master, Unique ID number Create User Create user for replication Snapshot Snapshot data, copy onto slave SLAVE running CHANGE MASTER command and START SLAVE SLAVE running GTIDs make it much easier to automate many functions With GTIDs (MySQL 5.6 & Later) 35
  36. 36. Replication Setup with GTIDs mysql>mysqladmin -uusername -p shutdown shell> mysqld --gtid-mode=ON --enforce- gtid-consistency & mysql> CHANGE MASTER TO > MASTER_HOST = host, > MASTER_PORT = port, > MASTER_USER = user, > MASTER_PASSWORD = password, > MASTER_AUTO_POSITION = 1; mysql> START SLAVE; 36
  37. 37. MySQL InnoDB Cluster 37
  38. 38. MySQL InnoDB Cluster 38 MySQL InnoDB cluster is a collection of products that work together to provide a high availability solution. A group of MySQL servers can be configured to create a cluster using MySQL Shell. The cluster of servers has a single master, called the primary, which acts as the read-write master. Multiple secondary servers are replicas of the master. A minimum of three servers are required to create a high availability cluster. A client application is connected to the primary via MySQL Router. If the primary fails, a secondary is automatically promoted to the role of primary, and MySQL Router routes requests to the new primary.
  39. 39. Components MySQL Shell 1.0.8 or higher. Includes the AdminAPI, which enables you to create and administer an InnoDB cluster, using either JavaScript or Python scripting. MySQL Shell also requires Python 2.7 and above to run cluster provisioning scripts. MySQL Router 2.1.2 or higher. Caches the metadata of the InnoDB cluster and performs high availability routing to the MySQL Server instances which make up the cluster. If the primary instance becomes unavailable, MySQL Router automatically routes client requests to a promoted secondary (the new primary). MySQL Server 5.7.17 or higher. This provides the Group Replication mechanism to allow data to be replicated from the primary to the secondaries in the cluster. 39
  40. 40. 40
  41. 41. 41 MySQL Router Connection routing is the ability to permit the redirection of connections sent to the router to an available MySQL server. Connection routing is simply the redirection of the MySQL packets in their entirety without inspection. This means you can set up your application to connect to the router and should the current MySQL server fail, retry the connection and the router will select a new MySQL server to redirect the connection. So check you return codes, retry on failure
  42. 42. Configuration [routing:simple_redirect] bind_port = 7002 mode = read-write destinations = localhost:3306,localhost:3307,localhost:3308 42
  43. 43. Group Replication MySQL Group Replication provides distributed state machine replication with strong coordination between servers. Servers coordinate themselves automatically when they are part of the same group. The group can operate in a single-primary mode with automatic primary election, where only one server accepts updates at a time. Alternatively, for more advanced users the group can be deployed in multi-primary mode, where all servers can accept updates, even if they are issued concurrently. This power comes at the expense of applications having to work around the limitations imposed by such deployments. 43
  44. 44. Group Replication There is a built-in group membership service that keeps the view of the group consistent and available for all servers at any given point in time. Servers can leave and join the group and the view is updated accordingly. Sometimes servers can leave the group unexpectedly, in which case the failure detection mechanism detects this and notifies the group that the view has changed. This is all automatic. 44
  45. 45. Group Replication For a transaction to commit, the majority of the group have to agree on the order of a given transaction in the global sequence of transactions. Deciding to commit or abort a transaction is done by each server individually, but all servers make the same decision. If there is a network partition, resulting in a split where members are unable to reach agreement, then the system does not progress until this issue is resolved. Hence there is also a built-in, automatic, split-brain protection mechanism. 45
  46. 46. Group Replication All of this is powered by the provided group communication protocols. These provide a failure detection mechanism, a group membership service, and safe and completely ordered message delivery. All these properties are key to creating a system which ensures that data is consistently replicated across the group of servers. At the very core of this technology lies an implementation of the Paxos algorithm. It acts as the group communication systems engine. 46
  47. 47. Group Replication All read-write (RW) transactions commit only after they have been approved by the group. Read-only (RO) transactions need no coordination within the group and thus commit immediately. For any RW transaction the group needs to decide whether it commits or not, thus the commit operation is not a unilateral decision from the originating server. When a transaction is ready to commit at the originating server, the server atomically broadcasts the write values (rows changed) and the correspondent write set (unique identifiers of the rows that were updated). Then a global total order is established for that transaction. Ultimately, this means that all servers receive the same set of transactions in the same order. As a consequence, all servers apply the same set of changes in the same order, therefore they remain consistent within the group. 47
  48. 48. 48
  49. 49. InnoDB Cluster How-to 49 http://lefred.be/content/mysql-innodb-cluster-mysql-shell-starter-guide/
  50. 50. 30 Minutes!!! Can’t Cover all of MySQL Replication in ~60 minutes!!!!! 50Or can I?????
  51. 51. https://github.com/datacharmer Giuseppe Maxia has lot of great code for getting used to replication -- see ~/mysql-replication-sample 51 Great Place to Find examples & MySQL Sandbox
  52. 52. MySQL Utilities FREE Scripts written in Python, used with MySQL Workbench or stand alone 1.Copy, diff databases 2.Disk usage, grants, copy users 3.Search for processed and kill ‘em 4.Setup replication and failover 52
  53. 53. Mysqlrplcheck -- check replication setup shell> mysqlrplcheck --master=root@host1:3310 --slave=root@host2:3311 # master on host1: ... connected. # slave on host2: ... connected. Test Description Status ------------------------------------------------------------------------ Checking for binary logging on master [pass] Are there binlog exceptions? [pass] Replication user exists? [pass] Checking server_id values [pass] Is slave connected to master? [pass] Check master information file [pass] Checking InnoDB compatibility [pass] Checking storage engines compatibility [pass] Checking lower_case_table_names settings [pass] Checking slave delay (seconds behind master) [pass] # ...done. 53
  54. 54. Mysqlrplcheck -- replication checker shell> mysqlrplsync --master=user:pass@localhost:3310 --slaves=rpl:pass@localhost:3311,rpl:pass@localhost:3312 # # GTID differences between Master and Slaves: # - Slave 'localhost@3311' is 15 transactions behind Master. # - Slave 'localhost@3312' is 12 transactions behind Master. # # Checking data consistency. # # Using Master 'localhost@3310' as base server for comparison. # Checking 'test_rplsync_db' database... # - Checking 't0' table data... # [OK] `test_rplsync_db`.`t0` checksum for server 'localhost@3311'. # [OK] `test_rplsync_db`.`t0` checksum for server 'localhost@3312'. # - Checking 't1' table data... # [OK] `test_rplsync_db`.`t1` checksum for server 'localhost@3311'. # [OK] `test_rplsync_db`.`t1` checksum for server 'localhost@3312'. # Checking 'test_db' database... 54
  55. 55. Mysqlslavetrx -- skip bad transactions shell> mysqlslavetrx --gtid-set=af6b22ee-7b0b-11e4-aa8d-606720440b68:7-9 --slaves=user:pass@localhost:3311,user:pass@localhost:3312 WARNING: Using a password on the command line interface can be insecure. # # GTID set to be skipped for each server: # - localhost@3311: af6b22ee-7b0b-11e4-aa8d-606720440b68:7-9 # - localhost@3312: af6b22ee-7b0b-11e4-aa8d-606720440b68:7-9 # # Injecting empty transactions for 'localhost:3311'... # Injecting empty transactions for 'localhost:3312'... # #...done. 55
  56. 56. MySQL Replication Evolution Q/A david.stokes@oracle.com @stoker slideshare.net/davidmstokes Elephantanddolphin.blogger.com & opensourcedba.wordpress.com 56

×