Mysql replication outside the box

MySQL replication outside the box

Giuseppe Maxia
QA Director
Continuent, Inc
@datacharmer
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Tuesday, October 25, 11 1

about me -Giuseppe Maxia
§ a.k.a. The Data Charmer
§ QA Director, Continuent, Inc
§ Long time hacking with database features
§ Formerly, database consultant, designer, coder.
§ A passion for QA
§ An even greater passion for open source
§ ... and community
§ Passionate blogger
§ http://datacharmer.blogspot.com


Why Tungsten?


transaction
transaction
MySQL transaction
transaction
transaction
DBMS transaction
transaction
transaction
transaction
transaction
transaction
transaction

BINARY LOG

ns act act tio n
tra ansio ns sac ntio
ac ion ion n
ac
nr tr tra aniton
tsa a n csio
nts ct n
rn a o
taa scti
r
trnsa

n
n
MySQL replication
tio
ct
tra

tra
REPLICATION
transaction
transaction
is single threaded

MySQL
DBMS


master master master master
MySQL MySQL MySQL MySQL
DBMS DBMS DBMS DBMS

MySQL MySQL MySQL MySQL
DBMS DBMS DBMS DBMS

slave slave slave slave

single source multi source (fan-in)

multiple sources?

from this to this
MySQL MySQL
DBMS DBMS

master master

MySQL MySQL
DBMS DBMS

master master

multiple masters?


INSERT INSERT
RECORD RECORD
A A
MySQL MySQL
DBMS DBMS

master master

MySQL MySQL
DBMS DBMS

MODIFY master master MODIFY
RECORD RECORD
B B
Avoiding conﬂicts?


master
Seamless failover?
MySQL
DBMS

master
MySQL
DBMS

master
MySQL
DBMS

MySQL MySQL
DBMS DBMS

slave slave MySQL
DBMS
MySQL
DBMS

slave slave
MySQL MySQL
DBMS DBMS

slave slave

Replicating to something else?

mysql master
MySQL
DBMS

MySQL
DBMS

mysql postgresql oracle mongodb


All these examples tell
us:

Nice dream, but
MySQL can’t do it

Enter Tungsten Replicator


http://code.google.com/p/tungsten-replicator



Open Source



Open Source
100% GPL v2


What can it do?
• Easy failover
• Multiple masters
• Multiple sources to a single slave
• Conﬂict prevention
• Parallel replication
• Replicate to Oracle and PostgreSQL database

MySQL to foreign services

MySQL setup
to run as
MySQL master

Master DB
Data is applied to
PostgreSQL or
Oracle or to NoSQL
entities

Bin
Logs
(Binlogs
enabled) Slave DB

© Continuent 2010


Tungsten Replicator in
a nutshell
host1
host2
master
slave

binlog THL
global
transaction ID THL

trep_commit_seqno
origin seqno eventid
trep_commit_seqno
origin seqno eventid


Main components
• Transaction History Logs (THL)
• roughly corresponding to MySQL relay logs

• have a lot of metadata

• Service database
• contains metadata for latest transactions

• Metadata is committed together with data

• Makes slaves crash proof


A failover scenario
1: MySQL native replication


1. one Master, two slaves

/ Loading the “employees” test database

© Continuent 2011


2. Master goes away

/ Stop replication

/ Slaves are updated at different levels

# 2
select count(*) from titles
333,145

# 3
443,308

© Continuent 2011


3. Look into Slave #2 binary logs

/ find the last transaction

© Continuent 2011


4. Look into Slave #3 binary logs

1. find the transaction that was last in slave #2
2. Recognize that last transaction in the log of slave #3
(This can actually take you a LOOOONG TIME)
3. Get the position immediately after this transaction
4. (e.g. 134000 in file mysql-bin.000018)

© Continuent 2011


5. promote Slave #3 to master

/ in slave #2

CHANGE MASTER TO
master_host=‘slave_3_IP’,
master_user=‘slavename’,
master_password=‘slavepassword’,
master_log_file=‘mysql-bin.000018’,
master_log_pos=134000;

© Continuent 2011


A failover scenario
1: Tungsten Replicator


1. one master, two slaves

/ loading the ‘employees’ test database

© Continuent 2011


2. Master goes away

/ Stop replication

/ Slaves are updated at different levels

# 2
333,145

# 3
443,308

© Continuent 2011


3. no need to find the last transaction

/ simply change roles

trepctl -host slave3 setrole -role master
trepctl -host slave2 setrole -role slave
-uri thl://slave3

trepctl -host slave3 online
State: ONLINE

trepctl -host slave2 online
State: GOING-ONLINE:SYNCHRONIZING

© Continuent 2011


4. Check that the slave has synchronized

# new master
select seqno from tungsten.trep_commit_seqno;
78

# new slave
64

© Continuent 2011


4. Tell the replicator to hurry up

# new master
trepctl -node slave3 flush
Master log is synchronized with database at
log sequence number: 78

# new slave
trepctl host slave2 wait -applied 78
ONLINE
78

© Continuent 2011


4. ... and we’re done
# new master
select count(*) from employees.titles
count(*)
443308

# new slave:
count(*)
443308

© Continuent 2011


Parallel replication


parallel replication
From here ...


To here.


Parallel replication facts

✓Sharded by database
✓Good choice for slave lag problems
❖Bad choice for single database projects


Benchmarking parallel
replication


db0
preparation (1)
db1 sysbench

db2 sysbench

db3 sysbench

db4 sysbench

db5 sysbench

db6 sysbench

db7 sysbench

db8 sysbench

db9 sysbench

sysbench


before the test (2)

STOPPED

binary logs

MySQL slave

OFFLINE

Tungsten slave direct:
alpha
(slave)

replicator alpha


starting the test

STOPPED

binary logs

MySQL slave Concurrent sysbench
on 30 databases
running for 1 hour
OFFLINE
TOTAL DATA: 130 GB
alpha
RAM per server: 20GB
(slave)

replicator alpha

Slaves will have 1 hour lag

measuring results

START

binary logs

MySQL slave

ONLINE Recording
catch-up time
alpha
(slave)

replicator alpha


MySQL native
replication

slave catch up in 04:29:30


Tungsten parallel
replication

slave catch up in 00:55:40


Parallel replication made simpler

FROM HERE ....


TO HERE

direct slave facts


direct slave facts
✓No need to install Tungsten on the master


direct slave facts
✓Tungsten runs only on the slave


direct slave facts
✓Replication can revert to native slave with two
commands (trepctl offline; start
slave)


direct slave facts
slave)

✓Native replication can continue on other slaves


direct slave facts
slave)

✓Native replication can continue on other slaves
❖Failover (either native or Tungsten) becomes a
manual task

Multiple masters


multiple masters

• Tungsten Replicator recipe: use more
services


Bi-directional replication


Bi-directional replication with slaves


Multiple sources


multiple sources

• Tungsten Replicator recipe is still valid:
use more services


Multiple source replication


Multiple masters replication: 3 nodes


Multiple masters replication: 4 nodes


Conflict prevention


Conﬂict prevention
facts


facts
• Sharded by database


facts
• Defined dynamically


facts
• Applied either at the master or at the
slave


facts
slave
• methods:


facts
slave
• methods:
• make replication fail


facts
slave
• methods:

• drop silently


facts
slave
• methods:

• drop silently

• drop with warning

Tungsten Conﬂict prevention points
host1
host3

M S
S

host2 host4
master

S M S


Prevention methods: Fail on master
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
Prevention methods: Fail on master
host1
host3

M S
S

host2 host4
master

S M S
INSERT A
x,z

Prevention methods: Fail on slave
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
host1
host3

M S
S

host2 host4
master

S M S
INSERT A
x,z

host1
(Multiple sources)

M

host2 host3
master

M S


INSERT A
x,y
host1
(Multiple sources)

M

host2 host3
master

M S
INSERT A
x,z

Prevention methods: DROP on master
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
host1
host3

M S
S

host2 host4
master

S M S
INSERT A
x,z

host1
host3

M S
S

host2 host4
master

S M S
INSERT A
x,z

Prevention methods: DROP on slave
host1
(Multiple sources)

M

host2 host3
master

M S


INSERT A
x,y
host1
(Multiple sources)

M

host2 host3
master

M S
INSERT A
x,z

host1
(Multiple sources)

M

host2 INSERT A host3
x,y
master
INSERT A
M x,z S


host1
(Multiple sources)

M

host2 host3
master
INSERT A
M x,z S


Tungsten in practice
Installation


Installation

• Check the requirements
• Get the binaries
• Expand the tarball
• Run ./tools/tungsten-installer


REQUIREMENTS

• Java JRE or JDK (Sun/Oracle or Open-jdk)
• Ruby 1.8 (only during installation)
• ssh access to the same user in all nodes
• MySQL user with all privileges


Installation types

• master / slave
• slave - direct


master/slave
host2
host1

master slave

THL
binlog THL

host3

slave

THL


slave direct
host2
host1
slave
master
relay log

THL
binlog
host3

slave

relay log

THL


sample installation
• 4 servers
• qa.r1.continuent.com r1




• MySQL 5.1
• tungsten-replicator-2.0.4

Installation (1)
# starting at node 4, but any would do

for N in 1 2 3 4
do
ssh r$N mkdir tinstall
done

cd tinstall
tar -xzf /path/to/tungsten-replicator-2.0.4.tar.gz
cd tungsten-replicator-2.0.4


Installation (2)
export TUNGSTEN_BASE=$HOME/tinstall

1 ./tools/tungsten-installer
2 --master-slave
3 --master-host=r1
4 --datasource-user=tungsten
5 --datasource-password=secret
6 --service-name=dragon
7 --home-directory=$TUNGSTEN_BASE
8 --cluster-hosts=r1,r2,r3,r4
9 --start


Installation (2)
export TUNGSTEN_BASE=$HOME/tinstall

./tools/tungsten-installer
--master-slave # installation mode
--master-host=r1 # who’s the master
--datasource-user=tungsten # mysql username
--datasource-password=secret # mysql password
--service-name=dragon # name of the service
--home-directory=$TUNGSTEN_BASE # where we install
--cluster-hosts=r1,r2,r3,r4 # hosts in cluster
--start # start replicator after installing


What does the
installation do
1: Validate all servers

host4 host1 host2 host3

✔ ✔ ✔ ✔
✗ ✗ ✗ ✗
✔ ✔ ✔ ✔
✔ ✔ ✔ ✔

Report all errors

What does the
installation do
1 (again): Validate all servers

host4 host1 host2 host3

✔ ✔ ✔ ✔
✔ ✔ ✔ ✔
✔ ✔ ✔ ✔
✔ ✔ ✔ ✔


What does the
installation do
2: install Tungsten in all servers
host4
$HOME/ host1
tinstall/ host2
config/ host3
releases/
relay/
logs/
tungsten/


example
ssh r2 chmod 444 $HOME/tinstall
--master-slave --master-host=r1
--datasource-user=tungsten
--datasource-password=secret
--service-name=dragon
--home-directory=$HOME/tinstall
--thl-directory=$HOME/tinstall/logs
--relay-directory=$HOME/tinstall/relay
--cluster-hosts=r1,r2,r3,r4 --start

ERROR >> qa.r2.continuent.com >> /home/tungsten/
tinstall is not writeable


example
ssh r2 chmod 755 $HOME/tinstall
--master-slave --master-host=r1
--datasource-user=tungsten
--datasource-password=secret
--service-name=dragon
--home-directory=$HOME/tinstall
--thl-directory=$HOME/tinstall/logs
--relay-directory=$HOME/tinstall/relay
--cluster-hosts=r1,r2,r3,r4 --start

# no errors


Installation - direct
--direct
--master-host=r1
--slave-host=r4
--master-user=tungsten
--slave-user=tungsten
--master-password=secret
--slave-password=secret
--service-name=Castor
--channels=1
--home-directory=$TUNGSTEN_BASE
--start-and-report


Tungsten in practice
Admin


Main components

• THL (Tungsten generated Transaction
History Log)
• service database


Tools

• replicator
• trepctl
• thl


replicator

• It’s the service provider
• You launch it once when you start
• You may restart it when you change
config


trepctl

• Tungsten Replicator ConTroLler
• It’s the driving seat for your replication
• You can start, update, and stop services
• You can get specific info


thl

• Transaction History List
• Gives you access to the Tungsten relay
logs


http://bit.ly/tr20_cookbook


Tungsten Replicator vs. Enterprise

Feature Replicator Enterprise
Simple admin with trepctl X X
MySQL Row/Statement 5.0/5.1/5.5 X X
Multi-master replication X X
Parallel apply (2.0) X X
Distributed management with cctrl X
Automated failover and recovery X
Zero-downtime upgrade X
SQL routing and load-balancing X
Floating IP support X

© Continuent 2011


Conclusion and Q&A


Contact Information

Worldwide
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128 @datacharmer
Tel (866) 998-3642
Fax (408) 668-1009 @continuent
e-mail: sales@continuent.com

Continuent Web Site:
http://www.continuent.com
Tungsten Project
http://tungsten-replicator.org
Getting started with Tungsten Replicator:
http://datacharmer.blogspot.com
© Continuent 2011


Mysql replication outside the box

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Mais de Giuseppe Maxia

Mais de Giuseppe Maxia (20)

Último

Último (20)

Mysql replication outside the box