Moving Data is Key to Web Success

MOVING DATA FOR THE MASSES

Giuseppe Maxia
QA Director
Continuent, Inc
@datacharmer
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Friday, November 11, 11 1

Databases are the backbone of the web economy

www

$$$


Actually, database replication is the backbone of
the web economy


Actually, database replication is the backbone of
the web economy

WHY?

Why moving data?

• Scaling reads (load balancing)
• Scaling writes (sharding)
• Parallelize computation (sharding+map/reduce)
• High availability (replacing master on failure)
• Creating information (feeding data warehouses)


Scaling reads

M

S S

S


Scaling writes
(sharding)
M1 M3

M2

S S S S

S S

High availability

M

S S

S


High availability

S
✘ M

S

S


High availability

S S

M


Creating new
information

S

M M

M

Creating new
information
S S

M S

M M

M

More about data
replication


database server

r/w requests

web server

clients

database server

r/w requests

web servers

load balancer

clients

database load
on a simple
web
application
r
e
write a
d


write read

database load on a
successful web
application

database server

✘ r/w requests

web servers

load balancer

clients


read/write
master a web
application
read/only
slaves
scheme with
replication
load balancer
R/W

R/O

web servers

load balancer

clients


r
e
write a read
d

read/write
master

read/only
slaves

database load
with
replication


r
e
write a read
d

read/write
master

read/only
slaves

scaling
database load
with
replication

• If you want a successful
company


company
• in the 21st century


company
• you need to move data


company
• fast


company
• fast
• and often


MOVING DATA
FOR
THE MASSES


How do you move
data?


How do you move
data?
• Commodity hardware


How do you move
data?
• Basic
• Commodity software (MySQL)
• MySQL native replication


How do you move
data?
• Basic
• Commodity software (MySQL)
• MySQL native replication
• Advanced
• MySQL and other DBMS
• Powerful tools (Tungsten Replicator)

Why Tungsten?


transaction
transaction
MySQL transaction
transaction
transaction
DBMS transaction
transaction
transaction
transaction
transaction
transaction
transaction

BINARY LOG

ns act act tio n
tra ansio ns sac ntio
ac ion ion n
ac
nr tr tra aniton
tsa a n csio
nts ct n
rn a o
taa scti
r
trnsa

n
n
MySQL replication
tio
ct
tra

tra
REPLICATION
transaction
transaction
is single threaded

MySQL
DBMS


master master master master
MySQL MySQL MySQL MySQL
DBMS DBMS DBMS DBMS

MySQL MySQL MySQL MySQL
DBMS DBMS DBMS DBMS

slave slave slave slave

single source multi source (fan-in)

multiple sources?

from this to this
MySQL MySQL
DBMS DBMS

master master

MySQL MySQL
DBMS DBMS

master master

multiple masters?


INSERT INSERT
RECORD RECORD
A A
MySQL MySQL
DBMS DBMS

master master

MySQL MySQL
DBMS DBMS

MODIFY master master MODIFY
RECORD RECORD
B B
Avoiding conﬂicts?


master
Seamless failover?
MySQL
DBMS

master
MySQL
DBMS

master
MySQL
DBMS

MySQL MySQL
DBMS DBMS

slave slave MySQL
DBMS
MySQL
DBMS

slave slave
MySQL MySQL
DBMS DBMS

slave slave

Replicating to something else?

mysql master
MySQL
DBMS

MySQL
DBMS

mysql postgresql oracle mongodb


All these examples tell
us:

Nice dream, but
MySQL can’t do it

Enter Tungsten Replicator


http://code.google.com/p/tungsten-replicator



Open Source



Open Source
100% GPL v2


What can it do?
• Easy failover
• Multiple masters
• Multiple sources to a single slave
• Conﬂict prevention
• Parallel replication
• Replicate to Oracle and PostgreSQL database

MySQL to foreign services

MySQL setup
to run as
MySQL master

Master DB
Data is applied to
PostgreSQL or
Oracle or to NoSQL
entities

Bin
Logs
(Binlogs
enabled) Slave DB

© Continuent 2010


Main components
• Transaction History Logs (THL)
• roughly corresponding to MySQL relay logs

• have a lot of metadata

• Service database
• contains metadata for latest transactions

• Metadata is committed together with data

• Makes slaves crash proof


A failover scenario
1: MySQL native replication


Deﬁning the problem
Slave 3

Master Replication Slave 2

Slave 1
Binary log

Transactions Relay log Binary log
--------
-------- Transactions Transactions
-------- -------- ========
-------- -------- ========
-------- ========
-------- ========


The master crashes
Slave 3


Slave 1
Binary log

Transactions Relay log Binary log
--------
-------- Transactions Transactions
-------- -------- ========
-------- -------- ========
-------- ========
-------- ========


Where do the slaves
stand?
Slave 1 Slave 2 Slave 3

Binary log Binary log Binary log

Transactions Transactions Transactions
======== ======== ========
======== ======== ========
======== ======== ========
======== ======== ========


Slave 3 becomes
master
New master

Slave 1 Slave 2
Binary log

Transactions
========
Binary log Binary log ========
========
Transactions Transactions ========
======== ========
======== ========
======== ========
======== ========


Need to synch the
missing transactions
New master

Slave 1 Slave 2
Binary log

Transactions
========
========
======== ========
======== ========
======== ========
======== ========


Transaction position in the new
master binlog may not match
positions in the old master binlog
New master

Slave 1 Slave 2
Binary log

Transactions
? ? ========
========
======== ========
======== ========
======== ? ======== ?
======== ========


A failover scenario
1: Tungsten Replicator


Slave 3


Slave 1
THL

Transactions THL
--------
-------- Transactions
-------- ========
-------- ========
========
========

Includes Global Transaction ID


The master crashes
Slave 3


Slave 1
THL

Transactions THL
--------
-------- Transactions
-------- ========
-------- ========
========
========


Where do the slaves
stand?
Slave 1 Slave 2 Slave 3

THL THL THL

Transactions Transactions Transactions
======== ======== ========
======== ======== ========
======== ======== ========
======== ======== ========


Slave 3 becomes
master
New master

Slave 1 Slave 2
THL

Transactions
========
THL THL ========
========
======== ========
======== ========
======== ========
======== ========


Need to synch the
missing transactions
New master

Slave 1 Slave 2
THL

Transactions
========
THL THL ========
========
======== ========
======== ========
======== ========
======== ========


Transaction IDs in the new master
THL are understood immediately by
the slaves
New master

Slave 1 Slave 2
Binary log

Transactions
========
========
======== ========
======== ========
======== Give me ======== Give me
======== seqno #51 ======== seqno #67


Parallel replication


parallel replication
From here ...


To here.


Parallel replication facts

✓Sharded by database
✓Good choice for slave lag problems
❖Bad choice for single database projects


Benchmarking parallel
replication


db0
preparation (1)
db1 sysbench

db2 sysbench

db3 sysbench

db4 sysbench

db5 sysbench

db6 sysbench

db7 sysbench

db8 sysbench

db9 sysbench

sysbench


before the test (2)

STOPPED

binary logs

MySQL slave

OFFLINE

Tungsten slave direct:
alpha
(slave)

replicator alpha


starting the test

STOPPED

binary logs

MySQL slave Concurrent sysbench
on 30 databases
running for 1 hour
OFFLINE
TOTAL DATA: 130 GB
alpha
RAM per server: 20GB
(slave)

replicator alpha

Slaves will have 1 hour lag

measuring results

START

binary logs

MySQL slave

ONLINE Recording
catch-up time
alpha
(slave)

replicator alpha


MySQL native
replication

slave catch up in 04:29:30


Tungsten parallel
replication

slave catch up in 00:55:40


Parallel replication made simpler

FROM HERE ....


TO HERE

direct slave facts


direct slave facts
✓No need to install Tungsten on the master


direct slave facts
✓Tungsten runs only on the slave


direct slave facts
✓Replication can revert to native slave with two
commands (trepctl offline; start
slave)


direct slave facts
slave)

✓Native replication can continue on other slaves


direct slave facts
slave)

✓Native replication can continue on other slaves
❖Failover (either native or Tungsten) becomes a
manual task

Multiple masters


multiple masters

• Tungsten Replicator recipe: use more
services


Bi-directional replication


Bi-directional replication with slaves


Multiple sources


multiple sources

• Tungsten Replicator recipe is still valid:
use more services


Multiple source replication


Multiple masters replication: 3 nodes


Multiple masters replication: 4 nodes


4 nodes multiple masters simpliﬁed


star topologies


Conflict prevention


Conﬂict prevention
facts


facts
• Sharded by database


facts
• Defined dynamically


facts
• Applied either at the master or at the
slave


facts
slave
• methods:


facts
slave
• methods:
• make replication fail


facts
slave
• methods:

• drop silently


facts
slave
• methods:

• drop silently

• drop with warning

Tungsten Conﬂict prevention points
host1
host3

M S
S

host2 host4
master

S M S


Prevention methods: Fail on master
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
Prevention methods: Fail on master
host1
host3

M S
S

host2 host4
master

S M S
INSERT A
x,z

Prevention methods: Fail on slave
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
host1
host3

M S
S

host2 host4
master

S M S
INSERT A
x,z

host1
(Multiple sources)

M

host2 host3
master

M S


INSERT A
x,y
host1
(Multiple sources)

M

host2 host3
master

M S
INSERT A
x,z

Prevention methods: DROP on master
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
host1
host3

M S
S

host2 host4
master

S M S


INSERT A
x,y
host1
host3

M S
S

host2 host4
master

S M S
INSERT A
x,z

host1
host3

M S
S

host2 host4
master

S M S
INSERT A
x,z

Prevention methods: DROP on slave
host1
(Multiple sources)

M

host2 host3
master

M S


INSERT A
x,y
host1
(Multiple sources)

M

host2 host3
master

M S
INSERT A
x,z

host1
(Multiple sources)

M

host2 INSERT A host3
x,y
master
INSERT A
M x,z S


host1
(Multiple sources)

M

host2 host3
master
INSERT A
M x,z S


Tungsten in practice
Installation


Installation

• Check the requirements
• Get the binaries
• Expand the tarball
• Run ./tools/tungsten-installer


REQUIREMENTS

• Java JRE or JDK (Sun/Oracle or Open-jdk)
• Ruby 1.8 (only during installation)
• ssh access to the same user in all nodes
• MySQL user with all privileges


Installation types

• master / slave
• slave - direct


master/slave
host2
host1

master slave

THL
binlog THL

host3

slave

THL


slave direct
host2
host1
slave
master
relay log

THL
binlog
host3

slave

relay log

THL


sample installation
• 4 servers
• qa.r1.continuent.com r1




• MySQL 5.1
• tungsten-replicator-2.0.4

Installation (1)
# starting at node 4, but any would do

for N in 1 2 3 4
do
ssh r$N mkdir tinstall
done

cd tinstall
tar -xzf /path/to/tungsten-replicator-2.0.4.tar.gz
cd tungsten-replicator-2.0.4


Installation (2)
export TUNGSTEN_BASE=$HOME/tinstall

1 ./tools/tungsten-installer
2 --master-slave
3 --master-host=r1
4 --datasource-user=tungsten
5 --datasource-password=secret
6 --service-name=dragon
7 --home-directory=$TUNGSTEN_BASE
8 --cluster-hosts=r1,r2,r3,r4
9 --start


Installation (2)
export TUNGSTEN_BASE=$HOME/tinstall

./tools/tungsten-installer
--master-slave # installation mode
--master-host=r1 # who’s the master
--datasource-user=tungsten # mysql username
--datasource-password=secret # mysql password
--service-name=dragon # name of the service
--home-directory=$TUNGSTEN_BASE # where we install
--cluster-hosts=r1,r2,r3,r4 # hosts in cluster
--start # start replicator after installing


What does the
installation do
1: Validate all servers

host4 host1 host2 host3

✔ ✔ ✔ ✔
✗ ✗ ✗ ✗
✔ ✔ ✔ ✔
✔ ✔ ✔ ✔

Report all errors

What does the
installation do
1 (again): Validate all servers

host4 host1 host2 host3

✔ ✔ ✔ ✔
✔ ✔ ✔ ✔
✔ ✔ ✔ ✔
✔ ✔ ✔ ✔


What does the
installation do
2: install Tungsten in all servers
host4
$HOME/ host1
tinstall/ host2
config/ host3
releases/
relay/
logs/
tungsten/


example
ssh r2 chmod 444 $HOME/tinstall
--master-slave --master-host=r1
--datasource-user=tungsten
--datasource-password=secret
--service-name=dragon
--home-directory=$HOME/tinstall
--thl-directory=$HOME/tinstall/logs
--relay-directory=$HOME/tinstall/relay
--cluster-hosts=r1,r2,r3,r4 --start

ERROR >> qa.r2.continuent.com >> /home/tungsten/
tinstall is not writeable


example
ssh r2 chmod 755 $HOME/tinstall
--master-slave --master-host=r1
--datasource-user=tungsten
--datasource-password=secret
--service-name=dragon
--home-directory=$HOME/tinstall
--thl-directory=$HOME/tinstall/logs
--relay-directory=$HOME/tinstall/relay
--cluster-hosts=r1,r2,r3,r4 --start

# no errors


Installation - direct
--direct
--master-host=r1
--slave-host=r4
--master-user=tungsten
--slave-user=tungsten
--master-password=secret
--slave-password=secret
--service-name=Castor
--channels=1
--home-directory=$TUNGSTEN_BASE
--start-and-report


http://bit.ly/tr20_cookbook


Tungsten Replicator vs. Enterprise

Feature Replicator Enterprise
Simple admin with trepctl X X
MySQL Row/Statement 5.0/5.1/5.5 X X
Multi-master replication X X
Parallel apply (2.0) X X
Distributed management with cctrl X
Automated failover and recovery X
Zero-downtime upgrade X
SQL routing and load-balancing X
Floating IP support X

© Continuent 2011


Conclusion and Q&A


Contact Information

Worldwide
560 S. Winchester Blvd., Suite 500
San Jose, CA 95128 @datacharmer
Tel (866) 998-3642
Fax (408) 668-1009 @continuent
e-mail: sales@continuent.com

Continuent Web Site:
http://www.continuent.com
Tungsten Project
http://tungsten-replicator.org
Getting started with Tungsten Replicator:
http://datacharmer.blogspot.com
© Continuent 2011


Moving Data is Key to Web Success

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (10)

Mais de Giuseppe Maxia

Mais de Giuseppe Maxia (20)

Último

Último (20)

Moving Data is Key to Web Success