2. How to shard database
without stopping the service
3. How to shard database
What is database sharding?
When the data volume increases or the updating traffic
increases, your updating database server cannot process
effectively.
We often use the technique for dividing data into two or
more databases to solve the problem. This is database
sharding.
Here, I will explain how to shard a data,
without stopping the service.
4. Initial Structure
tbl_a
Create table tbl_a (
col_a int, DB1
col_b int,
primary key(col_a)
) engine = InnoDB;
There is 1 MySQL server without Spider.
11. How to re-shard database
What is re-sharding?
When the data volume increases or the updating traffic
increases so much, even if you had your database sharded,
your updating database server cannot process right again.
So we solve that problem by increasing the number of
servers and distributing the load.
It is called re-sharding to increase the number of servers,
and to distribute the load.
Here, I will explain how to re-shard
without stopping the service.
12. Initial Structure
col_a%2=0 col_a%2=1
Create table tbl_a (
col_a int,
col_b int,
primary key(col_a)
tbl_a tbl_a
) engine = Spider
Connection â
DB2 DB3
table âtbl_aâ,
user âuserâ,
password âpassâ
â
partition by list(
mod(col_a, 2)) ( tbl_a
partition pt1 values in(0)
comment âhost âDB2ââ,
partition pt2 values in(1)
DB1
comment âhost âDB3ââ
);
There are 1 MySQL server with Spider and 2 remote
MySQL servers without Spider.
18. How to add an index
without stopping the service
19. How to add an index
If you add an index in MySQL, you cannot
update your data until the process is completed.
When it comes to a big table, it takes
a long time to complete, sometimes you cannot
use the service during the change.
Here, I will explain how to add an index,
without stopping the update of your data.
20. Initial Structure
tbl_a
Create table tbl_a (
col_a int, DB1
col_b int,
primary key(col_a)
) engine = InnoDB;
There is 1 MySQL server.
22. Step 2
tbl_a2
tbl_a5
tbl_a3
tbl_a
DB1
Rename table on DB1.
(rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
23. Step 3
tbl_a2
tbl_a5
tbl_a3
tbl_a
DB1
Copy data from tbl_a2 to tbl_a3 on DB1.
(select vp_copy_tables(âtbl_aâ, âtbl_a2â, âtbl_a3â))
24. Step 4
tbl_a2
tbl_a5
tbl_a
tbl_a4
DB1
Rename table on DB1.
(rename table tbl_a to tbl_a4, tbl_a3 to tbl_a)
25. Finish
tbl_a
DB1
Drop table on DB1.
(drop table tbl_a2, tbl_a4, tbl_a5)
26. How to change the schema
without stopping the service
27. How to change the schema
If you change schema in MySQL, you cannot
update your data until the process is completed.
When it comes to a big table, it takes
a long time to complete, sometimes you cannot
use the service during the change.
Here, I will explain how to change schema,
without stopping the update of your data.
28. Initial Structure
tbl_a
Create table tbl_a (
col_a int, DB1
col_b int,
primary key(col_a)
) engine = InnoDB;
There is 1 MySQL server.
30. Step 2
tbl_a2
tbl_a5
tbl_a3
tbl_a
DB1
Rename table on DB1.
(rename table tbl_a2 to tbl_a5, tbl_a to tbl_a2, tbl_a4 to tbl_a)
31. Step 3
tbl_a2
tbl_a5
tbl_a3
tbl_a
DB1
Copy data from tbl_a2 to tbl_a3 on DB1.
(select vp_copy_tables(âtbl_aâ, âtbl_a2â, âtbl_a3â))
32. Step 4
tbl_a2
tbl_a5
tbl_a
tbl_a4
DB1
Rename table on DB1.
(rename table tbl_a to tbl_a4, tbl_a3 to tbl_a)
33. Finish
tbl_a
DB1
Drop table on DB1.
(drop table tbl_a2, tbl_a4, tbl_a5)
34. How to set up a cluster
for fault tolerance
without stopping the service
35. How to set up a cluster for fault tolerance
Spider can set up a cluster for fault tolerance
by each table.
Here, I will explain how to set up cluster,
without stopping service.
'Monitoring node' in this slide is a node that works to observe
the trouble of each node that composes clustering.
'Spider_copy_tables' in this slide is in development , so please
wait for a while to use it.
36. Initial Structure
tbl_a Create table tbl_a (
col_a int,
DB2 col_b int,
primary key(col_a)
Create table tbl_a ( ) engine = InnoDB;
col_a int,
col_b int,
primary key(col_a)
) engine = Spider
Connection â tbl_a
table âtbl_aâ,
user âuserâ,
password âpassâ,
DB1
host âDB2â
â;
There are 1 MySQL server with Spider and 1 remote
Mysql servers without Spider.
37. Step 1 (for clustering)
tbl_a tbl_a tbl_a
DB2 DB3 DB4
tbl_a Create table tbl_a (
col_a int,
DB1 col_b int,
primary key(col_a)
) engine = InnoDB;
Add new data nodes(DB3 and DB4) and tables.
42. How to add new node
after failover
and preparing new server
without stopping the service
43. Create a table of a new node to the clustered table
You need to create a new node, in order to
maintain redundancy, when there is a trouble
at the node that composes the cluster.
Here, I will explain how to add a table of a
new node, without stopping the service.
'Monitoring node' in this slide is a node that works to observe
the trouble of each node that composes clustering.
'Spider_copy_tables' in this slide is still in development , it will
be available in future releases.
44. Initial Structure
tbl_a tbl_a tbl_a
DB2 DB3 DB4
tbl_a
DB1 tbl_a
DB7
DB6
DB5
There are 4 MySQL servers with Spider
(include 3 monitoring nodes) and
3 MySQL servers without Spider (including 1 broken node).
50. How to avoid table partitioning
UNIQUE column limitation
without stopping the service
51. How to avoid table partitioning UNIQUE column limitation
Right now, there is a restriction of MySQL that
you cannot partition in other columns when
there is a PK or UNIQUE.
Here, I will show you how to partition a table by
any columns even if there is a PK or
UNIQUE.
52. Initial Structure
tbl_a
Create table tbl_a (
col_a int, DB1
col_b int,
primary key(col_a)
) engine = InnoDB;
There is 1 MySQL server.
59. About MicroAd
MicroAd is an advatising company.
This company can advertise efficiently
using "behavioral targeting" technology.
ăMicroAd, Inc.]
http://www.microad.jp/english/
60. The previous architecture
âŠâŠ âŠâŠ
AP AP AP AP
LVS
Slave Slave
DB DB Register new statistical rules
replication from batch server
Master
Batch
DB
Batch processing updates new statistical rules every day.
(For every advertisers, every advertising medias
and every users)
61. The problem with business expansion
Increase data and request.
At that time the limit of updates were 20 million
records a day.
They needed to update 100 million records a day.
They also wanted to improve the performance of
the reference slave by decreasing the amount of
the update by one slave.
They did not want to change or modify their
application to support the increase.
Then, Spider was used.
62. The architecture with Spider
âŠâŠ AP AP AP AP âŠâŠ
with Spider with Spider with Spider with Spider
Spider sharding
LVS LVS LVS
SlaveDB SlaveDB SlaveDB SlaveDB SlaveDB SlaveDB
replication replication replication
MasterDB MasterDB MasterDB
Spider sharding Register new
statistical rules from batch server
SpiderDB
(MySQL with Spider) Batch
They created the shards with
the unit of the replication.
63. Resolved the problem
As a result,
They achieved update 100 million records a day
and improved the performance of the reference.
They didn't need to change or modify their
applications so much.
They are planning in the near future of
resharding, when they expand the business.
64. Any Questions?
Thank you for taking
your time!!
Kentoku SHIBA (kentokushiba at gmail dot com)
http://wild-growth.blogspot.com/
http://spiderformysql.com