18. Dynamic Migration
• One-time Migration
- e.g. Shard Rebalancing
• Real-time Migration (as a Service?)
- e.g. MySQL => HBase in Real-time
19. Ex: Add new shard
Shard1 Shard2
Shard1 Shard3Shard2
20. Ex: Change shard rule
ID: 1 - 1000
ID: 1001 - 2000
ID: 2001 - 3000
Range
ID mod 3: 0
ID mod 3: 1
ID mod 3: 2
Modulus
21. Ex: Copy to diff. DBMS
MySQL
HBase
MongoDB
NO side effect
22. Ex: Copy to diff. schema
MySQL
MySQL
MySQL
ID AGE CNT V
1 30 5 msg1
2 29 10 msg2
ID CNT V
1 5 msg1
ID CNT V
2 10 msg2
NO side effect
23. Ex: Write w/ custom processing
MySQL
MySQL
MySQL
Events
ID AGE CNT V
1 30 5 msg1
2 29 10 msg2
1) REQ
INFO2)
3) GET INFO
4) MERGE
ID FRIENDS
1 4,5,6,7,8,9
2 3,4,5,8,10
31. Features
• Table Crawler
• SELECT 쿼리의 반복
SELECT * FROM ? [ WHERE id > ? ] LIMIT ?;
• Binlog Receiver
• MySQL Replication 프로토콜
• Custom Data Handler
• 수집한 데이터의 처리 부분
e.g. Shard reconstruction handler
• 여러 스레드에 의해 동시에 실행됨
32. Features: Binlog Processor
Binlog
Recv
Queue #1
Custom
Data
Handler
Queue #2
Queue #3
Queue #n
…
ADT Binlog Processor
•Binlog Receiver
1. PK/UK 값을 읽음
2. 읽은 값을 기준으로 적절한
Queue에 넣음
3. 해당 Queue를 처리할 담당
Thread 지정
•1 Thread / 1 Queue
- 같은 Queue: 순차 처리
- 다른 Queue: 병렬 처리
33. Features: Data Crawler
1. SELECT … LIMIT n
2. 파이프라이닝
•SELECT한 값을 처리하기
전에 다른 쓰레드를 깨워 다
음 SELECT 실행
•SELECT한 순서대로
Custom Handler가 호출되
지는 않음
Select Crawl from Last PK
Custom Data Handler
ADT Data Crawler
Thread
Pool
35. Req 1. Row Format
각 Binlog는 Before, After 값이 필요합니다
N/A 1 : a=1, b=2, c=3
Before After
INSERT
1 : a=1, b=99, c=99 1 : a=1, b=2, c=3UPDATE
1 : a=1, b=99, c=99 N/ADELETE
36. Req 2. Primary Key
DELETE
Source Destination
INSERT
UPDATE
Delete Old/Insert New
Delete Old/Insert New
Delete Old
모든 Table은 Primary Key가 존재해야 합니다
37. ADT
Req 3. Custom Data Handler
MySQL
Binlog
Processor
MySQL
Data
Crawler
Custom Data Handler
(예: Shard 재분배 핸들러)
New
Masters
Load
Current
Master
Data
이 부분을 직접 구현해야 합니다
38. 기타 요구/제약 사항들
• Millisecond 사용 불가
• Alter Table 실시간 반영 안 됨
• 기타 등등...
42. Why Ignore FK?
• 샤드 재구성 하려고 했던 곳이 성능을 위해 FK를 안 쓰는 곳이어서
• FK constraint 체크는 not null, check와 같이 master에서 이미 했
으므로 slave에서 할 필요 없다고 판단
• FK가 데이터 변경을 유발하지 않는 경우: 고려 X
• FK로 인해 다른 테이블 값 변경되는 경우
• table이 다르면 병렬 처리 시 conflict 가능성이 없음
• FK로 인해 같은 테이블 값 변경되는 경우
• 어차피 parent, child row 둘 다 다른 constraint에서 문제가 없
을 경우에만 binlog에 기록되므로 PK, UK만 고려
47. How Data Crawler Works
• SELECT
• SELECT using PK of ex-selected rows
• INSERT
• INSERT IGNORE is required
if Binlog Processor runs together
48. How to Handle Binlog? (1/4)
Row Event
Type
Query to Dest. (normally)
WRITE insert( after )
DELETE delete( before )
UPDATE update( before, after )
Normally binlog events are handled like this.
49. How to Handle Binlog? (2/4)
• However, we should consider…
- Unexpected restart
- Data inserted by Crawler
Overwriting!
50. How to Handle Binlog? (3/4)
Row Event
Type
Query to Dest.
(Overwriting)
WRITE replace( after )
DELETE delete( before )
UPDATE
if( before.pk!=after.pk ){
delete( before )
}
replace( after )
51. How to Handle Binlog? (4/4)
• Normal Query
UPDATE … SET @1=after.1, @2=after.2,…
WHERE pk_col=before.pk
• Transformation 1: Unrolling
DELETE FROM … WHERE pk_col=before.pk;
INSERT INTO … VALUES(after.1, after.2,…);
• Transformation 2: Overwriting
DELETE FROM … WHERE pk_col=before.pk;
REPLACE INTO … VALUES(after.1, after.2,…);
• Transformation 3: Reducing
• Delete [before] only if PK is changed
52. Strategy 1
• Run sequentially
1. Crawl Data
2. Process Binary Log (after 1 is finished)
• Binlog file (created before starting) is required
• If crawling takes more than 3 days, then…?
53. Strategy 2
• Run parallel with master DB
1. Start binlog processor
2. Start Data Crawler (ASAP after 1 is started)
with INSERT IGNORE
• Problem: conflicts
55. Strategy 2-1
• Data Crawler
- SELECT … FOR UPDATE when crawling
- ROLLBACK after INSERT IGNORE
• Binlog Processor
- Just act normally
Because there’s no logs for locked rows
• Problem: Multi row lock is dangerous for master DB
56. Strategy 2-2
• Binlog Processor
- Cache deleted history during a few minutes
• Data Cralwer
- If delete history exists, no INSERT
• Problems
- Complicated: Lock is neccesary for history cache
57. Strategy 2-3
• Same as Strategy 2-1, except using slave DB
• If sync is finished with slave,
restart with new config
- Receive binlog from master
59. Test Scenario
•Split into 2 shards from 1 master DB
• Binary log only
•Query many updates into master DB
• 1K active sessions with random DML & data
•Make errors
• Master DB: ifdown —> mysql restart —> ifup
•Compare data
60. Test DML List
INSERT INTO …
INSERT IGNORE INTO …
INSERT INTO … ON DUPLICATE KEY …
REPLACE INTO …
UPDATE …
DELETE …
61. Test Table Schema
Column
Name
Type
Table 1 Table 2
Value
(when exec. DML)Primary Unique Primary Unique
no int 1 2 1 Random(1~n)
seq int 2 Random(1~n)
uk int 1 1 Random(1~n)
update_cnt int Incr. when update
random_val text Random String
(이하 생략)
62. Step1. Split into 2 Shards
MySQL
Source
Query
Tool
ADT
Binlog
Events
Write
MySQL
Dest.
Split
Shards
67. Wish to Apply for…
Shard reconstruction (default)
MySQL binary log —> NoSQL
Copy data change history into OLAP
MySQL binary log —> Push Notification
Re-construct shards by GPS Point (Kakao Taxi?)
……
69. Next Dev. Plans
• Change language: Java —> GoLang
• Control Tower: Admin & Monitoring
• Is ADT alive?
• Save checkpoint for ungraceful restart
• Support Multiple DB Types
• Redis, PgSQL, ……