The document summarizes several issues encountered with high load on Alibaba MySQL databases and solutions implemented: 1) Hotspot updating of a single row caused deadlocks; implementing queueing on the primary key resolved this. 2) Unexpected long transactions under high load led to clients waiting long periods; committing transactions early where possible addressed this. 3) More than 50,000 active threads overwhelmed MySQL's capabilities; implementing actions based on low and high thread thresholds helped.

1. High Load on
Alibaba MySQL
Xiaobin Lin

2. About Me
• Xiaobin Lin
• RDS @ Aliyun, Alibaba
• Database source code team
• Work around MySQL more than 7 years

3. Catalog
• hotspot updating
• unexpected long transaction
• high concurrency
• slave latency
• thread holding
• performance enhancement

4. hotspot updating
• Problem
• reduce stock of hot commodities
• hotspot updating on a single InnoDB rows
• dead lock detecting

5. hotspot updating
0
5000
10000
15000
20000
1 12 48 192 768
tps
tps
Thread tps
1 6100
6 19160
12 14000
24 10000
48 9800
96 7000
192 3600
384 2000
768 600
1536 150

6. hotspot updating
Dead lock detection

7. hotspot updating
0
5000
10000
15000
20000
1 6 12 24 48 96 192 384 768 1536
TPS1 TPS2
Disable dead lock detection

8. hotspot updating
• Solution: Queue on
primary key

9. hotspot updating
0
5000
10000
15000
20000
1 6 12 24 48 96 192 384 768 1536
TPS1 TPS2
TPS3

10. unexpected long transaction
simplest transaction
begin;
update table_name set …… where id=N;
commit;

12. unexpected long transaction
• Problem
• unexpected long transaction on high
load
• long time waiting for “commit“
• busy client
• network
• rowlock causes problem

14. hotspot updating
• Last Case: Queue on primary
key

15. unexpected long transaction
• Solution
• Commit/rollback transactions as early as
possible
• start transaction;
update
/*! commit_on_success
rollback_on_fail*/
table_name set ……

17. high concurrency
• Problem
• more than 5w active thread
• cpu cost on switch
• lot of timeout in client
• out of MySQL’s capability

18. high concurrency
• Solution: actions on Low/High level
watermark

19. Slave latency
• Problem
• High load on master means the same
on slave
• speed of applying event on slave VS
generating event on master

20. Slave latency
• Solution
• multi-thread applying event, based on
table and row
• relay fetch
• bulk apply

21. Thread holding
• Problem
• Version upgrade / machine maintenance
need to shutdown master
• user connection broken

22. Thread holding
• Solution
• proxy layer
• thread holding

23. performance enhancement
• async redo log copy and write
• innodb redo crc32 checksum
• reuse dummy index on change buffer
• remove read-only transaction list
• cache transaction read-view
• reduce overload of read-only transaction
• split lock_grant lock
• ……
• 80% improvement on TPS/QPS

24. Q&A