This document discusses how a new performance dashboard for TiDB databases helps reduce performance tuning time significantly. It introduces TiDB and its architecture, then explains why performance tuning was previously difficult due to the large number of metrics. The dashboard uses "tuning by database time" and "tuning by color" to identify issues. Case studies show how the dashboard helped optimize small table caching, data migration, and social media platform performance. The dashboard articulates issues numerically and reduces human resource needs for tuning by orders of magnitude.

1. Brought to you by
How We Reduced Performance
Tuning Time by Orders of Magnitude
with Database Observability
Yuying Song
Database Performance Engineer at PingCap

2. Yuying Song
Database Performance Engineer
■ Keen on performance diagnosis tool design
■ Keen on performance tuning

3. Agenda
■ Why Performance Tuning is Still Hard even though Software is
Well-Measured?
■ Tuning by Database Time + Tuning by Colour
■ How the New Dashboard Help to Reduce the Performance Tuning Time by
Orders of Magnitude?
■ Beyond Performance Overview Dashboard

4. TiDB
TiDB is an open-source NewSQL database:
■ Hybrid Transactional and Analytical Processing (HTAP) workloads
■ MySQL compatible
■ Horizontal scalability, strong consistency, and high availability
The goal of TiDB is to provide users with a one-stop database solution that covers
OLTP (Online Transactional Processing), OLAP (Online Analytical Processing), and
HTAP services.

5. TiDB Architecture

6. TiDB Monitoring Framework Overview

7. Why Performance Tuning is Still
Hard Even Though Software is
Well-Measured?

8. The Performance Metrics Map of TiDB
Almost every component has its own metrics for each request or operation.

9. Pain Point: Too Many Metrics

10. Metrics

11. Tuning by Database Time
+ Tuning by Colour

12. Tuning by Database Time

13. The Distribution of User Response Time
Is the performance bottleneck inside or outside the database?

14. Database Time
■ To measure latency in a distributed system using a top-down (holistic)
approach
■ DB Time is the total amount of time TiDB takes to process application
requests
■ ΔT DB Time =
● QPS × avg duration × ΔT
● get token + parse + compile + execute
● avg active connections × ΔT
● rate(TiDB_server_handle_query_duration_seconds_sum) × ΔT

15. Tuning by Colour

16. The Performance Metrics Map of TiDB
Colour-based Performance Data

17. How the New Dashboard Help to
Reduce the Performance Tuning
Time by Orders of Magnitude?

18. Case 1 - Feature Testing: Small Table Cache

19. Case 2 - A Data Migration Scenario
1. The Parse and Compile operations made up nearly 60% of DB time.
2. As AWS DMS uses Query API to submit insert statements to TiDB, the prepared plan cache feature cannot
be used to reduce the time spent on parse and compile phases.
3. Batch insert pattern cause high QPS vs low kv request:
BEGIN; INSERT INTO TABLE1 VALUES(VAL1); INSERT INTO TABLE1 VALUES(VAL2);...2k
times...; COMMIT
4.

20. Case 3 - A Social Media Platform
Performance difference between two runs, causing by auto analyze

21. Beyond Performance Overview

22. TiDB Performance Tuning Guide
Performance Tuning Guide based on Performance Overview Dashboard
■ Performance Tuning Overview
■ Performance Analysis and Tuning
■ Performance Tuning Practices for OLTP Scenarios

23. Key Takeaways
■ Tuning by database time and tuning by colour
■ Articulate performance issue by numbers, avoid trial and error
■ The requirements of human resource and expertise has been reduced
by orders of magnitude
■ Quickly Deploy a Local TiDB Cluster
■ PingCAP Learning Channel

24. Brought to you by
Yuying Song
songyuying@yahoo.com
@YuiSoong