2. Who am I ?
Frédéric Descamps
@lefred
http://about.me/lefred
Percona Consultant
Managing mysql since 3.23 (as far as I
remember)
devops believer
www.percona.com
3. Agenda
● What is Percona
Toolkit ?
● What tools are
included and what
can they do ?
● Quick intro to several
tools
● Resources
www.percona.com
4. Percona Toolkit
● Origin: Maatkit and Aspersa (both projects
started by Baron Schwartz with Daniel Nichter's
contribution)
● Indispensable tools for MySQL administrators
● Opensource GNU GPL v2
● https://launchpad.net/percona-toolkit
● http://www.percona.com/software/percona-toolkit/
● Current version is 2.0.4
www.percona.com
8. Installation
●
Requirements:
➢
Perl, DBI, DBD::pgsql, Wine or Mono
➢
Better performance with Wine
●
Download zip:
➢
http://megaupload.com/PTK204
➢
It's in Percona's MSN repository
●
We are working on a Java version
●
Easy to run: wine pt-diskstats_win32.exe
www.percona.com
9. Installation
●
Requirements:
➢
Perl, DBI, DBD::pgsql, Wine or Mono
➢
Better performance with Wine
●
Download zip:
➢
http://megaupload.com/PTK204
➢
It's in Percona's MSN repository
●
We are working on a Java version
●
Easy to run: wine pt-diskstats_win32.exe
www.percona.com
12. Know your environment
(Hardware & OS)
We need to find the answers to these questions:
● On what kind of machine runs my db ? Physical/Virtual ?
● How much memory is used/free ? Does it swap ? Can I add more ?
● How many CPUs and CPU cores ? Speed ?
● Is this a 64bit machine ?
● What kind of RAID controller ? Volumes ? Disks ?
● What about network interfaces ? Connections ? Processes ?
● What's the Operating System ? Kernel ? Release ?
● Which IO scheduler is in use ?
● On which filesystem is the data stored ?
● Do we have free space ?
www.percona.com 12
13. Summary !
Name: pt-summary
Function: summarize system information in
a nice way
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-summary.html
www.percona.com 13
14. Taking inventory
$ wget percona.com/get/pt-summary
$ chmod +x pt-summary
$ ./pt-summary
# Percona Toolkit System Summary Report ######################
Date | 2012-02-02 12:25:18 UTC (local TZ: EET +0200)
Hostname | db01
Uptime | 42 days, 16:14, 1 user, load average: 1.15, 1.22, 1.08
System | HP; ProLiant DL580 G7; vNot Specified (Rack Mount Chassis)
Service Tag | FOSDEM2012
Platform | Linux
Release | CentOS release 6.1 (Final)
Kernel | 2.6.32-131.21.1.el6.x86_64
Architecture | CPU = 64-bit, OS = 64-bit
Threading | NPTL 2.12
Compiler | GNU CC version 4.4.5 20110214 (Red Hat 4.4.5-6).
SELinux | Disabled
Virtualized | No virtualization detected
# Processor ##################################################
Processors | physical = 4, cores = 40, virtual = 80, hyperthreading = yes
Speeds | 80x2397.393
Models | 80xIntel(R) Xeon(R) CPU E7- 4870 @ 2.40GHz
[ Further output omitted ]
www.percona.com 14
16. Know your environment (MySQL)
We need to find the answers to these questions:
● Version ? Build ?
● How many database ? Where is the data directory ?
● What about replication ? How many replicas ?
● Do you use some Key XtraDB features ?
● What are the Key InnoDB settings ?
● Which storage engines are used ?
● What types of data fields are used ? Index types ?
● Do we use foreign keys ? Views ? Triggers ?
● Do we use Full-text indexes ? Partitioning ?
● What about Privileges ?
www.percona.com 16
17. Summary... again !
Name: pt-mysql-summary
Function: Summarize MySQL information in
a nice way
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-mysql-summary.html
www.percona.com 17
18. Taking inventory (cont.)
$ wget percona.com/get/pt-mysql-summary
$ chmod +x pt-mysql-summary
$ ./pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
System time | 2012-02-02 12:24:37 UTC (local TZ: EET +0200)
# Instances ##################################################
Port Data Directory Socket
===== ========================== ======
/mysql
# Report On Port 3306 ########################################
User | root@localhost
Time | 2012-02-02 14:24:38 (EET)
Hostname | db01
Version | 5.5.18-55-log Percona
Built On | Linux x86_64
Started | 2011-12-26 21:09 (up 37+17:15:35)
Databases | 6
Datadir | /mysql/
Processes | 10 connected, 4 running
Replication | Is a slave, has 1 slaves connected
Pidfile | /mysql/mysqld.pid (exists)
[ Further output omitted ]
www.percona.com 18
19. Master ? Slave ?... both ?
Name: pt-slave-find
Function: find and print replication hierarchy
tree of MySQL slaves.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-slave-find.html
www.percona.com 19
20. Inventory of Replicas
$ ./pt-slave-find localhost
localhost
Version 5.5.18-55-log
Server ID 11
Uptime 37+17:45:50 (started 2011-12-26T21:09:03)
Replication Is a slave, has 1 slaves connected, is not read_only
Filters
Binary logging MIXED
Slave status 0 seconds behind, running, no errors
Slave mode STRICT
Auto-increment increment 2, offset 1
InnoDB version 1.1.8-rel23.0
+- 10.0.0.17
Version 5.5.18-55-log
Server ID 12
Uptime 27+21:02:35 (started 2012-01-05T17:52:18)
Replication Is a slave, has 1 slaves connected, is not read_only
Filters
Binary logging MIXED
Slave status 0 seconds behind, running, no errors
Slave mode STRICT
Auto-increment increment 2, offset 2
InnoDB version 1.1.8-rel23.0
www.percona.com 20
21. Where's My Disk I/O Going ?
● Use pt-diskstats as an improved iostat
➢ Interactively controllable, slice-and-dice
➢ Doesn't lump reads and writes together
➢ Meaningful, correct columns
● Use pt-ioprofile to measure & time your
I/O
➢ Which files is MySQL using ? How much times does it
spend reading, writing, syncing, etc? How many
reads/writes? How large are they ?
pt-ioprofile can be dangerous in production
www.percona.com 21
22. Disk I/O
Name: pt-diskstats
Function: Aggregate and summarize
/proc/diskstats
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-diskstats.html
www.percona.com 22
24. I/O Profile
Name: pt-ioprofile
Function: Watch process IO and print a table
of file and I/O activity.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-ioprofile.html
www.percona.com 24
25. I/O profile
# ./pt-ioprofile
Sat Feb 4 13:44:08 CET 2012
Tracing process ID 24179
total read write open close lseek filename
0.007547 0.000583 0.004245 0.002181 0.000268 0.000270 /tmp/#sql_5e73_1.MYI
0.000686 0.000000 0.000000 0.000420 0.000266 0.000000 /tmp/#sql_5e73_1.MYD
0.000065 0.000000 0.000024 0.000029 0.000012 0.000000 /var/lib/mysql/fred/db.opt
0.000055 0.000000 0.000000 0.000055 0.000000 0.000000 /var/lib/mysql/mysql/db.opt
0.000046 0.000000 0.000000 0.000046 0.000000 0.000000 /var/lib/mysql/test/db.opt
www.percona.com 25
27. Let's try to find the answer to these
indiscreet questions:
● How are the indexes used ?
● Are there duplicate keys ?
● Which queries are eating most of the
resources ?
www.percona.com 27
28. Analyze Schema & Queries
● Duplicate/redundant indexes or foreign keys?
➢ pt-duplicate-key-checker can tell you that
● Which indexes are unused?
➢ pt-index-usage can tell you that
➢ It can also show which queries have unstable
plans, which indexes have alternatives that are
never chosen, and lots more
● Bad SQL?
➢ pt-query-advisor to the rescue
www.percona.com 28
29. Analyze & Profile Your Workload
● Use pt-query-digest to analyze the “slow
query log” and show a profile of the workload
● It does the Right Thing by default
● Mostly used with slow query logs & tcpdumps
www.percona.com 29
30. The Report
● Each class of query is ranked by execution time
● There is a report for each class of query
● It's easy to use
● It's a great way to start optimizing queries
www.percona.com 30
31. Overview Report
# 13903.5s user time, 39.9s system time, 1.78G rss, 1.92G vsz
# Current date: Mon Jan 30 20:36:26 2012
# Hostname: hp-db01
# Files: /var/lib/mysql/db01-slow.log
# Overall: 32.98M total, 1.32k unique, 1.73k QPS, 2.06x concurrency ______
# Time range: 2012-01-30 11:08:22 to 16:25:40
# Attribute total min max avg 95% stddev median
# ============ ======= ======= ======= ======= ======= ======= =======
# Exec time 39309s 1us 161s 1ms 515us 109ms 30us
# Lock time 4471s 0 161s 135us 89us 90ms 0
# Rows sent 118.72M 0 1010.29k 3.77 1.96 963.89 0
# Rows examine 15.54G 0 3.51M 505.92 1.96 11.61k 0
# Rows affecte 1.64M 0 2.85k 0.05 0 1.82 0
# Rows read 118.72M 0 1010.29k 3.77 1.96 963.89 0
# Bytes sent 52.40G 0 45.18M 1.67k 5.20k 20.30k 537.02
# Merge passes 128 0 1 0.00 0 0.00 0
# Tmp tables 957.76k 0 4 0.03 0 0.17 0
# Tmp disk tbl 297.92k 0 2 0.01 0 0.10 0
# Tmp tbl size 5.21T 0 2.22G 169.59k 0 1.78M 0
# Query size 8.52G 6 130.40k 277.38 833.10 709.99 174.84
# InnoDB:
# IO r bytes 672.00k 0 32.00k 0.06 0 35.90 0
# IO r ops 42 0 2 0.00 0 0.00 0
# IO r wait 19ms 0 2ms 0 0 1us 0
# pages distin 329.98M 0 32.08k 28.18 42.48 307.62 4.96
# queue wait 0 0 0 0 0 0 0
# rec lock wai 3402s 0 161s 277us 0 147ms 0
# Boolean:
# Filesort 2% yes, 97% no
# Filesort on 0% yes, 99% no
# Full join 0% yes, 99% no
# Full scan 2% yes, 97% no
# QC Hit 55% yes, 44% no
# Tmp table 2% yes, 97% no
# Tmp table on 0% yes, 99% no
www.percona.com 31
35. Log & Analyze Errors ?
● Use pt-deadlock-logger to capture
deadlocks
● Use pt-fk-error-logger for foreign key
errors
● The tools give you the possibility to log the
results to a database table for analysis
www.percona.com 35
36. Monitor dead locks
Name: pt-deadlock-logger
Function: extract and log MySQL deadlock
information
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-deadlock-logger.html
● SHOW ENGINE INNODB STATUS reports only the most
recent deadlock
● Information in status output lacks details about user, client
host, other locks in the deadlock cycle.
● pt-deadlock-logger formats the needed information
nicely and saves it
www.percona.com 36
37. Monitor Foreign Key Errors
Name: pt-fk-error-logger
Function: extract and log MySQL foreign key
errors.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-fk-error-logger.html
● Like pt-deadlock-logger, this tool collects more useful
information, formats it nicely and saves it.
www.percona.com 37
38. Non-Blocking ALTER TABLE
● Use pt-online-schema-change to alter
tables
➢ It makes a “shadow copy” and swaps them
➢ Extremely useful for large, long-running
ALTER
➢ Facebook uses the same technique
www.percona.com 38
39. Validate Upgrades
● Upgrades are a leading cause of downtime
➢ Are any queries using different indexes ? Is QEP
different ?
➢ Are there new errors or warnings ?
➢ Do queries return the same results ?
➢ Are there significant response time differences ?
● Validate the results before you upgrade!
➢ Capture a snapshot of the data
➢ Capture a log of queries
➢ Compare the queries on two test servers running the
old and the new version
www.percona.com 39
40. Validate upgrades
Name: pt-upgrade
Function: Execute queries on multiple
servers and check for differences.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-upgrade.html
www.percona.com 40
41. pt-upgrade
# Query 2: ID 0xE2C8DC0286B7F44E at byte 0 _______________________________
# Found 23281 differences in 58207 samples:
# checksums 0
# column counts 0
# column types 0
# query times 23281
# row counts 0
# warning counts 0
# warning levels 0
# warnings 0
# AF002020:3306 AF001885:3306
# Errors 0 0
# Warnings 0 0
# Query_time
# sum 34s 70s
# min 395us 342us
# max 218ms 308ms
# avg 587us 1ms
# pct_95 725us 3ms
# stddev 2ms 4ms
# median 541us 690us
# row_count
# sum 1.75k 1.75k
# min 0 0
# max 1 1
# avg 0.03 0.03
# pct_95 0 0
# stddev 0.17 0.17
# median 0 0
www.percona.com 41
43. Verify replication Integrity
Name: pt-table-checksum
Function: perform an online replication
consistency check, or checksum MySQL
tables efficiently on one or many servers.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-table-checksum.html
● Use it routinely (mandatory for 95% of MySQL users)
● It used to be complicated; no more. Just run it!
● Put it in a weekly crontab
● Repair differences with pt-table-sync
www.percona.com 43
45. Find problems
● What do you advise me doctor ?
● Get the symptoms when it hurts !
www.percona.com 45
46. Get the symptoms when it hurts !
To perform this action, do a check-up when
you feel bad:
● pt-stalk
● pt-sift
www.percona.com 46
47. Does it hurt ?
Name: pt-stalk
Function: wait for a condition to occur then
begin collecting data.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-stalk.html
www.percona.com 47
48. We need your health parameters
Name: pt-collect
Function: collect information from a server
for some period of time.
OBSOLETE
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-collect.html
www.percona.com 48
49. Let me check...
Name: pt-sift
Function: browses files created by pt-stalk.
Url: http://www.percona.com/doc/percona-toolkit/2.0/pt-collect.html
www.percona.com 49
50. Very useful
Name: pt-mext
Function: Look at many samples of MySQL
SHOW GLOBAL STATUS side-by-side
Url: http://www.percona.com/doc/percona-toolkit/pt-mext.html
● Default STATUS shows counter since the MySQL
instances started.
● It is more helpful to see a delta of recent activity, to check if
a change you applied made a difference.
www.percona.com 50
51. pt-mext
pt-mext -r -- mysqladmin ext -i10 -c5
All values are interesting but you should pay even more attention to:
● Created_tmp_*
● Handler_read_*
● Innodb_buffer_pool_read*
● Innodb_os_log_written
● Key_read_requests
● Key_reads
● Qchache*
● Select_full_*join
● Select_range
● Select_scan
● Table_locks_waited
● Threads_running
www.percona.com 51
52. Stop Bad Queries
● Use pt-kill to watch for long-running queries
and optionally kill them
● Prevent long-running idle transactions too.
www.percona.com 52
53. Percona Toolkit's Future
● Rapid development & improvement of tools
● Fully supported via Percona Support contract
● Upcoming:
➢ pt-*summary redesigned
➢ pt-stalk “magical fault detection algorithm”
➢ Redesign of pt-online-schema-change (see
http://www.percona.com/docs/wiki/ptdev:blueprints:redesign-pt-
online-schema-change)
➢ more goodies
•https://blueprints.launchpad.net/percona-toolkit
www.percona.com 53
57. Due to time I was not able to present
two new Percona projects related to
High Availability.
If you want to discuss PRM and
XtraDB Cluster... I am available
58. PRM
● Percona Replication Manager
● Uses Corosync / Pacemaker
● Distributed architecture, no SPOF
● One master multiple slaves
● Slaves maintain a master score attribute
based on replication state
● Full stonith/fencing stack
● Flexible logic, score based
www.percona.com 58
59. Percona XtraDB Cluster
● Percona Server + Galera
● Synchronous replication
● Multi-master replication
● Multi-threaded slaves
● Data consistency across all peers
● Automatic nodes provisioning
www.percona.com 59