You’re running MySQL as backend database, how do you tune it to make best use of the hardware? How do you optimize the Operating System? How do you best configure MySQL for a specific database workload? Do these questions sound familiar to you? Maybe you’re having to deal with that type of situation yourself? In this webinar, we’ve discussed some of the settings that are most often tweaked and which can bring you significant improvement in the performance of your MySQL database. We also covered some of the variables which are frequently modified even though they should not. Performance tuning is not easy, but you can go a surprisingly long way with a few basic guidelines. AGENDA Database tuning - the what and why Principles of the tuning process Tuning the Operating System configuration Tuning the MySQL configuration Useful tools pt-summary pt-mysql-summary What to avoid when tuning OS and MySQL configuration SPEAKER Krzysztof Książek, Senior Support Engineer at Severalnines, is a MySQL DBA with experience managing complex database environments for companies like Zendesk, Chegg, Pinterest and Flipboard. This webinar is based on our popular blog series ‘Become a MySQL DBA’.

1. Copyright 2016 Severalnines AB
1
Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team
and I'm your host for today's webinar!
Feel free to ask any questions in the Questions
section of this application or via the Chat box.
You can also contact me directly via the chat
box or via email: jj@severalnines.com during
or after the webinar.

2. Copyright 2016 Severalnines AB
Database Performance Tuning
June 14, 2016
Krzysztof Książek
Severalnines
krzysztof@severalnines.com
2

3. Copyright 2016 Severalnines AB
! We want to help all non-DBA people who look after MySQL infrastructures
! Share tips and good practices
! Watch the replays of the previous webinars:
! http://severalnines.com/webinars-replay
“Become a MySQL DBA” series
3

4. Copyright 2016 Severalnines AB
Configuration tuning - what to tune and why?
4

5. Copyright 2016 Severalnines AB
5
Configuration tuning - what to tune and why?
! Operating system
! Swapping
! I/O scheduler
! CPU scheduler
! TCP/IP stack
! MySQL configuration
! Per-session buffers
! InnoDB Buffer Pool
! InnoDB I/O settings

6. Copyright 2016 Severalnines AB
6
Tuning Process

7. Copyright 2016 Severalnines AB
7
Tuning process
! Never-ending story which starts once you install MySQL on the host
! You have to tune for a specific workload
! Workload may change in time
! More data can make it I/O-bound
! Different query mix may increase CPU load and put stress on different parts of the InnoDB
! Keep in mind that configuration tuning is not likely to give you a huge increase in performance
(except if the server is really badly configured)

8. Copyright 2016 Severalnines AB
8
Tuning process

9. Copyright 2016 Severalnines AB
9
Tuning process
! You need a deterministic, test environment to make sure you can measure the impact of the
changes
! Environment should mirror production as close as possible, to make it more relevant
! Changes should be introduced one at a time to ensure you understand the impact of each of
them
! Benchmark the system using queries as close to production as possible
! Restore it to the original state for another round of tweaking
! Rinse and repeat until you are happy with results

10. Copyright 2016 Severalnines AB
10
Tuning process
! Grab a backup of your production systems
! Restore it on a host
! Capture real-world queries using slow log or tcpdump
! Make _one_ change in my.cnf or OS settings
! Restart MySQL or reboot the host itself
! Replay queries using Percona Playback or pt-upgrade
! Measure the difference, repeat the process by restoring the backup if you want to make one
more change

11. Copyright 2016 Severalnines AB
11
Tuning the operating system for MySQL

12. Copyright 2016 Severalnines AB
12
Tuning the operating system for MySQL - memory
! Minimize the flushing operations in OS to ensure no stalls are happening when dirty data is flushed
to disk
! vm.dirty_ratio - a hard limit of the memory that can be used to cache dirty disk pages
! vm.dirty_background_ratio - percentage of system memory that can be used to cache
modified (“dirty”) pages before the background flush process kicks in
! We want them to be set to low numbers - 5 - 10% to trigger flushing often and make it quick
! Default settings may result in a situation where 40% of your memory has to be flushed to disk at
once
! For large instances (128GB) it can be even >50GB of data to flush

13. Copyright 2016 Severalnines AB
13
Tuning the operating system for MySQL - memory
! Swapping - a process of moving data from memory to disk
! MySQL doesn’t like active swapping - we want data in memory, not on disk
! but we even more don’t want to be killed when system runs out of memory
! vm.swappiness = 1 - starting from kernel 3.5 this is the setting you want to use - allow some
swapping but only when there’s no other option
! You can also deprioritize MySQL when it comes to OOM Killer:
! echo -1000 > /proc/[pid]/oom_score_adj
! should help in environments shared with other processes

14. Copyright 2016 Severalnines AB
14
Tuning the operating system for MySQL - memory
! When running MySQL on bare iron, you need to make sure you have NUMA interleave policy set
! Starting from 5.6.27 a variable is available: innodb_numa_interleave
! Before that you could run MySQL as:
! numactl --interleave=all $command
! Memory allocation library may impact performance of MySQL
! Many choices: glibc malloc, tcmalloc and jemalloc
! Use LD_PRELOAD or malloc-lib variable in [mysqld_safe] section of my.cnf

15. Copyright 2016 Severalnines AB
15
Tuning the operating system for MySQL - memory

16. Copyright 2016 Severalnines AB
16
Tuning the operating system for MySQL - I/O
! Disk schedulers may impact performance of I/O-bound workloads
! CFQ is tuned for desktop workloads
! Noop or deadline are usually better for MySQL
! Benchmark to see which one of them will work better for you
! Filesystem - most often you’ll use EXT4 or XFS
! Performance depends on workload, hardware and kernel version
! Benchmark to see which one will work better in your environment

17. Copyright 2016 Severalnines AB
17
Tuning MySQL configuration

18. Copyright 2016 Severalnines AB
18
Tuning MySQL configuration - memory
! InnoDB buffer pool - used to cache data and
store dirty pages
! More is better but you need to leave some
memory for other buffers
! Per join buffers
! Per session buffers
! Temporary tables
! You may have heard about 80% rule
! It’s more like 90% for large (i.e. 128GB) hosts
! Make sure you err on the side of ‘too
small’
! Unless you run MySQL 5.7 where you can
resize InnoDB buffer pool dynamically,
without restart
! For fairly loaded (~20-30 running threads)
host with 128GB of memory it should be
ok to leave ~15GB of memory free
! All depends on the workload so your
mileage may vary

19. Copyright 2016 Severalnines AB
19
Tuning MySQL configuration - memory
! Per-session buffers in InnoDB:
! sort_buffer_size, read_buffer_size, read_rnd_buffer_size
! Per-join buffer: join_buffer_size
! By default - small values
! More _not_ always better
! At 256KB the way how memory allocates change
! smaller chunks use malloc() which is faster than mmap()
! Make sure to benchmark your system after any change to those settings

20. Copyright 2016 Severalnines AB
20
Tuning MySQL configuration - I/O performance
! innodb_flush_log_at_trx_commit - governs the durability in InnoDB
! 1 - full ACID compliance
! 2 - you may lose up to 1s of transactions when hardware crashes
! 0 - you may lose up to 1s of transactions when MySQL crashes
! Significant change in the I/O performance - less flushes means less I/O and less overhead
! Pick whatever you like and whatever you need
! Slaves may not require full durability if you have many of them
! Galera Cluster nodes may also not require full durability

21. Copyright 2016 Severalnines AB
21
! innodb_io_capacity, innodb_io_capacity_max and innodb_lru_scan_depth - define number of disk operations
InnoDB can execute
! Set it too low and you may not fully utilize your hardware
! More not always better - aggressive flushing is not always the best option
! Redo logs are there for some reason - to minimize number of writes to tablespaces
! innodb_flush_method:
! O_DIRECT for BBU-backed hardware
! O_DSYNC may work better with SAN
! Benchmark your setup before you go live
Tuning MySQL configuration - I/O performance

22. Copyright 2016 Severalnines AB
22
Tuning MySQL configuration - I/O performance
! InnoDB Redo Logs are used to store write transactions and they are written sequentially
! MySQL must not run out of space in them
! Larger logs help with better write merging
! Larger logs help with more stable flushing
! Larger logs may seriously impact recovery time in case of a crash
! The rule of thumb is to make them large enough to store at least 1h of writes

23. Copyright 2016 Severalnines AB
23
Tuning MySQL configuration

24. Copyright 2016 Severalnines AB
24
Tuning MySQL configuration - contentions
! innodb_buffer_pool_instances, table_open_cache_instances
! metadata_locks_hash_instances, innodb_adaptive_hash_index_partitions
! Those options can help you to reduce contention on some of those structures
! Increase number of buffer pools or adaptive hash index partitions if you notice a congestion on
them
! Or, preemptively, if you have to handle highly concurrent traffic
! Don’t use buffer pool instances smaller than 1GB (use 2GB+, more instances can slow down the
system)

25. Copyright 2016 Severalnines AB
! max_connections - keep it large enough to handle incoming connections but avoid hundreds of open
connections
! If you need to handle thousands of connections, check the connection pooling options or a proxy,
ideally with connection multiplexing
! log_bin - you want to have binlogs enabled
! sync_binlog=0 for MySQL 5.7 - it used to be a default on MySQL 5.6, it has been changed in 5.7.7
! skip_name_resolve - just to make sure your database won’t suffer when DNS will not be reachable
! Query cache - optimize it away by disabling
! Use external caching layer (Redis, Memcached)
25
Tuning MySQL configuration

26. Copyright 2016 Severalnines AB
26
Tools useful in tuning the configuration

27. Copyright 2016 Severalnines AB
27
Tools useful in tuning the configuration
! Percona developed two great tools, part of the Percona Toolkit suite, which helps you to collect useful
data about MySQL and underlying operating system
! pt-summary collects data about hardware and OS settings
! CPU, memory, disk, load on the system
! pt-mysql-summary connects to MySQL instance and gives you data about workload and configuration
! Type of queries that are running, handlers, configuration
! snapshot of current traffic
! and many others

28. Copyright 2016 Severalnines AB
28
Tools useful in tuning the configuration - pt-summary

29. Copyright 2016 Severalnines AB
29
Tools useful in tuning the configuration - pt-mysql-summary

30. Copyright 2016 Severalnines AB
30
Tools useful in tuning the configuration - pt-mysql-summary

31. Copyright 2016 Severalnines AB
31
What to avoid when tuning MySQL?

32. Copyright 2016 Severalnines AB
32
What to avoid when tuning MySQL?
! Tuning MySQL is not a simple task - it’s very time-consuming to go through the process of change
- test - change - test…
! People like to simplify things and look for shortcuts
! Auto-tuning tools have been created
! mysqltuner.pl
! Configurators and wizards have been created (i.e. by Percona)
! Monitoring and management tools offer advisors (i.e. ClusterControl)

33. Copyright 2016 Severalnines AB
33
What to avoid when tuning MySQL?
! Some of those tools base on counters from MySQL internal status and this is tricky
! The fact you have joins performed without indices doesn’t justify increasing join_buffer_size
! Sometimes making a good change is a bad decision
! If you run MySQL without binlog enabled, without double_buffer or without XA enabled, you
may see a suggestion to enable those features
! Binlog is great for point-in-time recovery
! double_buffer and XA helps keep your data safe
! Enabling them will add serious I/O load on the system - it may not handle it

34. Copyright 2016 Severalnines AB
34
What to keep in mind when tuning MySQL?
! Make sure you follow the tuning process as we described - one change at a time
! If you use external tools that provide suggestions, take them with a grain of salt
! How competent is the vendor in databases?
! Even if the vendor has the competence - do I understand what this change is and how it may
impact my system?
! If you setup a new host, you may use external tools more freely
! This would be initial tuning, not changing the way how application already works
! Make sure you use common sense and each of your changes has a reason

35. Copyright 2016 Severalnines AB
35
Thank you!
! More blogs in “Become a MySQL DBA” series:
! http://www.severalnines.com/blog/become-mysql-dba-blog-series-
query-tuning-process
! http://www.severalnines.com/blog/become-mysql-dba-blog-series-
configuration-tuning-performance
! MySQL Replication Blueprint
! http://severalnines.com/blog/new-whitepaper-mysql-replication-
blueprint
! Contact: jj@severalnines.com