Become a MySQL DBA: performing live database upgrades - webinar slides

1
Your host & some logistics
I'm Jean-Jérôme from the Severalnines Team
and I'm your host for today's webinar!
Feel free to ask any questions in the
Questions section of this application or via
the Chat box.
You can also contact me directly via the chat
box or via email: jj@severalnines.com
during or after the webinar.

Confidential
Copyright 2013 Severalnines AB
ClusterControl Overview
2

Confidential
ClusterControl 
In a nutshell
3
Manage Scale
MonitorDeploy

Confidential
Copyright Severalnines AB
ClusterControl 
Automation & Management
! Provisioning
! Deploy a cluster in minutes
! On-premises or in the cloud (AWS)
! Monitoring
! Systems view
! 1sec resolution
! DB / OS stats & performance advisors
! Configurable dashboards
! Query Analyzer
! Real-time / historical
4

Confidential
ClusterControl 
Automation & Management
! Management
! Multi cluster/data-center
! Automate repair/recovery
! Database upgrades
! Backups
! Configuration management
! Cloning
! One-click scaling
5

Confidential
Supported Databases
SQL
! MySQL (Oracle/Percona/
MariaDB)
! MySQL Galera Cluster
! MySQL Cluster (NDB)
! MariaDB Cluster
! MySQL Replication 5.6
! Standalone MySQL/MariaDB
! Postgres 9.4
NoSQL
! Percona TokuMX Cluster
! MongoDB Sharded Cluster
! MongoDB Replica Set
6

Confidential
Customers
7

Performing MySQL database upgrades in Replication
and Galera setups
November 24, 2015
Krzysztof Książek
Severalnines
krzysztof@severalnines.com
8

! We want to help all non-DBA people who look after
MySQL infrastructures
! Share tips and good practices
! Watch the replays of the previous webinars on our
Slideshare page
! http://www.slideshare.net/Severalnines/videos
9
“Become a MySQL DBA” series

! Why upgrade?
! Different types of upgrade
! How to prepare for an upgrade
! Minor version upgrade
! for MySQL and Galera
! Major version upgrade
! for MySQL and Galera
! ClusterControl and MySQL upgrades 
 
10
Agenda

Why upgrade?
11

! Security incidents and downtimes related to bugs can
induce high costs
! Patches and upgrades require constant attention - these
have to be installed on a regular basis to protect systems
from known stability problems or security holes.
! Many companies are failing to do this
! On the next slide you can find some interesting exceptions
from MySQL 5.6.27 changelog
12
Why upgrade?

! InnoDB: Setting lower_case_table_names=0 on a case-insensitive file
system could result in a hang condition when running an INSERT INTO ...
SELECT ... FROM tbl_name operation with the wrong tbl_name letter case.
! Replication: Repeatedly checking for ERR_LOCK_WAIT_TIMEOUT (as done,
for example by repeatedly executing SHOW SLAVE STATUS) during a
prolonged write lock on a table led to an assert.
! An assertion could be raised due to incorrect error handling if a SELECT ...
FOR UPDATE subquery resulted in deadlock and caused a rollback.
! DELETE could check privileges for the wrong database when table aliases
were used. 
13
Why upgrade?

! MySQL 5.6.27 (2015-09-30)
! MySQL 5.6.26 (2015-07-24)
! MySQL 5.6.25 (2015-05-29)
! MySQL 5.6.24 (2015-04-06)
! MySQL 5.6.23 (2015-02-02)
! MySQL 5.6.22 (2014-12-01)
! MySQL 5.6.21 (2014-09-23)
! MySQL 5.6.20 (2014-07-31) 
14
Why upgrade?

Different types of upgrade
15

! Minor version upgrades
! MySQL 5.6.x -> 5.6.y
! Versions should be compatible with regards to features
! Backward compatibility should be maintained
! The biggest concern is performance - execution plans
may change, MySQL host has to be restarted
16

! Major version upgrades
! 5.x -> 5.y or even x.y -> a.b (for example 4.1 -> 5.7)
! Everything may have been changed
! Changes may be incompatible
! Extensive testing is required
! Read through the documents covering upgrade
process
17

How to prepare for an upgrade?
18

! We’ll focus on major version upgrade as it has much
bigger potential to impact your workload than simple
5.6.x -> 5.6.y upgrade
! First step - plan your upgrade process. Will you use logical
dumps or binary in-place upgrade? Are you going to use
a slave?
! Test your plan - is it doable? Did everything work as
planned? Sometimes differences between versions make
some upgrade plans impossible to execute
19

! Once you have a test database on a new version,
upgraded in a way you plan to upgrade production, it’s
time to run further test
! You need to check if app works ok with a new version of
the database
! Check for both logical issues and performance regressions
! More profiling you can do from within your app (or
external tool like New Relic), more data to compare
20

! Performance regressions
! Some settings may need to be tuned differently
! Query execution plans may change from version to
version
! Logical issues
! Implicit row order may differ
! SQL syntax may have changed
21

! Minor version upgrades should be compatible (after GA)
! They tend to contain bug fixes and some simple
improvements
! Ideally, you’d perform the same suite of tests like you’d
perform when upgrading 5.x -> 5.y
! As a minimum, check for performance regressions
! Plan for warming up the node - MySQL restart will be
required
22

! pt-upgrade is one of tools you can use to automate
regression testing
! It uses data from slow log and replay them on two MySQL
nodes (i.e. running old and new version)
! Queries are checked (among others) for the execution
time and their result set (missing rows, incorrect order)
! Helps DBA to pinpoint regressions and potential issues
23

Minor version upgrade
24

! This process involves performing binary upgrade
! apt-get, yum, binary package - depending on the distro
! Once you perform binary upgrade, keep in mind to run
mysql_upgrade script
! It’s designed to check schemas for any possible
incompatibilities and fix them if needed
25
Minor version upgrade - MySQL and Galera

! You want to go slave by slave, upgrading them serially
! Keep the majority of traffic off upgraded slave to allow it to
warm up
! You can preload buffer pool data (available in Oracle’s MySQL
5.6 and Percona Server for MySQL 5.5) to minimize the impact
! Keep in mind that the rest of the environment will have to handle
higher load for the duration of upgrade and warmup phase
! Once you upgraded all the slaves, failover needs to be
performed and then, finally, old master has to be upgraded 
26
Minor version upgrade - MySQL

! You want to perform upgrade node by node
! Nodes will have to be restarted so traffic needs to be
steered out of them for the duration of a maintenance
! Warmup phase applies here too - ensure that upgraded
node is warmed up correctly
! Again, preloading buffer pool may speed up the process
significantly
27
Minor version upgrade - Galera

Major version upgrade
28

! Much more complex process
! Logical upgrade is a recommended way to perform
upgrade (only in 5.6 -> 5.7 Oracle removed this
recommendation)
! For large databases this is serious issue to solve as
dumping and then loading data takes very long time
! Setting up new nodes and connecting them via the
replication may be needed to stay online for the duration
of the upgrade
29
Major version upgrade - MySQL

! In-place upgrade is possible (just run apt-get/yum) but
you have to be sure it works for you - tests have to be very
detailed
! We’ve seen databases haunted by issues long time after
the upgrade
! This is definitely not the best option to choose
30

! Assuming you can’t take a
downtime for the duration of
the upgrade, step-by-step
approach is required
! First - build a new slave or use
one of currently used in the
topology (if the rest of them can
handle the workload)
! Stop the replication at a known
position
! Dump the data using
mysqldump, mydumper or any
other mean (SELECT INTO
OUTFILE)
31

! Clean up the host - remove
MySQL data directory, remove
MySQL packages
! Install MySQL in the new version,
clean install
! Load the data
! Setup the host as a slave using
binlog position from the dump
! Let the new slave catch up with
the master (it can be even days
behind, all depends on the
dataset size and dump/load
speed)
32

! Use binary backup
(xtrabackup/snapshots) to
upgrade other nodes
! Finally, failover to one of the
slaves with the new version
and upgrade the old
master
33

! Galera cluster can be treated as a single MySQL instance
! Yet it is possible (given some precautions) to run with a mix
of versions in the cluster
! For larger data sets it may be not doable to dump/reload
data before gcache will be filled - after that SST will be
triggered
! Replication solves this problem - you can build a slave
and it will catch up even if initial lag is in days
34
Major version upgrade - Galera

! Beginning is similar to MySQL
replication - build a new
slave to Galera Cluster
! Dump the data on the slave
(keep in mind binlog
coordinates)
! Upgrade to the new
MySQL/Galera version
! Bootstrap the node as a
separate cluster
! Load the data
35

! Set it up as a slave to the
old Galera Cluster
! One by one, remove nodes
from the old cluster and
add them to the new one
! Upgrade MySQL/Galera to
the new version, wipe out
old data directory - nodes
will be provisioned by SST
36

! All the time you need to
ensure that the production
cluster (initially it will be old
version but later it will be
the new one) has enough
nodes to handle the traffic
! Finally, you will have to
switch production traffic to
the new Galera Cluster
37

! If you can dump and reload data while last applied writeset
still exists in gcache, you can try in-place upgrade
! Take the node off the cluster, dump, upgrade, reload, bring
it back to the cluster
! If IST executes, things are looking good
! Such node has to be read only - writeset replication may not
work from newer to older MySQL version
! If you also upgrade Galera (i.e. 2.x -> 3.x) some settings may
have to be added to my.cnf (socket.checksum for example)
38

! Plan your upgrade process
! Check if it works
! Test your application on new MySQL version
! Check for any regressions
! Execute your upgrade process
! Remember about warmup phase
39
Short summary

MySQL Upgrades and ClusterControl
40

! ClusterControl can be used to some extend to help you in
preparations for upgrade
! It’s also important to keep in mind few things when
upgrading clusters managed by ClusterControl
! Building a slave - ClusterControl has option to automate
slave provisioning for both MySQL replication and Galera
Cluster
41

42
! You can leverage it to
provision additional slave
which then be upgraded
manually to the new
version
! Remove the slave from
the cluster once you
setup new Galera there
! Add new Galera Cluster
to the ClusterControl
once it’s set up

! It is important to disable at least node recovery in the
ClusterControl when performing manual upgrades
! This is to prevent ClusterControl automated recovery from
interrupting upgrade process
! You can also use Nodes -> Node’s IP -> Shutdown Node to
stop MySQL on given host and ensure that ClusterControl
won’t start it again
43

! More blogs in “Become a MySQL DBA” series:
! http://severalnines.com/blog/become-mysql-dba-
blog-series-optimizer-hints-faster-query-execution
blog-series-understanding-mysql-error-log
blog-series-galera-cluster-diagnostic-logs
! Contact: krzysztof@severalnines.com
44
Thank You!

Become a MySQL DBA: performing live database upgrades - webinar slides

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Become a MySQL DBA: performing live database upgrades - webinar slides

Semelhante a Become a MySQL DBA: performing live database upgrades - webinar slides (20)

Mais de Severalnines

Mais de Severalnines (20)

Último

Último (20)

Become a MySQL DBA: performing live database upgrades - webinar slides