2. ⏠whoami
â Federico Razzoli
â Freelance consultant
â Writing SQL since MySQL 2.23
hello@federico-razzoli.com
â I love open source, sharing,
Collaboration, win-win, etc
â I love MariaDB, MySQL, Postgres, etc
â Even Db2, somehow
5. Itâs always about Data
â Since then, the purpose of hardware and software never changed:
â Receive data
â Process data
â Output data
6. A rose by any other name...
â Feel free to use synonyms
â Validate
â Sanitise
â Parse
â Persist
â Normalise / Denormalise
â Cache
â Map / Reduce
â Print
â Ping
â ...
7. ...would smell as sweet
â The database of known stars is not a ping package
â You use a DBMS to abstract data management as much as possible
â Persistence
â Consistence
â Queries
â Fast search
â âŠ
â Thatâs why âdatabase is magicâ
8. Busy
â But itâs just a very busy person, performing many tasks concurrently
â And each is:
â Important (must be reliable)
â Complex (must be fast)
â Expected (if something goes wrong,
you will complain)
In practice, the DBMS is usually the bottleneck.
9. Terminology
Statement: An SQL command
Query: A statement that returns a resultset
...or any other statement :)
Resultset: output 0 or more rows
Optimiser / Query Planner: Component responsible of deciding a queryâs
execution plan
Optimised query: A query whose execution plan is reasonably good
...this doesnât imply in any way that the query is fast
Database: set of tables (schema)
Instance / Server: running MySQL daemon (cluster)
11. When should a query be optimised?
mysql> EXPLAIN SELECT * FROM t WHERE c < 10 G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: t
partitions: NULL
type: ALL
possible_keys: idx_c
key: idx_c
key_len: 4
ref: NULL
rows: 213030
filtered: 50.00
Extra: Using where
1 row in set, 1 warning (0.01 sec)
12. When should a query be optimised?
mysql> SELECT * FROM performance_schema.events_statements_summary_by_digestWHERE DIGEST =
'254a65744e661e072103b7a7630dee1c3a3b8e906f19889f7c796aebe7cdd4f8' G
*************************** 1. row ***************************
SCHEMA_NAME: test
DIGEST: 254a65744e661e072103b7a7630dee1c3a3b8e906f19889f7c796aebe7cdd4f8
DIGEST_TEXT: SELECT * FROM `t` WHERE `c` < ?
COUNT_STAR: 1
...
SUM_ROWS_AFFECTED: 0
SUM_ROWS_SENT: 57344
SUM_ROWS_EXAMINED: 212992
SUM_CREATED_TMP_DISK_TABLES: 0
SUM_CREATED_TMP_TABLES: 0
SUM_SELECT_FULL_JOIN: 0
SUM_SELECT_FULL_RANGE_JOIN: 0
SUM_SELECT_RANGE: 0
SUM_SELECT_RANGE_CHECK: 0
SUM_SELECT_SCAN: 1
SUM_SORT_MERGE_PASSES: 0
SUM_SORT_RANGE: 0
SUM_SORT_ROWS: 0
SUM_SORT_SCAN: 0
SUM_NO_INDEX_USED: 1
SUM_NO_GOOD_INDEX_USED: 0
FIRST_SEEN: 2019-05-14 00:31:24.078967
LAST_SEEN: 2019-05-14 00:31:24.078967
...
QUERY_SAMPLE_TEXT: SELECT * FROM t WHERE c < 10
QUERY_SAMPLE_SEEN: 2019-05-14 00:31:24.078967
QUERY_SAMPLE_TIMER_WAIT: 117493874000
13. But how do I find impacting queries?
â It depends what you mean by âimpactingâ
â There are several monitoring methods (USE, etc)
â But 3 philosophies:
14. But how do I find impacting queries?
â It depends what you mean by âimpactingâ
â There are several monitoring methods (USE, etc)
â But 3 philosophies:
â Panicking when you hear that something is down or slow
15. But how do I find impacting queries?
â It depends what you mean by âimpactingâ
â There are several monitoring methods (USE, etc)
â But 3 philosophies:
â Panicking when you hear that something is down or slow
â System-centric monitoring
16. But how do I find impacting queries?
â It depends what you mean by âimpactingâ
â There are several monitoring methods (USE, etc)
â But 3 philosophies:
â Panicking when you hear that something is down or slow
â System-centric monitoring
â User-centric
17. But how do I find impacting queries?
â Panicking when you hear that something is down or slow
â System-centric monitoring
â User-centric
You can use them all.
18. Panicking
â Simplest method
â Do nothing do prevent anything
â Optionally, take a lot of actions to prevent imaginary problems in
imaginary ways
â There is no evidence that your job is useless, so your boss will not fire
you
19. System-centric
â pt-query-digest, PMM, etc
â Merge queries into one, normalising its text and replacing parameters
â SELECT * FROM t WHERE b= 111 AND a = 0 -- comment
â Select * From t Where a = 24 and b=42;
â SELECT * FROM t WHERE a = ? AND b = ?
â Sum execution time of each occurrence (Grand Total Time)
â Optimise the queries with highest GTT
20. User-Centric
â Calculate the cost of slowness (users donât buy, maybe leave 4ever)
â Cost of slowness is different for different
â URLs
â Number of users
â ...other variables that depend on your business
â (day of month, country, etc)
â Set Service Level Objectives
â Monitor the HTTP calls latency, and the involved services
â Find out whatâs slowing them down
22. What makes a query âimportantâ?
â How many times itâs executed
â Itâs locking
23. What makes a query slow?
â Number of rows read
â Read != return
â Indexes are there to lower the number of reads
â Number of rows written
â In-memory temp tables are not good
â On-disk temp tables are worse
24. How do I optimise a query?
â Use indexes properly
â Avoid creation of temp tables, if possible
26. Index Types
â BTREE - ordered data structure
â HASH - hash table
â PostgreSQL has much more
â Each storage engine can implement any of both
â InnoDB uses BTREE and internally uses HASH when it thinks itâs better
â The syntax CREATE INDEX USING [BTREE | HASH] is generally useless
We will focus on BTREE indexes in InnoDB
27. Index Properties
â Primary key: unique values, not null
â UNIQUE
â Multiple columns
â The order matters
â Column prefix (only for strings)
28. InnoDB Indexes
â InnoDB tables should always have a Primary Key
â The table is stored ordered by primary key
â The table itself is the primary key
â Columns ânot part of the primary keyâ simply donât affect the order of
rows
30. InnoDB Indexes
â Secondary indexes are stored separately
â They are ordered by the indexed column
â Each entry contain a reference to a primary key entry
32. Which queries will be faster?
Table columns: {a, b, c, d, e}
Primary key: {a, b}
Index idx_c: {c}
â SELECT * FROM t WHERE a = 1
â SELECT * FROM t WHERE a = 1 AND b = 2
â SELECT a, b FROM t WHERE c = 0
â SELECT d, e FROM t WHERE c = 0
34. More performance considerations?
â Big primary key = big indexes
â Primary key should be append-only
INTEGER UNSIGNED AUTO_INCREMENT
â These indexes are duplicates: {a} - {a, b}
â This index is wrong: {a, id}
40. Phone Book
â Indexes are ordered data structures
â Think to them as a phone book
Table: {first_name, last_name, phone, address}
Index: {last_name, first_name}
41. Phone Book
â I will show you some queries, and you will tell me which can be solved by
using the index
â You may not know, but your mind contains a pretty good SQL optimiser
Table: {first_name, last_name, phone, address}
Index: {last_name, first_name}
42. Queries
SELECT * FROM phone_book âŠ
WHERE last_name = 'Baker'
WHERE last_name IN ('Hartnell','Baker', 'Whittaker')
WHERE last_name > 'Baker'
WHERE last_name >= 'Baker'
WHERE last_name < 'Baker'
WHERE last_name <= 'Baker'
WHERE last_name <> 'Baker'
43. Queries
SELECT * FROM phone_book âŠ
WHERE last_name IS NULL
WHERE last_name IS NOT NULL
45. Queries
WHERE last_name >= 'B' AND last_name < 'C'
WHERE last_name BETWEEN 'B' AND 'C'
WHERE last_name LIKE 'B%'
46. Queries
WHERE last_name LIKE 'B%'
WHERE last_name LIKE '%B%'
WHERE last_name LIKE '%B'
WHERE last_name LIKE 'B_'
WHERE last_name LIKE '_B_'
WHERE last_name LIKE '_B'
47. Rule #2
A LIKE condition
whose second operand starts with a 'constant string'
is a range
48. Queries
WHERE first_name = 'Tom'
WHERE last_name = 'Baker'
WHERE first_name = 'Tom' AND last_name = 'Baker'
WHERE last_name = 'Baker' AND first_name = 'Tom'
51. Rule #4
Optimiser cannot make assumptions on functions/expression results.
However, wrapping a constant value into a function will produce another
constant value, which is mostly irrelevant for query optimisation.
53. Rule #5
Comparing a column with another results in a comparison
whose operands change at every row.
The optimiser cannot ïŹlter out any row in advance.
55. Rule #6
We can use an index to restrict the search to a set of rows
And search those rows in a non-optimised fashion
Depending on this setâs size, this could be a brilliant or a terrible strategy
56. Queries
WHERE last_name = 'Baker' AND first_name > 'Tom'
WHERE last_name > 'Baker' AND first_name = 'Tom'
WHERE last_name > 'Baker' AND first_name > 'Tom'
57. Queries
WHERE last_name = 'Baker' AND first_name > 'Tom'
WHERE first_name = 'Tom' AND last_name > 'Baker'
WHERE first_name > 'Tom' AND last_name = 'Baker'
WHERE last_name > 'Baker' AND first_name > 'Tom'
Baker, Colin
Baker, Tom
Baker, Walter
Capaldi, Ada
Capaldi, Peter
Whittaker, Jody
Whittaker, Vadim
58. Rule #7
If we have a range condition on an index column
The next index columns cannot be used
If you prefer:
Index usage stops at the ïŹrst >
63. Rule #8
ORDER BY and GROUP BY can take advantage of an index order
or create an internal temp table
Note: GROUP BY optimisation also depends on the function weâre using
(MAX, COUNTâŠ).
64. Queries
WHERE last_name > 'Baker' ORDER BY last_name
WHERE last_name = 'Baker' ORDER BY first_name
WHERE last_name > 'Baker' ORDER BY first_name
65. Rule #9
If we have an ORDER BY / GROUP BY on an index column
The next index columns cannot be used
67. Queries
Table: {id, a, b, c, d}
idx_a: {a, d}
idx_b: {b}
WHERE a = 10 OR a = 20
WHERE a = 24 OR c = 42
WHERE a = 24 OR d = 42
WHERE a = 24 AND b = 42
WHERE a = 24 OR b = 42
WHERE a = 24 ORDER BY b
GROUP BY a ORDER BY b
68. Rule #10
Using multiple indexes for AND or OR (intersect) is possible,
but there is a beneïŹt only if we read MANY rows
Using different indexes in WHERE / GROUP BY / ORDER BY
is not possible