1. 1
3 1
5
1
4
1
2
Five Steps
to PostgreSQL
1
1 Performance
Josh Berkus
PostgreSQL Project
MelPUG 2013

2. 1
3 1
5
postgresql.conf
1
4 Hardware
1
2 OS &
Filesystem
Query
Tuning
1
1 Application
Design

3. 0. Getting Outfitted

4. 5 Layer Cake
Queries Transactions Application
Drivers Connections Caching Middleware
Schema Config PostgreSQL
Filesystem Kernel Operating System
Storage RAM/CPU Network Hardware

5. 5 Layer Cake
Queries Transactions Application
Drivers Connections Caching Middleware
Schema Config PostgreSQL
Filesystem Kernel Operating System
Storage RAM/CPU Network Hardware

6. Scalability Funnel
Application
Middleware
PostgreSQL
OS
HW

7. What Flavor is Your DB? O
1
W ►Web Application (Web)
●DB smaller than RAM
●90% or more “one-liner” queries

8. What Flavor is Your DB? O
1
O ►Online Transaction Processing
(OLTP)
●DB slightly larger than RAM to 1TB
●20-70% small data write queries,
some large transactions

9. What Flavor is Your DB? O
1
D ►Data Warehousing (DW)
●Large to huge databases (100GB to
100TB)
●Large complex reporting queries
●Large bulk loads of data
●Also called "Decision Support" or
"Business Intelligence"

10. Tips for Good Form O
1
►Engineer for the problems you have
●not for the ones you don't

11. Tips for Good Form O
1
►A little overallocation is cheaper than
downtime
●unless you're an OEM, don't stint a few
GB
●resource use will grow over time

12. Tips for Good Form O
1
►Test, Tune, and Test Again
●you can't measure performance by “it
seems fast”

13. Tips for Good Form O
1
►Most server performance is
thresholded
●“slow” usually means “25x slower”
●it's not how fast it is, it's how close you
are to capacity

14. 1 Application
Design

15. Schema Design 1
1
►Table design
●do not optimize prematurely
▬normalize your tables and wait for a proven
issue to denormalize
▬Postgres is designed to perform well with
normalized tables
●Entity-Attribute-Value tables and other
innovative designs tend to perform poorly

16. Schema Design 1
1
►Table design
●consider using natural keys
▬can cut down on the number of joins you
need
●BLOBs can be slow
▬have to be completely rewritten,
compressed
▬can also be fast, thanks to compression

17. Schema Design 1
1
►Table design
●think of when data needs to be updated,
as well as read
▬sometimes you need to split tables which
will be updated at different times
▬don't trap yourself into updating the same
rows multiple times

18. Schema Design 1
1
►Indexing
●index most foreign keys
●index common WHERE criteria
●index common aggregated columns
●learn to use special index types:
expressions, full text, partial

19. Schema Design 1
1
►Not Indexing
●indexes cost you on updates, deletes
▬especially with HOT
●too many indexes can confuse the
planner
●don't index: tiny tables, low-cardinality
columns

20. Right indexes? 1
1
►pg_stat_user_indexes
●shows indexes not being used
●note that it doesn't record unique index
usage
►pg_stat_user_tables
●shows seq scans: index candidates?
●shows heavy update/delete tables: index
less

21. Partitioning 1
1
►Partition large or growing tables
●historical data
▬data will be purged
▬massive deletes are server-killers
●very large tables
▬anything over 10GB / 10m rows
▬partition by active/passive

22. Partitioning 1
1
►Application must be partition-compliant
●every query should call the partition key
●pre-create your partitions
▬do not create them on demand … they will
lock

23. Query design 1
1
►Do more with each query
●PostgreSQL does well with fewer larger
queries
●not as well with many small queries
●avoid doing joins, tree-walking in
middleware

24. Query design 1
1
►Do more with each transaction
●batch related writes into large
transactions

25. Query design 1
1
►Know the query gotchas (per version)
●Always try rewriting subqueries as joins
●try swapping NOT IN and NOT EXISTS
for bad queries
●try to make sure that index/key types
match
●avoid unanchored text searches "ILIKE
'%josh%'"

26. But I use ORM! 1
1
►ORM != high performance
●ORM is for fast development, not fast
databases
●make sure your ORM allows "tweaking"
queries
●applications which are pushing the limits
of performance probably can't use ORM
▬but most don't have a problem

27. It's All About Caching 1
1
►Use prepared queries W O
●whenever you have repetitive loops

28. It's All About Caching 1
1
►Cache, cache everywhere W O
●plan caching: on the PostgreSQL server
●parse caching: in some drivers
●data caching:
▬in the appserver
▬in memcached/varnish/nginx
▬in the client (javascript, etc.)
●use as many kinds of caching as you can

29. It's All About Caching 1
1
But …
►think carefully about cache invalidation
●and avoid “cache storms”

30. Connection Management 1
1
►Connections take resources W O
●RAM, CPU
●transaction checking

31. Connection Management 1
1
►Make sure you're only using W O
connections you need
●look for “<IDLE>” and “<IDLE> in
Transaction”
●log and check for a pattern of connection
growth
●make sure that database and appserver
timeouts are synchronized

32. Pooling 1
1
►Over 100 connections? You need
pooling!
Webserver
Webserver Pool PostgreSQL
Webserver

33. Pooling 1
1
►New connections are expensive
●use persistent connections or connection
pooling sofware
▬appservers
▬pgBouncer
▬pgPool (sort of)
●set pool side to maximum connections
needed

34. 2
1
Query
Tuning

35. Bad Queries 1
2
Ranked Query Execution Times
5000
4000
3000
execution time
2000
1000
0
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
% ranking

36. Optimize Your Queries
1
2
in Test
►Before you go production
●simulate user load on the application
●monitor and fix slow queries
●look for worst procedures

37. Optimize Your Queries
1
2
in Test
►Look for “bad queries”
●queries which take too long
●data updates which never complete
●long-running stored procedures
●interfaces issuing too many queries
●queries which block

38. Finding bad queries 1
2
►Log Analysis
●dozens of logging
options
●log_min_duration_
statement
●pgfouine
●pgBadger

39. Fixing bad queries 1
2
►EXPLAIN ANALYZE
►things to look for:
●bad rowcount estimates
●sequential scans
●high-count loops
●large on-disk sorts

40. Fixing bad queries 1
2
►reading explain analyze is an art
●it's an inverted tree
●look for the deepest level at which the
problem occurs
►try re-writing complex queries several
ways

41. Query Optimization Cycle 1
2
log queries run pgbadger
explain analyze
apply fixes worst queries
troubleshoot
worst queries

42. Query Optimization Cycle
1
2
(new)
check pg_stat_statements
explain analyze
apply fixes worst queries
troubleshoot
worst queries

43. Procedure Optimization
1
2
Cycle
log queries run pg_fouine
instrument
apply fixes worst
functions
find slow
operations

44. Procedure Optimization
1
2
Cycle (new)
check pg_stat_function
instrument
apply fixes worst
functions
find slow
operations

45. 3
1
postgresql.conf

46. max_connections 3
1
►As many as you need to use
●web apps: 100 to 300 W O
D
●analytics: 20 to 40
►If you need more than 100 regularly,
use a connection pooler
●like pgbouncer

47. shared_buffers 3
1
►1/4 of RAM on a dedicated server
W O
●not more than 8GB (test)
●cache_miss statistics can tell you if you
need more
►less buffers to preserve cache space
D

48. Other memory parameters 3
1
►work_mem
●non-shared
▬lower it for many connections W O
▬raise it for large queries D
●watch for signs of misallocation
▬swapping RAM: too much work_mem
▬log temp files: not enough work_mem
●probably better to allocate by task/ROLE

49. Other memory parameters 3
1
►maintenance_work_mem
●the faster vacuum completes, the better
▬but watch out for multiple autovacuum
workers!
●raise to 256MB to 1GB for large
databases
●also used for index creation
▬raise it for bulk loads

50. Other memory parameters 3
1
►temp_buffers
●max size of temp tables before swapping
to disk
●raise if you use lots of temp tables D
►wal_buffers
●raise it to 32MB

51. Commits 3
1
►checkpoint_segments
●more if you have the disk: 16, 64, 128
►synchronous_commit
●response time more important than data
integrity?
●turn synchronous_commit = off W
●lose a finite amount of data in a
shutdown

52. Query tuning 3
1
►effective_cache_size
●RAM available for queries
●set it to 3/4 of your available RAM
►default_statistics_target D
●raise to 200 to 1000 for large databases
●now defaults to 100
●setting statistics per column is better

53. Query tuning 3
1
►effective_io_concurrency
●set to number of disks or channels
●advisory only
●Linux only

54. A word about
Random Page Cost
3
1
►Abused as a “force index use”
parameter
►Lower it if the seek/scan ratio of your
storage is actually different
●SSD/NAND: 1.0 to 2.0
●EC2: 1.1 to 2.0
●High-end SAN: 2.5 to 3.5
►Never below 1.0

55. Maintenance 3
1
►Autovacuum
●leave it on for any application which gets
constant writes W O
●not so good for batch writes -- do manual
vacuum for bulk loads D

56. Maintenance 3
1
►Autovacuum
●have 100's or 1000's of tables?
multiple_autovacuum_workers
▬but not more than ½ cores
●large tables? raise
autovacuum_vacuum_cost_limit
●you can change settings per table

57. 1
4
OS &
Filesystem

58. Spread Your Files Around 1
4
►Separate the transaction log if O D
possible
●pg_xlog directory
●on a dedicated disk/array, performs
10-50% faster
●many WAL options only work if you have
a separate drive

59. Spread Your Files Around 1
4
number of drives/arrays 1 2 3
which partition
OS/applications 1 1 1
transaction log 1 1 2
database 1 2 3

60. Spread Your Files Around 1
4
►Tablespaces for temp files D
●more frequently useful if you do a lot of
disk sorts
●Postgres can round-robin multiple temp
tablespaces

61. Linux Tuning 1
4
►Filesystems
●Use XFS or Ext4
▬butrfs not ready yet, may never work for DB
▬Ext3 has horrible flushing behavior
●Reduce logging
▬data=ordered, noatime,
nodiratime

62. Linux Tuning 1
4
►OS tuning
●must increase shmmax, shmall in kernel
●use deadline or noop scheduler to speed
writes
●disable NUMA memory localization
(recent)
●check your kernel version carefully for
performance issues!

63. Linux Tuning 1
4
►Turn off the OOM Killer!
● vm.oom-kill = 0
● vm.overcommit_memory = 2
● vm.overcommit_ratio = 80

64. OpenSolaris/IIlumos 1
4
►Filesystems
●Use ZFS
▬reduce block size to 8K W O
●turn off full_page_writes
►OS configuration
●no need to configure shared memory
●use packages compiled with Sun
compiler

65. Windows, OSX Tuning 1
4
►You're joking, right?

66. What about The Cloud? 1
4
►Configuring for cloud servers is
different
●shared resources
●unreliable I/O
●small resource limits
►Also depends on which cloud
●AWS, Rackspace, Joyent, GoGrid
… so I can't address it all here.

67. What about The Cloud? 1
4
►Some general advice:
●make sure your database fits in RAM
▬except on Joyent
●Don't bother with most OS/FS tuning
▬just some basic FS configuration options
●use synchronous_commit = off if
possible

68. Set up Monitoring! 1
4
►Get warning ahead of time
●know about performance problems
before they go critical
●set up alerts
▬80% of capacity is an emergency!
●set up trending reports
▬is there a pattern of steady growth

69. 1
5
Hardware

70. Hardware Basics 1
5
►Four basic components:
●CPU
●RAM
●I/O: Disks and disk bandwidth
●Network

71. Hardware Basics 1
5
►Different priorities for different
applications
●Web: CPU, Network, RAM, ... I/O W
●OLTP: balance all O
●DW: I/O, CPU, RAM D

72. Getting Enough CPU 1
5
►One Core, One Query
●How many concurrent queries do you
need?
●Best performance at 1 core per no more
than two concurrent queries
►So if you can up your core count, do
►Also: L1, L2 cache size matters

73. Getting Enough RAM 1
5
►RAM use is "thresholded"
●as long as you are above the amount of
RAM you need, even 5%, server will be
fast
●go even 1% over and things slow down a
lot

74. Getting Enough RAM 1
5
►Critical RAM thresholds W
●Do you have enough RAM to keep the
database in shared_buffers?
▬Ram 3x to 6x the size of DB

75. Getting Enough RAM 1
5
►Critical RAM thresholds O
●Do you have enough RAM to cache the
whole database?
▬RAM 2x to 3x the on-disk size of the
database
●Do you have enough RAM to cache the
“working set”?
▬the data which is needed 95% of the time

76. Getting Enough RAM 1
5
►Critical RAM thresholds D
●Do you have enough RAM for sorts &
aggregates?
▬What's the largest data set you'll need to
work with?
▬For how many users

77. Other RAM Issues 1
5
►Get ECC RAM
●Better to know about bad RAM before it
corrupts your data.
►What else will you want RAM for?
●RAMdisk?
●SWRaid?
●Applications?

78. Getting Enough I/O 1
5
►Will your database be I/O Bound?
●many writes: bound by transaction log O
●database much larger than RAM: bound
by I/O for many/most queries D

79. Getting Enough I/O 1
5
►Optimize for the I/O you'll need
●if you DB is terabytes, spend most of
your money on disks
●calculate how long it will take to read
your entire database from disk
▬backups
▬snapshots
●don't forget the transaction log!

80. I/O Decision Tree 1
5
lots of fits in
No Yes mirrored
writes? RAM?
Yes No
afford
terabytes HW RAID
good HW Yes No
of data?
RAID?
Yes
No
mostly
SW RAID Storage read?
Device
Yes No
RAID 5 RAID 1+0

81. I/O Tips 1
5
►RAID
●get battery backup and turn your write
cache on
●SAS has 2x the real throughput of SATA
●more spindles = faster database
▬big disks are generally slow

82. I/O Tips 1
5
►DAS/SAN/NAS
●measure lag time: it can kill response
time
●how many channels?
▬“gigabit” is only 100mb/s
▬make sure multipath works
●use fiber if you can afford it

83. I/O Tips 1
5
iSCSI
=
death

84. SSD 1
5
►Very fast seeks D
●great for index access on large tables
●up to 20X faster
►Not very fast random writes
●low-end models can be slower than HDD
●most are about 2X speed
►And use server models, not desktop!

85. NAND (FusionIO) 1
5
All the advantages of SSD, Plus:
►Very fast writes ( 5X to 20X ) W O
●more concurrency on writes
●MUCH lower latency
►But … very expensive (50X)

86. Tablespaces for NVRAM 1
5
►Have a "hot" and a "cold" tablespace
●current data on "hot" O D
●older/less important data on "cold"
●combine with partitioning
►compromise between speed and size

87. Network 1
5
►Network can be your bottleneck
●lag time
●bandwith
●oversubscribed switches
●NAS

88. Network 1
5
►Have dedicated connections
●between appserver and database server
●between database server and failover
server
●between database and storage

89. Network 1
5
►Data Transfers
●Gigabit is only 100MB/s
●Calculate capacity for data copies,
standby, dumps

90. The Most Important
Hardware Advice:
1
5
►Quality matters
●not all CPUs are the same
●not all RAID cards are the same
●not all server systems are the same
●one bad piece of hardware, or bad driver,
can destroy your application
performance

91. The Most Important
Hardware Advice:
1
5
►High-performance databases means
hardware expertise
●the statistics don't tell you everything
●vendors lie
●you will need to research different
models and combinations
●read the pgsql-performance mailing list

92. The Most Important
Hardware Advice:
1
5
►Make sure you test your hardware
before you put your database on it
●“Try before you buy”
●Never trust the vendor or your sysadmins

93. The Most Important
Hardware Advice:
1
5
►So Test, Test, Test!
●CPU: PassMark, sysbench, Spec CPU
●RAM: memtest, cachebench, Stream
●I/O: bonnie++, dd, iozone
●Network: bwping, netperf
●DB: pgBench, sysbench

94. Questions? 1
6
►Josh Berkus ►More Advice
● josh@pgexperts.com ● www.postgresql.org/docs
● www.pgexperts.com ● pgsql-performance
▬ /presentations.html mailing list
● www.databasesoup.com ● planet.postgresql.org
● irc.freenode.net
▬ #postgresql
This talk is copyright 2013 Josh Berkus, and is licensed under the creative commons attribution license