Databases are a key part of any application. The storage subsystem contributes most to performance of the database. In recent days, new storage technologies like Solid State Storage (SSD) and high performance drives are becoming cheaper and more accessible, but it takes a lot of planning to use these technologies in a cost effective way for best price-performance.

1. Tuning Storage
Subsystem for
Databases
Angelo Rajadurai

2. Agenda
Performance issues in Storage
Hybrid Storage (Disks, SSDs, Memory)
ZFS - Not Just Another File System
Tuning for databases (General principles)
Tuning for MySQL
Tuning for PostgreSQL
Tuning for Oracle

3. Why?
• Some very practical advice based on
> recent test results
> Improved pgbench results from 70 tps for pure disk to 5003 tps with
SSD and tuning
> Improved sysbench results from 425 tps to 1811 tps with SSD and
tuning for read/write.
> Improved sysbench results from 786 tps to 3085 tps with SSD and
tuning for read.
> collection of tuning knowledge from Sun performance
engineers and the community
• Some very good resources at the end of the talk for
further study

4. Storage Performance
Cache
Memory
0,0 00 X t ial
10 en
fer
e dif
nc
Disk r ma
High Cache r fo
Performance Disks Pe
Large Capacity
Disks

5. Latency Comparison
Bridging the DRAM to HDD Gap
1S
100mS
10m
S
1mS
100uS
10uS TAPE
1u HDD
S
100nS FLASH/
SSD
10nS
DRAM
1n
S CPU

6. Storage Technology
Price, Performance & Capacity
Capacity Latency Cost/IOPS Cost/GB
Technologies IOPs
(GB) (microS) ($) ($)
Cloud
Storage
Unlimited 60,000 20 17c/GB 0.15/month
Capacity
HDDs 2,500 12,000 250 1.67 0.15
Performance
HDDs 300 7,000 500 1.52 1.30
SSDs
(write) 64 300 5000 0.20 13
SSDs
(read only) 64 45 30,000 0.03 13
DRAM 8 0.005 500,000 0.001 52

7. Incorporating Flash
Storage Hierarchy

8. Hybrid Storage
Flash as Cache
Application
DRAM
Level 1 Cache
Write
Read
Flash
Write side Log
Flash
Level 2 Cache
Disk Primary Storage

9. ZFS - Last Word in Filesystem
Pooled Storage Vs Traditional Volumes

10. Data Management Unit
Smarts Built Right Into the Filesystem

11. Administering ZFS in two slides
As easy as pie
• zpool commands
> create a single disk pool:
# zpool create newpool diskname
> create a pool with a mirror
# zpool create newpool mirror disk1name disk2name
> Add device to a pool:
# zpool add poolname diskname
> Replace a bad disk
# zpool replace poolname baddiskname newdiskname
> History of commands on the pool:
# zpool history poolname
> How is my pool performing:
# zpool iostat poolname
No format command, No fdisk partitions, No volumes

12. Administering ZFS in two slides
As easy as pie
• zfs commands
> create a filesystem:
# zfs create poolname/fs-name
> set filesystem property:
# zfs set quota=size poolname/fs-name
# zfs set compression=on poolname/fs-name
# zfs set nfsshare=on poolname/fs-name
# zfs set recordsize=16k poolname/fs-name
> get filesystem property:
# zfs get compressratio poolname/fs-name
# zfs get all poolname/fs-name
> snapshot the filesystem:
# zfs snapshot poolname/fs-name@snapshotname
No newfs, No mkfs, No /etc/vfstab, No fsck

13. ZFS and Hybrid Storage
As easy as pie
• Read side
> Add ssd as a read side cache
> # zpool add poolname cache ssd-device
• Write side
> Add SSD as a ZFS Intent Log device
> # zpool add poolname log ssd-device

14. ZFS Performance Features
• Copy-on-write
> Turns Random writes to Sequential writes
• Dynamic Striping across all devices
> Maximize throughput
• Multiple Block Sizes
> Automatically chosen to match workload
• IO Pipelining
> Priority/Deadline scheduling, sorting, aggregation
• Intelligent prefetch
• Compression - Improves performance & Capacity
• Can safely use write cache on disks

15. Databases
Not Just Another Application
• Most Databases do their own buffering
> Filesystem caching can get in the way
> “double buffer” problem
• Most Databases do “prefetch”
> Filesystems prefetch can cause extra IO
> “directio” gets filesystem out of the way
• Have their own “log” mechanism.
> Interesting interaction with a transaction based filesystem
• Multiple blocks sizes
> Database & Transaction log, block sizes are normally different

16. Tuning ZFS for Databases
Tuning is Evil - Long live Tuning
• In general tuning is evil. Let ZFS do it for you.
• A few fine tuning tips for databases
> Get to the latest update of OS
> Set the recordsize to match database
block size
> Separate Transaction logs and data
onto separate zpools
> [Note: This will be addressed with the ZIL bypass property fix]
> Reduce the impact of double buffering by changing the caching
method to “metadata only”
> Use separate ZIL (ZFS Intent Log) preferably SSD
> Use SSD as secondary cache - L2ARC (Level 2. Adaptive
Replacement Cache)

17. ZFS tuning for MySQL
• Many tuning depends on storage engine
• For Innodb
> Prefer to cache in Innodb rather than ARC
zfs set primarycache=metadata poolname/database
> Set recordsize to 16k for data and 128k for log
zfs set recordsize=16k poolname/database
(Note: do this before you load any data)
> Turn off prefetch
set zfs:zfs_prefetch_disable = 1 (in /etc/system)
(File level prefetch not triggered if you change record size to 16k)
> Use raid0 or mirror over raidz
raidz is no suitable for random IO
> Add SSDs for either read side or write side based on workload
zpool add datapool cache ssd-disk
zpool create logpool ssd-disk3
In my.cnf set innodb_data_home_dir & innodb_log_group_home_dir

18. ZFS tuning for MySQL
• More tuning for Innodb
> Some device vendors flush cache even
when not needed. (eg. battery backed cache)
set zfs:zfs_nocacheflush = 1
> Turn on compression
zfs set compression=on poolname/database
ZFS does not turn on compression if less than 12.5% saving.
IO reduction may offset the extra cpu cost
> Disable double writes
innodb_doublewrite=0 (in my.cnf)
ZFS does not allow any partial writes so no need to guard against it.

19. ZFS tuning for PostgreSQL
• Postgres tuning hints
> Set recordsize to 8k
zfs set recordsize=8k poolname/database
> Turn down ARC cache.
set zfs:zfs_arc_max in /etc/system
> Add SSDs for either read side or write side based on workload
zpool add poolname cache ssd-name
zpool add poolname log ssd-name
> Use separate pool for log (preferably one with SSD) & data
initdb -X log_directory_name
create tablespace datatbs location 'database_directory_name'
create database mydb with tablespace datatbs
> Don’t forget to basic Postgres tuning on Solaris - (huge gains)
Set shared_buffers, temp_buffers, work_mem, maintenance_work_mem,
wal_sync_method, synchronous_commits etc
see: http://blogs.sun.com/jkshah/entry/best_practices_with_postgresql_8

20. ZFS tuning for Oracle
• Oracle tuning hints
> Set recordsize to match db_block_size (default 8k)
zfs set recordsize=8k poolname/database
> Use separate pool for Oracle logs
make sure record size of the log filesystem is left to the 128k default
> Add SSDs for either read side or write side based on workload
zpool add poolname cache ssd-name
zpool add poolname log ssd-name

21. Benchmark results
• Hardware
> Sun x4150 2 x Quad core 2.3 GHz Xeon
12 GB ram
3 x 10000 rpm drives
3 x 32 GB SSDs
• Software
> OpenSolaris 2009.06
> Postgres 8.3.7
> MySQL 5.4 beta

22. Benchmark results
• pgbench & Postgres
> command line: pgbench -c 10 -s 10 -t 10000 pgbench
Description TPS
Single disk ZFS 72 tps
2 Raid 0 disk + SSD as level 2 cache 241 tps
Above + general postgres optimization 2026 tps
+ all the data on SSD 2603 tps
+ data on hdd & log on SSD 4372 tps
+ primarycache=metadata 5003 tps

23. Benchmark results
• sysbench & mysql 5.4
> read/write test: sysbench --max-time=300 --max-requests=0 --test=oltp --
oltp-dist-type=special --oltp-table-size=10000000 --num-threads=20 run
Description TPS
Single disk ZFS 425 tps
raid0 ZFS 670 tps
+ SSD cache 788 tps
+ Separate intent log 1352 tps
+ With optimization 1809 tps

24. Benchmark results
• sysbench & mysql 5.4
> read test: sysbench --max-time=300 --max-requests=0 --test=oltp --oltp-
dist-type=special --oltp-table-size=10000000 --num-threads=20 --oltp-read-
only=on run
Description TPS
Single disk ZFS 786 tps
2 disk raid0 ZFS 1501 tps
+ SSD cache 1981 tps
+ Separate intent log on SSD 2567 tps
+ optimization 3065 tps

25. !"#$!%&'()*$'&(+,(-$.//012.$
Sun Unified Storage
*#%'345*6*5$!%(#+(5&#*
:$;<$=*='&-$>$?@A9BB;<$!6!$1)CDC
78)74+*#!8%3$!9(5(:5*
E($+'$F@;<$=*='&-G$142A?H<$!6H6$1)CDC
E($+'$IA?:$;< write-optimized SSDs
*#%'345*6*5$95"!%*'49(-(:5*
E($+'$F@;<$=*='&-G$ZFA?H<$!6H6$1)CDC
E($+'$FA?BB;<$&*,1.?FA?:;<$J&)+*$!!5C
6K+)L*M6K+)L*$/2"C+*&)#N$0'&$O.6
!9(5(:5*$95"!%*'49(-(:5*
!%(#+('+$@*(%"'*!$>(55$,&+*5!?
622$5,+,$%&'+'K'2C$,#1$5,+,$!*&L)K*C$V#K2"1*1 E($+'$?I:;<$=*='&-G$I::A?H<$!6H6$1)CDC
S6#,2-+)KCG$!#,(CR'+CG$7*(2)K,+)'#G$/'=(&*CC)'#G$4U!G$/VU!G$)!/!VWT E($+'$FA?BB;<$&*,1.?FA?:;<$J&)+*$!!5C
<")2+M)#$@A?$;P$Q+R*&#*+$('&+C 6K+)L*M6K+)L*$/2"C+*&)#N$0'&$O.6
7*='+*$/'#C'2*$S!*&),2$'&$Q+R*&#*+T
X)NR+CM'"+$Y,#,N*=*#+
&-%8&#(5$#*%;&'<8#)$=$9&##*9%868%3$>(55$,&+*5!?
IA?B$;P$Q+R*&#*+$S'(+)K,2T
@A?$;P$Q+R*&#*+$SK'((*&T$>$IA?;P$Q+R*&#*+$S'(+)K,2T
U/$'&$!/!V$O<6$0'&$+,(*$P,KD"(
!"#$ %&'(&)*+,&-./'#0)1*#+),23$ 456$ 7*8")&*1 9

26. Getting these systems at a discount
Sun Startup Essentials
• Exclusive program for startups
• Eligibility <6 yrs. Old, <150
sun.com/startup employees
• Co-marketing opportunities
• Funding assistance
• Deeply discounted storage and
servers certified for Linux,
Windows, and Solaris
• Hosting starting at $40
• Open source software, and
discounted MySQL
• Free email based tech support
• Free and discounted training on
Sun technologies
• Member-only webinars

27. Resources
• ZFS info: http://www.opensolaris.org/os/community/zfs/
• ZFS Best Practices Guide:
http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide
• ZFS Evil Tuning Guide:
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide
• Blogs of note:
> All things performance tuning:
http://blogs.sun.com/realneel
http://blogs.sun.com/roch
> Postgres tuning - Jignesh’s Blog
http://blogs.sun.com/jkshah
> Angelo’s blog
http://blogs.sun.com/angelo

28. Tuning Storage
Subsystem for
Databases
Angelo Rajadurai
angelo@sun.com
http://blogs.sun.com/angelo
twitter: rajadurai