IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in the DS8000

® ®
IBM System Storage DS8000 with SSDs
An In-Depth Look at SSD Performance in the DS8000

Performance White Paper
April 27, 2009

Lee LaFrese
Leslie Sutton
David Whitworth
Storage Systems Performance

Systems & Technology Group
International Business Machines Corporation

Notices and Disclaimer
Copyright © 2009 by International Business Machines Corporation.

No part of this document may be reproduced or transmitted in any form without written permission from
IBM Corporation.

Product data has been reviewed for accuracy as of the date of initial publication. Product data is subject
to change without notice. This information may include technical inaccuracies or typographical errors.
IBM may make improvements and/or changes in the product(s) and/or programs(s) at any time without
notice. References in this document to IBM products, programs, or services does not imply that IBM
intends to make such products, programs or services available in all countries in which IBM operates or
does business.

THE INFORMATION PROVIDED IN THIS DOCUMENT IS DISTRIBUTED "AS IS"
WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY
DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE OR NON-INFRINGEMENT. IBM shall have no responsibility to update
this information. IBM products are warranted according to the terms and conditions of the agreements
(e.g., IBM Customer Agreement, Statement of Limited Warranty, International Program License
Agreement, etc.) under which they are provided. IBM is not responsible for the performance or
interoperability of any non-IBM products discussed herein.

The performance data contained herein was obtained in a controlled, isolated environment. Actual
results that may be obtained in other operating environments may vary significantly. While IBM has
reviewed each item for accuracy in a specific situation, there is no guarantee that the same or similar
results will be obtained elsewhere.

Statements regarding IBM’s future direction and intent are subject to change or withdrawal without
notice, and represent goals and objectives only.

The provision of the information contained herein is not intended to, and does not; grant any right or
license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be
made, in writing, to:

IBM Director of Licensing
IBM Corporation
North Castle Drive
Armonk, NY 10504-1785
U.S.A.

IBM, System Storage, DS8000, FICON, FlashCopy, and z/OS are trademarks of International Business
Machines Corporation in the United States, other countries, or both.

Other company, product or service names may be trademarks or service marks of others.

IBM® System Storage DS8000® with SSDs
© Copyright IBM Corporation 2009 All Rights Reserved Page 1 of 14

Introduction
In February 2009, IBM announced the IBM System Storage DS8000 Turbo series with Solid State
Drives (SSDs).[1,2]

SSDs have no moving parts so they perform at electronic speeds without the mechanical delays
associated with traditional spinning Hard Disk Drives (HDDs). Because SSDs enable dramatically
higher throughput and lower response times for random I/O than HDDs, they provide the potential to
significantly lower operational costs in the data center despite substantially higher current acquisition
cost per GB. To realize these benefits, it is key to specifically target usage to applications that require
high IOPS/GB and/or low response times. Previously, customers were forced to buy large quantities of
15K RPM HDDs for these applications and use only a small portion of the capacity of each HDD
(known as short stroking) to meet their performance requirements. This practice can be costly as it
reduces capacity utilization. Now a large number of HDDs may be replaced with a small number of
SSDs, fully utilizing the capacity of each SSD and realizing improved system performance while also
saving on space, power, and cooling.

SSD Performance Best Practices
1. Place hot data on SSDs, warm data on 15K RPM HDDs, and cold data on 7200 RPM SATA
HDDs (see section on “Selecting Data to Place on SSDs” below for more details).

2. Use SSDs for applications that require low response times and are cache unfriendly.

3. SSDs are many times faster than HDDs at random I/O, but only slightly faster than HDDs at
sequential I/O. Data that is accessed randomly should be placed on SSDs and data that is
accessed sequentially should be placed on HDDs.

4. Use SSDs for applications that traditionally short stroke large numbers of 15K RPM HDDs.

5. Consider using a smaller storage cache when using SSDs than you might when using HDDs.
Disk Magic may be used to predict whether the combination of SSDs and a small storage cache
will meet the response time requirement. For hybrid DS8000s containing both SSDs and HDDs
it is advisable to use the same size storage cache as you would for a configuration of all HDDs so
that read hit ratios on the volumes placed on the HDDs are not reduced.

6. There is no additional benefit to using SSDs with remote copy services. In general, if SSDs are
used for remote copy source volumes they should also be used for the remote copy target
volumes. If not, then the secondary HDD based targets may become the bottleneck in the
system. This is especially problematic for synchronous replication (Metro Mirror) as delays will
be pushed back to applications. For asynchronous replication (Global Mirror) you may see an
increase in recovery point objective (RPO) if the throughput to the primary far exceeds the
secondary capability. This may or may not be acceptable depending on service level agreements.
The important thing here is to do the appropriate capacity planning before placing SSDs into a
remote copy environment.

7. SSDs may be used with FlashCopy either for source or target volumes. If SSDs are used for
source volumes while HDDs are used for the secondary, it is a good idea to do the FlashCopy
with background copy and during a period when the write rate to source volumes does not


exceed the capability of the targets. Additionally, although SSDs would likely perform well as a
Space Efficient FlashCopy (SEFLC) target repository, it does not fit with the basic premise of
the technology. SEFLC is intended for cost reduction by not fully provisioning the FlashCopy
target space. Since SSDs are costly it is likely that fully provisioned HDD space would be less
expensive and perform at least as well.

8. Use High Performance FICON for System z (zHPF) with SSDs for higher throughput and
additional reduction in the total response time.

Selecting Data to Place on SSDs
The DS8000 now supports 3 performance tiers of storage:

• Tier 0: SSDs. Highest performance and cost/GB

• Tier1: 15K RPM HDDs. High performance and lower cost/GB.

• Tier2: 7200 RPM HDDs. Lowest performance and cost/GB.

To maximize the benefit of SSDs it is important to only place data which requires a high IOPS/GB and
low response time on them. This data is referred to as “hot” data. Data that requires a low IOPS/GB is
referred to as “cold” data. Once hot data is moved to SSD, the remaining data may be cold enough to
allow moving a large portion of it to high capacity 7200 RPM HDDs and still meet the required
performance. Using the right mix of tier 0, 1, and 2 drives will provide optimal performance at the
minimum cost, power, cooling and space usage.

Determining the temperature of data and moving it to the proper tier can be difficult, so IBM provides
tools to help with this process.

Data placement on Power System servers
AIX provides performance tools that can be used to determine if a configuration has hot data that would
perform better if moved to SSDs.

First use the “iostat” command to check the CPU utilzation. iostat breaks down the CPU utilization into
usr, sys, idle, and iowait time. If there is a substantial amount of iowait time, then speeding up the
storage could potentially improve application performance.

The “iostat –D” command can then be used to get detailed storage performance statistics. For each LUN
it shows the percent time active and the throughput as well as details on read and write response times
and queueing delays. Look for LUNs that are 99 to 100% busy and have long response times/queueing
delays.

Next, run the “filemon” command to get detailed reports on the hotest AIX logical volumes and files.
Filemon’s detailed reports show statistics on read and write response times, I/O sizes, and seek
distances. Look for logical volumes and files that do a lot of very random small block I/O and have long
response times.

Consider moving data with the highest IOPS/GB to SSD first. The AIX migratepv command or the
Softek Data Mobility Services tools can be used to move data while it is online.

The whitepaper “Driving Business Value on Power Systems with Solid State Drives”3 provides a
detailed example of using to iostat and filemon to find hot data and of using Softek Data Mobility
Services to move that data to SSDs to improve application performance.

Data placement on System z servers
The System z I/O architecture provides a detailed breakdown of time spent executing I/O operations. On
a machine that runs z/OS in zArchitecture Mode with the Extended I/O Measurement Facility, these
times are stored into the I/O measurement word whenever an I/O interrupt occurs. Additionally, these
I/O measurements are aggregated into the channel measurement blocks for every device enabled to use
the I/O measurement facility.

One of the measurements surfaced to z/OS is the device disconnect time (DISC). Disconnect time is a
measure of the time spent resolving cache misses for read I/O operations. Disconnect time also includes
the time it takes to perform synchronous replication for write I/O operations using, for example, IBM’s
Metro Mirror technology on the DS8000. Solid state drives are ideally suited to benefit workloads that
are incurring high numbers of cache misses (for example, random reads) but are not expected to
substantially reduce the elapsed times for use of synchronous replication technologies.

For this reason, the DFSMS instrumentation that captures these components of I/O service time for
every I/O operation by data set has been enhanced to separate the disconnect time spent for read
operations from the disconnect time spent for write operations. (This support is available with DFSMS
SSD support APAR OA25559)

The SMF 42 subtype 6 record now includes two new fields:

• S42DSRDD reports the average read disconnect time for the interval per data set.
• S42DSRDT includes the total number of read operations performed to the data set
during the interval.

The DS8000 also provides the ability to obtain cache statistics for every volume in the storage
subsystem. These measurements include the count of the number of operations from DASD cache to the
backend storage, the number of random operations, the number of sequential reads and sequential
writes, the time to execute those operations, and the number of bytes transferred. These statistics are
reported in the SMF 74 subtype 5 record.

New z/OS tooling has been provided to analyze the SMF 42-6 and 74-5 records and produce a series of
reports identifying data sets and volumes that can benefit from residing on SSD. This new tooling is
based on SAS® software and is now available for download from the MVS Tools and Tips web page at
http://www.ibm.com/systems/z/os/zos/downloads

SMF data from an extended time period may be processed to get an overall historical view of I/O
activity, or much smaller time periods may be specified to focus on peak loads, batch windows, market
open, and other special conditions of interest.

The first report (generated from SMF 42-6 records) analyzes data sets for a high amount of total read
disconnect time which is a symptom of frequent read cache misses. Data set size is also factored in, if

available. Solid state drives significantly reduce the I/O service time caused by a cache miss, so data sets
appearing near the top of this list should be considered first for migration to SSD.

The second report (generated from SMF 74-5 records) analyzes DASD cache statistics to identify
volumes with high I/O rates. A pseudo device load is calculated in order to identify the devices that are
causing the highest load on the backend disks. Volumes with the highest pseudo device loads may be
considered for full volume migration to SSD.

The remaining reports are generated by merging the volume view with the data set view to identify the
hottest data sets on the most stressed volumes. Movement of individual data sets from these volumes
should be considered as an alternative to full volume migration when SSD space is limited.

More details on z/OS SSD instrumentation and tooling can be found in the article “Stop spinning your
storage wheels: z/OS Support for solid state drives in the DS8000 storage subsystem” in z/OS Hot
Topics Newsletter, Issue 20 at http://www.ibm.com/servers/eserver/zseries/zos/bkserv/hot_topics.html
and the FLASHDA User’s Guide at http://publibz.boulder.ibm.com/zoslib/pdf/flashda.pdf

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of
SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

Storage Modeling and Analysis Tools
Disk Magic is a performance modeling tool used by IBM which can help you predict the expected
performance of a DS8000 with a specific configuration running a specific workload. In the short term,
the tool has been enhanced to model the performance of SSDs. It will be possible to evaluate placement
of specific workloads on SSDs while other workloads remain on HDDs. Future enhancements are
planned to help the user determine which workloads or volumes are best to move from HDDs to SSDs.
As with any modeling tool, the quality of the output will depend on the accuracy of the input data
describing the workload characteristics. Customers should contact their IBM Storage Sales Specialist or
IBM Business Partner if they are considering SSDs for their DS8000 and would like to have a
performance modeling study done.

Updates to other IBM sales tools such as RMF Magic and Capacity Magic for SSD support are also
planned. Taken together, this tool set will enable thorough performance analysis of potential DS8000
SSD implementations.

Performance Results
Performance results in this section compare SSDs vs. HDDs in the DS8000 for a variety of different
workloads.

For random I/O, SSDs provide much higher throughput at a much lower response time than HDDs. The
SSDs supported in the DS8000 are so fast that the Device Adapter (DA) may become the performance
bottleneck on some random workloads. For sequential I/O, the DA was already the performance
bottleneck with HDDs. Using SSDs for sequential I/O does not provide a substantial increase in
performance.


System z Measurements on DS8000 with SSDs
System z measurements were done on DS8000 with SSDs and HDDs to compare their performance.
Results are shown both for both a single SSD rank and for a large configuration with 12 SSD ranks.

Figure 1 show 4KB read response times for cache hits, SSD reads, and HDD reads with short and long
seeks. These results were measured by a DB2 Sync I/O benchmark.

• Applications that require low response times may not be able to meet their requirement with HDDs
no matter how much they short stroke their HDDs. SSDs are a good match for these applications.

• Applications that traditionally use a very large storage cache may now be able to use a combination
of SSDs and a small cache and save on the capital cost and power usage of the large cache.

• zHPF has a lower response time than standard FICON. When performing either cache hits or I/O to
SSDs, using zHPF provides a significant additional reduction in the total response time.

For additional details on DB2 performance with SSDs in DS8000, see Jeffrey Berger’s paper
“Accessing DB2 for z/OS on SSDs” in the CMG Journal issue 123 (Spring 2009).

9
8
8

7
Response Time(ms)

6

5
3.9
4

3

2
0.74 0.84
1 0.29
0.23
0
zHPF cache Cache hit SSD+zHPF SSD Short seek Long seek
hit

Figure 1 - DB2 on CKD Sync I/O Read Response Time


The results in Figure 2 were measured by a DB2 I/O benchmark. They show random 4KB read
throughput and response times. SSD response times are very low across the curve. They are lower than
the minimum HDD response time for all data points.

20
Response Time(ms)

15
10
5
0
0 3 6 9 12 15 18
Throughput (K IOPS)

HDD Short seeks HDD Long seeks SSD

Figure 2 - DB2 on CKD Random Read Throughput/Response Time Curve

Figure 3 shows SSD vs. HDD performance in the following larger configurations:

1. 96 x 146GB SSDs on 6 Device Adapter(DA) pairs in 12 x 6+P RAID5 arrays. 84 SSDs are
active and 12 are hot spares.

2. 96 x 146GB HDDs on 6 DA pairs in 12 x 6+P RAID5 arrays. 84 HDDs are active and 12 are hot
spares.

3. 384 x 146GB HDDs on 6 DA pairs in 48 x 6+P RAID5 arrays. 360 HDDs are active and 24 are
hot spares.

These results show very good SSD response times all the way to 120K IOPS, where the DS8300
controllers become the performance bottleneck.


4KB Random Read: Large Config

20
R e s p o n s e T im e ( m s )

15
16-HPF-96SSD
10 16-HPF-96HDD
32-HPF-384HDD
5

0
0 20 40 60 80 100 120 140
Throughput (K IOPS)

Figure 3 - CKD 4KB Random Read Large Configuration

Distributed Systems Measurements
Distributed systems measurements were done with a Power Systems server running AIX attached to a
DS8000 with SSDs and HDDs. Results are shown for a single rank doing random and sequential reads
and writes.

Figure 4 shows distributed systems I/O response times. Note that they are similar to response times for
zHPF.


8
8
Response Time(ms)

6 Cache Hit

4 SSD Read Miss

2 0.7 HDD 15K RPM Rd
0.14 Miss
0

Figure 4 – Distributed Systems 4KB Read Response Time

Figure 5 shows SSD response times are very low across the curve. They are lower than the minimum
HDD response time for all data points.


15
Response Time(ms)

10

5

0
0 5 10 15 20 25
Throughput( K IOPS)

15K HDD SSD

Figure 5 – Distributed Systems 4KB Random Read: SSD vs HDD on one RAID5 Rank

Figure 6 shows SSDs provide about the same improvement on random writes as they do on random
reads. Note that random write performance is lower than random read performance on HDD and SSD
due to the extra drive I/Os done on RAID5 writes. Each application write requires 4 drive I/Os (2 reads
and 2 writes). Both the read and write SSD tests do about 20K IOPS to the SSD rank.

20

15
K IOPS

10 15K HDD
SSD
5

0
Read Write

Figure 6 – Distributed Systems 4KB Random IO: SSD vs HDD on one RAID5 Rank

Figure 7 shows that the scaling from one to two ranks on the same DA pair is very good for random I/O.

40
35
30
K IOPS

25
20 One 6+P
15 Two 6+Ps
10
5
0
Read Write

Figure 7 – Distributed Systems 4KB Random IO: One and Two SSD RAID5 Ranks

Figure 8 shows that the results are about the same for both HDDs and SSDs for sequential I/O. This is
because the DA is the bottleneck for sequential I/O for both HDDs and SSDs.

500

400
MB/sec

300
15K HDD
200 SSD
100

0
Read Write

Figure 8 – Distributed Systems Sequential I/O: SSD vs HDD on one RAID5 Rank


Figure 9 shows that the scaling from one to two ranks on the same DA pair is also very good for
sequential I/O.

1000

800
MB/sec

600
One 6+P
400 Two 6+Ps
200

0
Read Write

Figure 9 – Distributed Systems Sequential I/O: One and Two SSD RAID5 Ranks

Array Rebuild Results
The array rebuild rate for an idle 6+P SSD array is 72.6 MB/sec. This is around 10-15% faster than a
6+P array with 15K RPM HDDs. The faster rebuild rate lowers the risk of data loss due to a drive failure
during an array rebuild.

Reliability, Energy, Cooling and Space
In addition to the dramatic performance advantages SSDs provide over their HDD counterparts for
transaction-intensive applications, SSDs boast other key advantages, such as higher reliability, lower
energy usage, less cooling requirements, and the ability to reduce data center footprints. When combined
these advantages can add up to significant performance improvements as well as a lower costs structure
for business critical applications.

With no moving parts, SSDs can be more resilient than HDDs. In fact, the service life of an enterprise-
class HDD is around five years. According to one popular enterprise SSD manufacturer, SSDs may
double this with a service life of ten years. They go on to claim, “an SSD will last twice as long, on
average, when compared to HDD. This has a huge impact on the IOPS cost over time. In a ten-year
period the entire HDD population would need to be replaced (on average), while none of the SSD would
wear out.” 4


Since deploying SSDs can eliminate the expensive habit of “short stroking” HDDs to enable higher
throughput performance for critical applications, clients may see a considerable reduction in their
storage footprints. Recall that “short-stroking” HDDs require clients to use a small portion of the HDDs
capacity, which is the tradeoff for higher performance. By eliminating this, virtually 100% of the SSD is
utilized, which can greatly reduce the number of drives needed. As clients continue to struggle with
managing the tremendous growth of data they must manage, more efficient storage utilization can pay
big dividends, especially in metropolitan areas where real estate values are at a premium.

With respect to energy, each SSD uses about half of the energy of a 15K RPM HDD. For applications
that are able to replace large numbers of HDDs with a small number of SSDs, energy savings are
compounded.

The IBM technical brief titled “IBM System z® and System Storage DS8000:
Accelerating the SAP® Deposits Management Workload With Solid State Drives”5 provides an example
of the potential power, cooling and space savings from SSDs.

A brief summary of this example is included here.

The SAP® Deposits Management Workload was run on 2 configurations:
1. A traditional configuration using 896 HDDs
2. A hybrid configuration with 96 SSDs and 96 HDDs.

The hybrid SSD/HDD configuration provides the following benefits:

• 22% higher throughput at 50% lower response time

• 60% floor space savings

• 74% electrical power and cooling savings

Summary
SSDs are an emerging technology for enterprise storage clients that can show immediate benefits in
terms of performance as well as other operational characteristics. Given the distinct attributes and costs
of SSDs and HDDs, it’s clear that both drive types will coexist for some time. This will require a strong
focus on smart data placement and the subsequent data migration, which are two strategic areas for IBM
SSD solutions.

SSDs have no moving parts and provide much higher throughput and much lower response times for
random I/O than traditional spinning HDDs. They can also significantly lower operational costs in the
data center. Since SSDs currently have a substantially higher cost per GB than HDDs, they are
specifically targeted at applications that require high IOPS/GB and/or low response times and may
eliminate the practice of “short stroking” for these performance critical applications.

By eliminating the seek times of their spinning counterparts and providing direct access to data, SSDs
may dramatically boost performance and allow clients to maximize drive capacity utilization. This may
enable the replacement of a large number of HDDs with a much smaller quantity of SSDs, reducing
energy consumption, cooling expenses and floor space costs.


Appendix A: SSD in DS8000 Initial Configuration Recommendations
Use the following recommendations for configuring a DS8000 with SSDs.

1. For random I/O, SSDs provide much higher throughput at a much lower response time than
HDDs. The SSDs supported in the DS8000 are so fast that the DA may become the performance
bottleneck on some random workloads. It is therefore recommended to use just 16 SSDs per DA
pair to get the maximum performance from each SSD. (The DS8000 supports up to 8 pairs of
DAs.)

2. Using SSDs for sequential I/O does not provide a substantial increase in performance. It is
recommended that SSDs not be used for data that is predominantly accessed sequentially.

3. RAID5 is currently the only supported RAID level for SSDs in the DS8000. It provides more
user capacity than RAID6 or RAID10. The extra redundancy of RAID6 is not needed because
SSDs are at least as reliable as 15K RPM HDDs. The extra random write performance of
RAID10 is not needed because the SSDs are already more than fast enough.

4. Given SSDs current high cost/GB, configurations should typically have a mix of SSDs and
HDDs, with the SSDs being used for the hottest data.

5. In general, it is not recommended to use SSDs for remote copy source volumes if the targets are
HDDs unless careful capacity planning is done. The reason is that the high random write
throughput capability of the SSDs may overrun the relatively slower HDDs. If the write rate on
the source volumes is known to be within acceptable limits, then HDDs may be considered for
the secondary.

References
1
IBM Corp. 2009. .IBM System Storage DS8000 Turbo series
ftp://ftp.software.ibm.com/common/ssi/pm/sp/n/tsd00374usen/TSD00374USEN.PDF
2
IBM Corp. 2009. US Announcement Letter 109-120: IBM System Storage DS8000 series (Machine
types 2421, 2422, 2423, and 2424) delivers new security, scalability, and business continuity
capabilities.
http://www.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS109-
120&appname=USN
3
IBM Corp. 2009. Driving Business Value on Power Systems with Solid State Drives.
4
STEC Inc. SSD Performance and Power Advantage.
5
“IBM System z® and System Storage DS8000:
Accelerating the SAP® Deposits Management Workload With Solid State Drives”
http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101442


IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in the DS8000

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in the DS8000

Similar to IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in the DS8000 (20)

More from IBM India Smarter Computing

More from IBM India Smarter Computing (20)

Recently uploaded

Recently uploaded (20)

IBM System Storage DS8000 with SSDs An In-Depth Look at SSD Performance in the DS8000