1. IO Resource Management
CONFIDENTIAL |1
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

2. Contents
§ What is IO Resource Management?
§ IO Resource Management and DBRM
§ IORM Architecture
§ IORM Monitoring
§ IORM in Action
§ Business Justification for IORM
CONFIDENTIAL |2
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

3. About Me
§ John Clarke, Centroid
§ Specialized in
§ Oracle Exadata
§ Performance Tuning and RAC
§ Data Warehousing, 11g, Linux
§ Oracle Apps DBA
§ john.clarke@centroid.com, www.centroid.com
§ www.centroid.com
§ Blogs at www.centroid.com/knowledgebase
§ Sporadically dumps random thoughts at @jccclarke
CONFIDENTIAL |3
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

4. What is IORM?
§ IORM = IO Resource Management
§ IORM is a means to govern and prioritize IO requests to Exadata
storage servers
§ Often used in conjunction with DBRM
§ If consolidating databases on Exadata, use IORM
§ If you’re not consolidating, still often makes sense!
CONFIDENTIAL |4
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

5. What is IORM?
Some basics …
§ Exadata “IORM Plan” = Inter-Database plan + Category plan
§ A category plan specifies resource allocations per category of resource
consumer groups
– Categories are assigned to DBRM resource consumer groups
– Categories (can) span databases
– Consumer groups are within a database
§ A Inter-Database plan specifies IO resource allocations per database
§ An Intra-Database plan = a DBRM Plan
CONFIDENTIAL |5
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

6. What is IORM? (continued)
There are 3 IORM methods
§ Category IORM
– (implies categories assigned to resource consumer groups)
§ Inter-Database IORM
– (i.e., IO resource allocation between databases)
§ Intra-Database IORM
– (i.e., allocations within a database)
You can use combinations of the above …
CONFIDENTIAL |6
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

7. What is IORM? – In Pictures
PROD consumer
groups
CG: FIN_OLTP CAT: NORMAL_PRIORITY
DB% for PROD PROD CG: FIN_REPORTS CAT: LOW_PRIORITY
CG: SHIPPING_OLTP CAT: HIGH_PRIORITY
CG: DW_CRITICAL CAT: HIGH_PRIORITY
DB% for DWPRD DWPRD CG: DW_NORMAL CAT: NORMAL_PRIORITY
DWPRD consumer
groups
DEV consumer
group
DB% for DEV DEV CG: DEV_ALL CAT: LOW_PRIORITY
CONFIDENTIAL |7
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

8. Database Resource Management (DBRM)
§ Database Resource Management (DBRM) – see
http://docs.oracle.com/cd/E14072_01/server.112/e10595/dbrm.htm
§ Used within a single database when you’ve got workloads with different
performance requirements
§ Resource consumer groups allow you to group sessions by workload
§ Administrator specifies resource allocations per consumer group
§ Administrator maps sessions to consumer groups
§ The combination of consumer groups and session mappings = a
resource plan
§ A DBRM plan is also called an Intra-Database resource plan
CONFIDENTIAL |8
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

9. IORM and DBRM
Intra-Database IORM
§ If you have multiple workloads within a database, you need an Intra-Database
resource plan
§ If you only have only one database on your Exadata, the Intra-Database plan is
all you need for IORM
§ Exadata automatically prioritizes IO requests based on this Intra-Database plan
CONFIDENTIAL |9
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

10. IORM and DBRM
Inter-Database IORM
§ If you have multiple databases on an Exadata that you wish to control IO
allocations amongst, you need an Inter-Database plan
§ Rules for an Inter-Database plan specify IO allotments per database, not
resource consumer groups
CONFIDENTIAL | 10
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

11. IORM and DBRM
Category IORM
§ A category plan is used to control IO resource utilization by classification of
work being done – i.e., by categories of consumer groups
§ Category plans assume Intra-Database plans are in effect
CONFIDENTIAL | 11
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

12. IORM Architecture
DBRM runs in the database
IORM part of CELLSRV
CONFIDENTIAL | 12
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

13. IORM Architecture – Rules of Engagement
§ IORM only kicks in to prioritize IO when needed
§ “When needed” = when IO is saturated to cell disks
§ Any disk allocation not fully utilized is made available to other
workloads
§ Background IO is scheduled based on priority relative to user IO
§ Redo and control file writes always have higher priority
§ DBWR writes scheduled at same priority as user IO
CONFIDENTIAL | 13
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

14. IORM Architecture – IO Queues
§ For each cell disk, each database accessing cell has one
IO queue per consumer group and three background
queues
§ Background IO queues are mapped to high, medium, and
low priority requests with different IO types mapped to each
queue
§ If no Intra-Database plans are set, all non-background IO
requests are grouped into a single queue called
OTHER_GROUPS
CONFIDENTIAL | 14
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

15. IORM Architecture – IO Queues
No consumer groups
No consumer groups
All user IO goes to
Other Groups
queue
OTHER_GROUPS OTHER_GROUPS
High BG High BG
Medium BG Medium BG
Low BG Low BG
PROD Queues DEV Queues
Cell Disk CD_00_cm01cel01
CONFIDENTIAL | 15
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

16. IORM Architecture – IO Queues
DBRM consumer groups
CG: OLTP
PROD configured
CG: REPT with three resource
consumer groups
CG: FINANCE
OTHER_GROUPS OTHER_GROUPS
High BG High BG
Medium BG Medium BG
Low BG Low BG
PROD Queues DEV Queues
Cell Disk CD_00_cm01cel01
CONFIDENTIAL | 16
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

17. IORM Architecture
iDB message contains database
name, consumer group + category,
IO type, extents to get, etc.
PROD DWPRD DEV
PROD Cell disk 1 DWPRD Cell Disk 7 DEV Cell Disk 11
FIN_OLTP DW_CRITICAL DEV_ALL
FIN_REPORTS DW_NORMAL OTHER_GROUPS
SHIPPING_OLTP OTHER_GROUPS HIGH
OTHER_GROUPS HIGH MEDIUM
HIGH MEDIUM LOW
MEDIUM LOW
LOW
PROD Cell disk 2 DWPRD Cell Disk 2 DEV Cell Disk 2
PROD Cell disk 3 DWPRD Cell Disk 3 DEV Cell Disk 3
PROD Cell disk N DWPRD Cell Disk N DEV Cell Disk N
IO queues and cell services
Disk 1 Disk 3 Disk 5 Disk 7 Disk 9 Disk 11 Exadata Cell Disks
Disk 2 Disk 4 Disk 6 Disk 8 Disk 10 Disk 12
CONFIDENTIAL | 17
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

18. IORM Architecture
PROD DWPRD DEV
PROD Cell disk 1 DWPRD Cell Disk 7 DEV Cell Disk 11
FIN_OLTP DW_CRITICAL DEV_ALL
FIN_REPORTS DW_NORMAL OTHER_GROUPS
SHIPPING_OLTP OTHER_GROUPS HIGH
OTHER_GROUPS HIGH MEDIUM
HIGH MEDIUM LOW
MEDIUM LOW
LOW
PROD Cell disk N DWPRD Cell Disk N DEV Cell Disk N
IORM
IO queues and cell services
Disk IO queues
ordered after IORM
does its logic
Disk 1 Disk 3 Disk 5 Disk 7 Disk 9 Disk 11 Exadata Cell Disks
Disk 2 Disk 4 Disk 6 Disk 8 Disk 10 Disk 12
CONFIDENTIAL | 18
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

19. IORM Architecture – The Math
Methods are evaluated based on a fixed order:
§ Category IORM is evaluated first
Intra CG% / sum (x) =
§ Inter-Database IORM next “Consumer group
weight” (my words …)
§ Intra-Database IORM next
“cat%” = percentage of
CG% = (Intra CG% / Σ(x)) * db% * cat% resource allocation for
the category in which
the consumer group
belongs
“CG%” = IORM “Intra CG%” =
determined resource
resource allocation for
allocation for consumer group “x” = sum of Intra- db% = percentage of
consumer group within an Intra- Database consumer database allocation in
sessions Database plan group allocations for all the Inter-Database plan
consumer groups in the
same category
CONFIDENTIAL | 19
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

20. IORM Architecture – The Math
All User IO
No IORM
100%
9.87% = Cat% * DB% * CG weight
Cat% = 65%
Category IORM
Cat1: 65% Cat2: 25% Cat3: 10%= 40%
DB%
CG Weight = 60% / (sum over Cat1)
= 60% / (80% + 8% + 60% + 10%)
= 37.7%
9.87% = 65% * 40% * 37.7%
Inter-Database IORM
DB1: 60% DB2: 40% DB1: 60% DB2: 40% DB1: 60% DB2: 40%
39% 26% 15% 10% 6% 4%
CG1: 80% L1, Category=Cat1 19.97% CG1: 60% L1, Category=Cat1 9.87%
CG2: 40% L2, Category=Cat1 1.97% CG2: 10% L1, Category=Cat1 1.65%
Intra-Database IORM
CG3: 40% L2, Category=Cat3 0.38% CG3: 40% L2, Category=Cat2 5.45%
DB1 DB2
CG4: 50% L3, Category=Cat2 6.82% CG4: 40% L2, Category=Cat3 3.75%
CONFIDENTIAL | 20
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

21. Or if you like Excel …
Enter values in the blue
IORM resource calculation
https://dl.dropbox.com/u/92079964/IORM%20Planning%20Calculator.xlsx
CONFIDENTIAL | 21
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

22. Monitoring IORM
Here’s what I’m going to talk about in the next few slides …
§ IORM metrics
§ Using CellCLI to monitor IORM
§ metric_iorm.pl (MOS note 1337265.1)
§ cellsrv_statedump
§ What I like to do with the data
CONFIDENTIAL | 22
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

23. Monitoring IORM: IORM Metrics
§ Exadata tracks IORM metrics in the storage cell. Metrics include:
– Counter and wait metrics
– Utilization and throughput metrics
§ Exadata metrics can be queried from CellCLI as of:
– Now, using LIST METRICCURRENT
– Historically, using LIST METRICHISTORY
§ Exadata groups IORM metrics by:
– Category IORM method
§ objectType=‘IORM_CATEGORY’
§ name like ‘CT_.*’
– Inter-Database IORM method
§ objectType=‘IORM_DATABASE’
§ name like ‘DB_.*’
– Consumer group IORM method
§ objectType = ‘IORM_CONSUMER_GROUP’
§ name like ‘CG_.*’
CONFIDENTIAL | 23
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

24. Monitoring IORM: Category IORM Metrics
[root@cm01cel01 ~]# cellcli -e list metricdefinition where objectType='IORM_CATEGORY'!
Name Description Type Unit
CT_FC_IO_BY_SEC Number of megabytes of I/O per second for this category to flash cache Instantaneous MB/sec
CT_FC_IO_RQ Number of IO requests issued by an IORM category to flash cache Cumulative IO requests
CT_FC_IO_RQ_SEC Number of IO requests issued by an IORM category to flash cache per second Rate IO/sec
CT_FD_IO_BY_SEC Number of megabytes of I/O per second for this category to flash disks Instantaneous MB/sec
CT_FD_IO_LOAD Average I/O load from this category for flash disks Instantaneous Number
CT_FD_IO_RQ_LG Number of large IO requests issued by an IORM category to flash disks Cumulative IO requests
CT_FD_IO_RQ_LG_SEC Number of large IO requests issued by an IORM category to flash disks per second Rate IO/sec
CT_FD_IO_RQ_SM Number of small IO requests issued by an IORM category to flash disks Cumulative IO requests
CT_FD_IO_RQ_SM_SEC Number of small IO requests issued by an IORM category to flash disks per second Rate IO/sec
Throughput, MB/sec
CT_IO_BY_SEC Number of megabytes of I/O per second for this category to hard disks Instantaneous MB/sec
by category
CT_IO_LOAD Average I/O load from this category for hard disks Instantaneous Number
CT_IO_RQ_LG Number of large IO requests issued by an IORM category to hard disks Cumulative IO requests
CT_IO_RQ_LG_SEC Number of large IO requests issued by an IORM category to hard disks per second Rate IO/sec
CT_IO_RQ_SM Number of small IO requests issued by an IORM category to hard disks Cumulative IO requests/sec
IO requests
CT_IO_RQ_SM_SEC Number of small IO requests issued by an IORM category to hard disks per second Rate IO/sec
CT_IO_UTIL_LG Percentage of disk resources utilized by large requests from this category Instantaneous %
Disk utilization by
CT_IO_UTIL_SM Percentage of disk resources utilized by small requests from this category Instantaneous category
%
CT_IO_WT_LG IORM wait time for large IO requests issued by an IORM category Cumulative ms
CT_IO_WT_LG_RQ Average IORM wait time per request for large IO requests issued by an IORM category Rate ms/request
CT_IO_WT_SM IORM wait time for small IO requests issued by an IORM category Cumulative IO wait ms
time/sec
CT_IO_WT_SM_RQ Average IORM wait time per request for small IO requests issued by an IORM category Rate ms/request
CONFIDENTIAL | 24
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

25. Monitoring IORM: Inter-Database IORM Metrics
[root@cm01cel01 ~]# cellcli -e list metricdefinition where objectType='IORM_DATABASE'!
Name Description Type Unit
DB_FC_IO_BY_SEC Number of megabytes of I/O per second for this database to flash cache Instantaneous MB/sec
DB_FC_IO_RQ Number of IO requests issued by a database to flash cache Cumulative IO requests
DB_FC_IO_RQ_SEC Number of IO requests issued by a database to flash cache per second Rate IO/sec
DB_FD_IO_BY_SEC Number of megabytes of I/O per second for this database to flash disks Instantaneous MB/sec
DB_FD_IO_LOAD Average I/O load from this database for flash disks Instantaneous Number
DB_FD_IO_RQ_LG Number of large IO requests issued by a database to flash disks Cumulative IO requests
DB_FD_IO_RQ_LG_SEC Number of large IO requests issued by a database to flash disks per second Rate IO/sec
DB_FD_IO_RQ_SM Number of small IO requests issued by a database to flash disks Cumulative IO requests
DB_FD_IO_RQ_SM_SEC Number of small IO requests issued by a database to flash disks per second Rate IO/sec
Throughput, MB/sec
DB_IO_BY_SEC Number of megabytes of I/O per second for this database to hard disks Instantaneous MB/sec
by DB
DB_IO_LOAD Average I/O load from this database for hard disks Instantaneous Number
DB_IO_RQ_LG Number of large IO requests issued by a database to hard disks Cumulative IO requests
DB_IO_RQ_LG_SEC Number of large IO requests issued by a database to hard disks per second Rate IO/sec
DB_IO_RQ_SM Number of small IO requests issued by a database to hard disks Cumulative IO requests/sec
IO requests
DB_IO_RQ_SM_SEC Number of small IO requests issued by a database to hard disks per second Rate IO/sec
DB_IO_UTIL_LG Percentage of disk resources utilized by large requests from this database Instantaneous %
Disk utilization by
DB_IO_UTIL_SM Percentage of disk resources utilized by small requests from this database Instantaneous DB
%
DB_IO_WT_LG IORM wait time for large IO requests issued by a database Cumulative ms
DB_IO_WT_LG_RQ Average IORM wait time per request for large IO requests issued by a database Rate ms/request
DB_IO_WT_SM IORM wait time for small IO requests issued by a database Cumulative IO wait ms
time/sec
DB_IO_WT_SM_RQ Average IORM wait time per request for small IO requests issued by a database Rate ms/request
CONFIDENTIAL | 25
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

26. Monitoring IORM: Inter-Database IORM Metrics (cont)
[root@cm01cel01 ~]# cellcli -e list metricdefinition where objectType='IORM_DATABASE'!
Name Description Type Unit
DB_FL_IO_BY The number of MB written to the Flash Log Cumulative MB
DB_FL_IO_BY_SEC The number of MB written per second to the Flash Log Instantaneous MB/sec
DB_FL_IO_RQ The number of I/O requests issued to the Flash Log Cumulative IO Requests
DB_FL_IO_RQ_SEC The number of I/O requests per second issued to the Flash Log Instantaneous IO/sec
CONFIDENTIAL | 26
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

27. Monitoring IORM: Consumer Group IORM Metrics
[root@cm01cel01 ~]# cellcli -e list metricdefinition where objectType='IORM_CONSUMER_GROUP'!
Name Description Type Unit
CG_FC_IO_BY_SEC Number of megabytes of I/O per second for this consumer group to flash cache Instantaneous MB/sec
CG_FC_IO_RQ Number of IO requests issued by a consumer group to flash cache Cumulative IO requests
CG_FC_IO_RQ_SEC Number of IO requests issued by a consumer group to flash cache per second Rate IO/sec
CG_FD_IO_BY_SEC Number of megabytes of I/O per second for this consumer group to flash disks Instantaneous MB/sec
CG_FD_IO_LOAD Average I/O load from this consumer group for flash disks Instantaneous Number
CG_FD_IO_RQ_LG Number of large IO requests issued by a consumer group to flash disks Cumulative IO requests
CG_FD_IO_RQ_LG_SEC Number of large IO requests issued by a consumer group to flash disks per second Rate IO/sec
CG_FD_IO_RQ_SM Number of small IO requests issued by a consumer group to flash disks Cumulative IO requests
CG_FD_IO_RQ_SM_SEC Number of small IO requests issued by a consumer group to flash disks per second Rate IO/sec
Throughput, MB/sec
CG_IO_BY_SEC Number of megabytes of I/O per second for this consumer group to hard disks Instantaneous MB/sec
by CG
CG_IO_LOAD Average I/O load from this consumer group for hard disks Instantaneous Number
CG_IO_RQ_LG Number of large IO requests issued by a consumer group category to hard disks Cumulative IO requests
CG_IO_RQ_LG_SEC Number of large IO requests issued by a consumer group to hard disks per second Rate IO/sec
CG_IO_RQ_SM Number of small IO requests issued by a consumer group to hard disks Cumulative IO requests/sec
IO requests
CG_IO_RQ_SM_SEC Number of small IO requests issued by a consumer group to hard disks per second Rate IO/sec
CG_IO_UTIL_LG Percentage of disk resources utilized by large requests from this consumer group Instantaneous %
Disk utilization by
CG_IO_UTIL_SM Percentage of disk resources utilized by small requests from this consumer group Instantaneous CG
%
CG_IO_WT_LG IORM wait time for large IO requests issued by a consumer group Cumulative ms
CG_IO_WT_LG_RQ Average IORM wait time per request for large IO requests issued by a consumer group Rate ms/request
CG_IO_WT_SM IORM wait time for small IO requests issued by a consumer group Cumulative IO wait ms
time/sec
CG_IO_WT_SM_RQ Average IORM wait time per request for small IO requests issued by a consumer group Rate ms/request
CONFIDENTIAL | 27
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

28. Monitoring IORM
§ Even if you don’t implement IORM plans, database IO is tagged and
some database metrics are populated
§ These ‘DB_.*’ metrics can be valuable No IORM plan active
Did some large IO
on DWPRD, metrics
displayed
CONFIDENTIAL | 28
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

29. Monitoring IORM: Using CellCLI
§ Use cellcli “list” to display IORM metrics
§ “list metriccurrent” displays current metrics and “list metrichistory”
displays historical metrics
§ Query with dcli to show metrics on all the nodes
§ Examples:
Display historical metrics for
Show historical metrics
all cells using dcli
based on date range
Display current metrics for
MB/sec per database
CONFIDENTIAL | 29
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

30. Monitoring IORM: Common challenges with CellCLI
1. Difficult to remember metric names
2. Difficult to remember/type cellcli query syntax, especially with
metrichistory and dates
3. Lack of ability to monitor metrics over time interval, real-time (like
“sar –u <interval> <iterations>”
4. Difficult to assemble output and make it usable
Here’s how I address it:
§ celldig.sh => Addresses #1 & #2 above
§ cellstats.sh => Addresses #3 above
§ celliormbundle.sh => Addresses #4 above
§ celliormbundle2.sh => Uses ageInMinutes criteria (got lazy)
https://dl.dropbox.com/u/92079964/cellscr.zip
CONFIDENTIAL | 30
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

31. Monitoring IORM: celldig.sh
Download and try it out …
CONFIDENTIAL | 31
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

32. Monitoring IORM: celldig.sh
Start from now – 1 day
and display 20 min of data
Here’s the cellcli command
CONFIDENTIAL | 32
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

33. Monitoring IORM: cellstats.sh
Query all cells, 2 second
intervals, 5 iterations
Query single cell, 2 second
intervals, 5 iterations
CONFIDENTIAL | 33
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

34. Monitoring IORM: celliormbundle.sh
What it does:
1. Collects cell server metrics for IORM-related objects
2. Accepts a “from” and “to” date range
3. Output directed to temp files and loaded from external tables
Prep work: Run celliormbundle_setup.sh
celliormbundle.sh and celliormbundle2.sh will be used to gather
performance metrics for IORM test scenarios later …
CONFIDENTIAL | 34
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

35. Monitoring IORM: celliormbundle.sh
Usage
Execution
Loading IORM_METRICS
CONFIDENTIAL | 35
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

36. Monitoring IORM: celliormbundle2.sh
Usage
Execution
Load IORM_METRICS
CONFIDENTIAL | 36
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

37. Monitoring IORM: metric_iorm.pl
§ metric_iorm.pl nicely summarizes IORM metrics and can be
downloaded from MOS note 1337265.1
§ Without any input parameters, displays current metrics
§ With collectionTime-scoped input parameters, displays historical
metrics summarized by minute
§ Good way to understand what resources are consuming the most IO,
whether disk latencies are reasonable, flash cache hit metrics per
database, and overall IORM plan impact on performance
§ Works regardless of whether IORM plan is active
CONFIDENTIAL | 37
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

38. Monitoring IORM: metric_iorm.pl
Example: Show current metrics for a cell:
CONFIDENTIAL | 38
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

39. Monitoring IORM: metric_iorm.pl
Example: Display metrics between 9:00am and 9:30am on 7/12/2012
Summarizes cell
stats as well
CONFIDENTIAL | 39
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

40. Monitoring IORM: metric_iorm.pl
Example: Display metrics from ½ hour ago for a duration of 10 minutes
Use “ageInMinutes” syntax
CONFIDENTIAL | 40
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

41. Monitoring IORM: metric_iorm.pl
Example: Collect across all cells using dcli
CONFIDENTIAL | 41
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

42. Monitoring IORM: Inside a CELLSRV statedump
§ Event cellsrv.cellsrv_statedump dumps current state of cell to trace file
§ Contains some good IORM information
§ Invoke like this:
CONFIDENTIAL | 42
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

43. Monitoring IORM: Inside a CELLSRV statedump
Under the “IORM State Dump” section, trace file organized like:
§ IORM Enabled heading
– Current IORM Plans
§ Category Plan details
– Map of database and consumer groups to categories
§ Inter-Database Plan details
§ Intra-Database Plan details
§ Complete Mappings
– Queued IO Requests for cellDisk /dev/sdX (repeated for each cell disk)
§ Category Plan
– Inter-Database Plan
» Intra-Database Plan
» IO queuing details per consumer group, database, category, and cell
disk
– Queued IO Requests for cellDisk /dev/sdY (repeat previous section)
CONFIDENTIAL | 43
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

44. Monitoring IORM: Inside a CELLSRV statedump
Search for this section in trace file
Currently, no IORM plan is enabled
CONFIDENTIAL | 44
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

45. Monitoring IORM: Inside a CELLSRV statedump
Sections arranged by disk device
IORM boost = ratio of IOs skipped
to IOs scheduled
= relative measure of how active
IO scheduling is happening
Broken down by Category first
… then database
… then intra db consumer group
CONFIDENTIAL | 45
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

46. Monitoring IORM: Inside a CELLSRV statedump
Here’s one with IORM enabled
Dumps Category plan first
… then DB + CG to Category mapping
CONFIDENTIAL | 46
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

47. Monitoring IORM: Inside a CELLSRV statedump
Next is displays Inter-Database plan
Intra-Database plans for each database
CONFIDENTIAL | 47
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

48. Monitoring IORM: Inside a CELLSRV statedump
For each devices, shows queued IO
requests – we have none here
Detail section similar to before
CONFIDENTIAL | 48
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

49. Monitoring IORM: What to do with the data
I like to do this:
§ When analyzing a specific time period:
– Run celliormbundle.sh or celliormbundle2.sh
– Query the data in CELLIORM.IORM_METRICS via SQL
– Dump output to Excel and graph
– Capture test timings
§ When analyzing performance for “right now”:
– Run metric_iorm.pl with no arguments and examine output
– Run cellstats.sh and iterate over a period of time for a specific metric for a metric or 2
CONFIDENTIAL | 49
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

50. IORM in Action
§ Test #1: Inter-Database IORM plan with 4 databases
§ Test #2: Inter-Database IORM plan with 4 databases + DB limits
§ Test #3: Category IORM plan
§ Test #4: Inter-Database IORM + Category IORM
§ Test #5: Inter-Database IORM + Category IORM, give DWPRD all the
resources it needs
CONFIDENTIAL | 50
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

51. IORM in Action
For each test, we’ll:
§ Plan
§ Show expected resource allocations
§ Implement
§ Validate
§ Conduct a load test
§ Analyze
CONFIDENTIAL | 51
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

52. IORM in Action: Baseline Test
First I’ll run a baseline without IORM plans active and analyze data …
§ I’ll use a modified version of Karl Arao’s IOsaturationtoolkit-v2 (
http://karlarao.wordpress.com/2012/05/14/iosaturationtoolkit-v2-with-
iorm-and-awesome-text-graph/) to generate load
§ In this, I’ll run 4 concurrent full-scans against a large table with 3
separate schemas (ORACLE,ORACLE2, and ORACLE3) across 4
databases (DWPRD, DWPROD, VISX, and VISY)
§ I’ll analyze the following:
– Output from IOsaturationtoolkit-v2 log files
– Data in IORM_METRICS from celliormbundle.sh
– Data from output of metric_iorm.pl
Test started at 2012-07-17T17:22:43 and ended at 2012-07-17T17:28:45
CONFIDENTIAL | 52
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

53. IORM in Action: Baseline Test
IOsaturationtoolkit-v2 Results
Statistics
relatively random
CONFIDENTIAL | 53
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

54. IORM in Action: Test 1 Planning
Inter-Database IORM with 4 databases
Inter-Database IO allocation:
§ VISX: 55%
§ DWPRD: 25%
§ DWPROD: 15%
§ VISY: 5%
CONFIDENTIAL | 54
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

55. IORM in Action: Test 1 Resource Allocations
Inter-Database IORM with 4 databases
Empty the Category cells
Enter IORM % per DB
per plan
Select DB, enter arbitrary
consumer groups, select OTHER/ALL
Category
Inter-Database IO allocations calculated
CONFIDENTIAL | 55
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

56. IORM in Action: Test 1 Implement
Inter-Database IORM with 4 databases
I like to do this first …
Set VISX to 55%, level 1
Set DWPRD to 25%, level 1
Set DWPROD to 15%, level 1
Set VISY to 5%, level 1
Set OTHER to 100%, level 2
Activate plan
CONFIDENTIAL | 56
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

57. IORM in Action: Test 1 Validate
Inter-Database plan using 4 databases
“list iormplan detail”
CONFIDENTIAL | 57
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

58. IORM in Action: Test 1 Validate (continued)
Inter-Database plan using 4 databases
Issue cell server statedump
IORM plan active
Just 2 categories …
OTHER = all DBs for now .
OTHER_GROUPS since
no Intra-Database plans
CONFIDENTIAL | 58
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

59. IORM in Action: Test 1 Load Test
Inter-Database IORM with 4 databases
We’ll conduct the same load test we did in our Test 0 baseline test
Sampled DB_IO_WT_LG_RQ
(avg. IO wait time / request / large)
Notice wait times per DB
align with inter-database plan
CONFIDENTIAL | 59
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

60. IORM in Action: Test 1 Analyze
Inter-Database IORM with 4 databases
IOsaturationtoolkit-v2 Results:
Test duration times matched expectations IO throughput matched expectations
visx < dwprd < dwprod < visy visx > dwprd > dwprod > visy
CONFIDENTIAL | 60
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

61. IORM in Action: Test 1 Analyze
Inter-Database IORM with 4 databases
CELLORM.IORM_METRIC data
Query or MBPS, IOPS,
Utilization, and Waits
CONFIDENTIAL | 61
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

62. IORM in Action: Test 1 Analyze
Inter-Database IORM with 4 databases
CELLORM.IORM_METRIC data
As expected, VISX
had best throughput
CONFIDENTIAL | 62
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

63. IORM in Action: Test 1 Analyze
Inter-Database IORM with 4 databases
CELLORM.IORM_METRIC data
VISX had highest
IOPs and VISY the lowest
CONFIDENTIAL | 63
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

64. IORM in Action: Test 1 Analyze
Inter-Database IORM with 4 databases
CELLORM.IORM_METRIC data
Utilization numbers match
expectations
CONFIDENTIAL | 64
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

65. IORM in Action: Test 1 Analyze
Inter-Database IORM with 4 databases
CELLORM.IORM_METRIC data
VISY waited the most,
on average, per request
CONFIDENTIAL | 65
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

66. IORM in Action: Test 1 Analyze
Inter-Database IORM with 4 databases
metric_iorm.pl output
Seeing non-zero queue
times
CONFIDENTIAL | 66
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

67. IORM in Action: Test 2 Planning
Inter-Database IORM plan with 4 databases and IORM database limits
§ Inter-Database plan requirements same as Test 1
– VISX: 55%
– DWPRD: 25%
– DWPROD: 15%
– VISXY: 5%
§ IORM database IO limits also imposed on each database:
– VISX: 80%
– DWPRD: 10%
– DWPROD: 5%
– VISXY: 5%
CONFIDENTIAL | 67
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

68. IORM in Action: Test 2 Resource Allocations
Inter-Database IORM plan with 4 databases and IORM database limits
IO limits per database
Adjusted resource limits
CONFIDENTIAL | 68
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

69. IORM in Action: Test 2 Implement
Inter-Database IORM plan with 4 databases and IORM database limits
Note limit clauses
CONFIDENTIAL | 69
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

70. IORM in Action: Test 2 Validate
Inter-Database IORM plan with 4 databases and IORM database limits
CONFIDENTIAL | 70
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

71. IORM in Action: Test 2 Validate
Inter-Database IORM plan with 4 databases and IORM database limits
DB limit
CONFIDENTIAL | 71
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

72. IORM in Action: Test 2 Load Test
Inter-Database IORM plan with 4 databases and IORM database limits
We’ll conduct the same load test we did previously
Note wait distribution
CONFIDENTIAL | 72
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

73. IORM in Action: Test 2 Analyze
Inter-Database IORM plan with 4 databases and IORM database limits
IOsaturationtoolkit-v2 Results:
VISX and DWPRD took the least amount of time IO throughput numbers significantly lower than Test 1
but limits caused overall IO time to be less than in Test 1 Relative MB/s/database as expected
CONFIDENTIAL | 73
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

74. IORM in Action: Test 2 Analyze
Inter-Database IORM plan with 4 databases and IORM database limits
CELLIORM.IORM_METRICS data:
Throughput less than Test 1, as
expected
CONFIDENTIAL | 74
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

75. IORM in Action: Test 2 Analyze
Inter-Database IORM plan with 4 databases and IORM database limits
CELLIORM.IORM_METRICS data:
IOPs less than Test 1, as
expected
CONFIDENTIAL | 75
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

76. IORM in Action: Test 2 Analyze
Inter-Database IORM plan with 4 databases and IORM database limits
CELLIORM.IORM_METRICS data:
Utilization less – imposing
limits yields IO capacity
CONFIDENTIAL | 76
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

77. IORM in Action: Test 2 Analyze
Inter-Database IORM plan with 4 databases and IORM database limits
CELLIORM.IORM_METRICS data:
Significantly more queue time,
a reflection of limits
CONFIDENTIAL | 77
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

78. IORM in Action: Test 3 Planning
Category IORM Plan
§ No Inter-Database Plan
§ Category Plan
– CAT_HIGH = 70%, Level 1
– CAT_MEDIUM = 20%, Level 1
– CAT_LOW = 80%, Level 2
§ Consumer Group Mappings
– DWPRD:
§ CG_CRITICAL (80% L1 intra) assigned CAT_HIGH category, mapped to user “ORACLE”
§ CG_NORMAL (20% L1 intra) assigned CAT_MEDIUM category, mapped to user “ORACLE2”
§ CG_OTHER (100% L2 intra) assigned CAT_LOW, mapped to user “ORACLE3”
– DWPROD:
§ CG_CRITICAL (70% L1 intra) assigned CAT_HIGH category, mapped to user “ORACLE”
§ CG_NORMAL (30% L1 intra) assigned CAT_MEDIUM category, mapped to user “ORACLE2”
§ CG_OTHER (100% L2 intra) assigned CAT_LOW , mapped to user “ORACLE3”
– VISX:
§ CG_SHIP_OLTP (50% L1 intra) assigned CAT_HIGH category, mapped to user “ORACLE”
§ CG_FIN_OLTP (40% L1 intra) assigned CAT_HIGH category, mapped to user “ORACLE2”
§ CG_FIN_REPT (10% L1 intra) assigned CAT_MEDIUM , mapped to user “ORACLE3”
– VISY:
§ CG_OLTP(100% L1 intra) assigned CAT_LOW category, mapped to user “ORACLE”
§ CG_REPT (100% L2 intra) assigned CAT_LOW category, mapped to user “ORACLE2”
§ CG_OTHER (100% L3 intra) assigned CAT_LOW category, mapped to user “ORACLE3”
CONFIDENTIAL | 78
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

79. IORM in Action: Test 3 Planning
Category IORM Plan
VISX DWPRD DWPROD VISY
CG_SHIP_OLTP: 50% L1 CG_CRITICAL: 80% L1 CG_CRITICAL: 70% L1 CG_OLTP: 100% L1
CG_FIN_OLTP: 40% L1 CG_NORMAL: 20% L1 CG_NORMAL: 30% L1 CG_REPT: 100% L2
CG_FIN_REPT: 10% L1 CG_OTHER: 100% L2 CG_OTHER: 100% L2 CG_OTHER: 100% L3
CG_OTHER: 100% L2
CATEGORY: HIGH_PRIORITY (70% L1)
CATEGORY: MEDIUM_PRIORITY (20% L1)
CATEGORY: LOW_PRIORITY (80% L2)
CONFIDENTIAL | 79
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

80. IORM in Action: Test 3 Resource Allocations
Category IORM Plan
Adjusted allocation percentages
CONFIDENTIAL | 80
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

81. IORM in Action: Test 3 Implement
Category IORM Plan
Implementation steps:
§ Creating consumer groups in each database to match scenario and
assign consumer groups to categories
§ Map sessions to consumer groups and activate plan
§ Enable IORM Category plan
CONFIDENTIAL | 81
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

82. IORM in Action: Test 3 Implement
Category IORM Plan (DWPROD Intra-Database plan)
Create categories
Create CG and
assign to category
Map users
Specify CG
allocations
Activate plan
CONFIDENTIAL | 82
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

83. IORM in Action: Test 3 Implement
Category IORM Plan (DWPRD Intra-Database plan)
CONFIDENTIAL | 83
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

84. IORM in Action: Test 3 Implement
Category IORM Plan (VISX Intra-Database plan)
CONFIDENTIAL | 84
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

85. IORM in Action: Test 3 Implement
Category IORM Plan (VISY Intra-Database plan)
CONFIDENTIAL | 85
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

86. IORM in Action: Test 3 Implement
Category IORM Plan
Null out previous DB plan
Category IORM
CONFIDENTIAL | 86
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

87. IORM in Action: Test 3 Validate
Category IORM Plan
CONFIDENTIAL | 87
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

88. IORM in Action: Test 3 Validate
Category IORM Plan
Shows my categories
Consumer group mapping per DB
per Category
cellsrv.statedump summarizes
this well
CONFIDENTIAL | 88
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

89. IORM in Action: Test 3 Load Test
Category IORM Plan
IO wait pattern:
CAT_HIGH < CAT_MEDIUM < CAT_LOW
Consumer group wait data similar during run
CONFIDENTIAL | 89
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

90. IORM in Action: Test 3 Analyze
Category IORM Plan
IOsaturationtoolkit-v2 Results
VISX had 2 peak MB/sec throughput due to two
Relatively smooth distribution because we had a
CAT_HIGH mappings
mix of category mappings between databases
(and corresponding “elapsed” time valleys)
CONFIDENTIAL | 90
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

91. IORM in Action: Test 3 Analyze
Category IORM Plan
CELLIORM.IORM_METRICS data
Let’s look at category
metrics first
CONFIDENTIAL | 91
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

92. IORM in Action: Test 3 Analyze
Category IORM Plan
CELLIORM.IORM_METRICS data
CONFIDENTIAL | 92
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

93. IORM in Action: Test 3 Analyze
Category IORM Plan
CELLIORM.IORM_METRICS data
CONFIDENTIAL | 93
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.

94. IORM in Action: Test 3 Analyze
Category IORM Plan
CELLIORM.IORM_METRICS data
CONFIDENTIAL | 94
2012 Centroid Systems, Inc. All rights reserved. Reproduction of this document or any portion thereof without prior written consent is prohibited.