1. Exadata Consolidation Success Story
Getting the kids to play nice with each other…
1
Presented
by:
Karl
Arao

2. whoami
Karl
Arao
• Senior
Technical
Consultant
@
Enkitec
• Performance
and
Capacity
Planning
Enthusiast
7
years
DBA
experience
Oracle
ACE,
OCP-­‐DBA,
RHCE,
OakTable
Blog:
karlarao.wordpress.com
Wiki:
karlarao.Kddlyspot.com
TwiLer:
@karlarao
www.enkitec.com
2

3. www.enkitec.com
3
100+
3

4. Agenda
• Architecture
• Tools
and
Methodology
• War
Stories
www.enkitec.com
4

5. General
Architecture
www.enkitec.com
5
Primary
Site
Standby
Site
ProducKon
Test
&
Dev
Disaster
Recovery
Future
Growth

6. General
Architecture
www.enkitec.com
6

7. The
Stats
Three
Half
Rack
Exadata
clusters
with
High
Cap.
drives
Cluster
#1
36
Dev/Test
Databases
Cluster
#2
11
ProducKon
Databases
Cluster
#3
13
Dev/Test
Databases
6
Standby
Databases
SKll
more
databases
to
come…
www.enkitec.com
7

8. Why
Consolidate?
Primary
drivers
for
consolidaKon
center
around
cost
savings
• Reduces
Oracle
soware
licensing
• 3rd
party
products
such
as
backup
agents,
ETL
tools,
etc…
• More
efficient
use
of
system
resources
• So
Costs
– Floor
space
– Power
&
Cooling
– AdministraKon,
Staffing
Costs
(training,
etc.)
www.enkitec.com
8

9. www.enkitec.com
9
7
Databases
A
Simple
ConsolidaKon
Example

10. www.enkitec.com
10
For
example,
the
first
row
should
read…
Database
‘A’
requires
4
CPU’s
and
will
run
on
nodes
1
and
2
(2
CPU’s
each)
Let’s say we have the following databases to migrate on Exadata:
Cluster
Level
UKlizaKon
A
Simple
ConsolidaKon
Example

11. www.enkitec.com
11
Let’s say we have the following databases to migrate on Exadata:
Per
compute
node
UKlizaKon
For
example,
the
first
row
should
read…
Database
‘A’
requires
4
CPU’s
and
will
run
on
nodes
1
and
2
(2
CPU’s
each)
A
Simple
ConsolidaKon
Example

12. www.enkitec.com
12
Cluster
Level
UKlizaKon
=
29.2%
Per
compute
node
UKlizaKon
25%
42%
33%
17%
A
Simple
ConsolidaKon
Example

13. www.enkitec.com
13
Cluster
Level
UKlizaKon
=
29.2%
Per
compute
node
UKlizaKon
8%
83%
17%
8%
A
Simple
ConsolidaKon
Example

14. www.enkitec.com
14
• Gather
UKlizaKon
Metrics
(usage
history)
• Create
Provisioning
Plan
• Implement
Plan
• Audit
Your
ImplementaKon
Tools
And
Methodology

15. www.enkitec.com
15

16. Provisioning
Worksheet
• Capacity
Planning
• CommunicaKon
Tool
• Hand
off
www.enkitec.com
16
**Supplement
to
exisKng
Exadata
installaKon
tools:
• Site
planning
checklist
• ConfiguraKon
Worksheet
• Exadata
Configurator
sheet
• CheckIP
• OneCommand
UKlizaKon
=
Requirements
/
Capacity

17. Capacity
www.enkitec.com
17
2
=
quarter
rack
4
=
half
rack
8
=
full
rack
SPECint_rate2006
hLp://goo.gl/doBI5
CPU_COUNT,
threads,
&
cores
hLp://goo.gl/CunHN
96
to
144GB
(frequency
of
the
memory
DIMMs
drops
to
800
MHz
from
1333
MHz)
Space
will
also
depend
on:
• ASM
redundancy
• DATA/RECO
allocaKon
hLp://goo.gl/I3pn
Query
Low
(4x)
Query
High
(6x)
Archive
Low
(7x)
Archive
High
(12x)

18. CPU
Core
Comparison
www.enkitec.com
18
Source
chip
efficiency
factor
=
source
SPEC
raKng
/
Exadata
SPEC
raKng
=
16/26
=
.6154
EXA
cores
requirement
=
source
host
cores
*
uKlizaKon
*
chip
efficiency
factor
=
32
*
.7
*
.6154
=
13.78
*
offload
factor
*
.5
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐
6.89
Sun
Fire
X4170
M2
X5670@2.93GHz
DesKnaKon
how
much
of
the
source
CPU
cores
are
being
used
mulKplier
for
equivalent
database
machine
cores
amount
of
CPU
resources
that
will
be
offloaded
to
the
storage
cells

19. The
Perfect
Storm
(Peopleso=
HR)
www.enkitec.com
19
Month-­‐end
Processing
+
Weekly
Time
Entry
+
SQL
Plan
Change
-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐
Uh-­‐oh!

20. CPU
AllocaKon
www.enkitec.com
20
DB
Uniq
Name
DB
Name
node
1
node
2
node
3
node
4
4 instance 5 instance 4 instance 3 instance
47% cpu used 75% cpu used 47% cpu used 18% cpu used
49% mem used 66% mem used 71% mem used 54% mem used
BIPRDDAL biprd P P
DBFSPRD DBFSPRD P P P P
HCMPRDDAL hcmprd P P
MTAPRD11DAL mtaprd11 P P
PAPRDDAL paprd P P
RMPRDDAL rmprd P P
dbm dbm F F F F
Fsprddal fsprd P P
= Preferred
= Failover

21. www.enkitec.com
21
Load
Map
(our
first
stop…)
Users
Complaint:
HR
Kme
entry
and
OBIEE
reports
painfully
slow…

22. www.enkitec.com
22
Top
AcKvity
-­‐
HCMPRD

23. www.enkitec.com
23
Instance
AcKvity
–
HCMPRD2
HCMPRD
Caged
at
12
CPU’s
SQL
Profile
Installed
to
lock
in
good
plan.
Problem:
A
single
SQL
stmt.
overwhelming
CPU
resources.
Node 2

24. Memory
ExhausKon
(OBIEE)
“1
Report
=
1
SQL
query,
right?”
WRONG!
www.enkitec.com
24

25. www.enkitec.com
25
Overlapping workloads of three databases
across 3 nodes.
BIPRD, HCMPRD, and MTAPRD
Node 1
Node 2
Node 3
Node 4

26. www.enkitec.com
26
Node
Layout
Revisited…

27. www.enkitec.com
27
Notice what happens to CPU waits
and the system load average when
this report is run.

28. www.enkitec.com
28
PGA Memory Spikes

29. www.enkitec.com
29

30. www.enkitec.com
30

31. Storage
Cell
SaturaKon
(OBIEE)
www.enkitec.com
31

32. www.enkitec.com
32

33. www.enkitec.com
33
I/O
Intensive
Workload

34. www.enkitec.com
34
Smart
Scans
as
seen
in
Grid
Control

35. www.enkitec.com
35
25
Sessions
Doing
Smart
Scans
…as
seen
in
gv$sql

36. www.enkitec.com
36

37. www.enkitec.com
37
Smart
Scan
in
AcKon.
The
cells
are
scanning
1T
but
only
returning
144G…
***That’s
on
each
of
the
highlighted
row
source
below…

38. www.enkitec.com
38
The
databases
on
other
nodes
see
the
contenKon
as
“System
I/O”
Without
I/O
resource
management
even
criKcal
processes
are
affected
(CKPT,
LGWR,
…)

39. www.enkitec.com
39
Inter-­‐database
IORM
Plan
(only
kicks
in
when
needed)
I/O
requests
from
criKcal
processes
like
CKPT,
LGWR,
LMON
get
priority
automaKcally.
Without
IORM
I/O
requests
from
these
important
processes
receive
the
same
priority
as
any
other
process.
*Side Benefit (automatic when IORM is enabled)

40. www.enkitec.com
40
IORM
Plan
DefiniKon
(on
each
storage
cell)

41. Wrap
up!
Provisioning
Methodology
&
Tools
– UKlizaKon
metrics
and
requirements
– Provisioning
Spreadsheet
Success
Stories
– CPU
resource
management
– Tuning
and
provisioning
adjustments
– I/O
resource
management
www.enkitec.com
41

42. www.enkitec.com
42
Q
&
A

43. 43
Fastest Growing Companies
in Dallas
Contact
Info…
karl.arao@enkitec.com