1. Managing Exadata in the Real
World
Arup Nanda

2. What is this About?
• How Exadata is different?
• Who manages it?
• Who does what?
• What you need to know about managing it
• What must you learn about potential issues
2

3. What is Exadata
• Looks like an appliance
• But is not an appliance. Why?
– It contains additional software to make it a better
database machine
– The components are individually administered
• That’s why Oracle calls it a Database Machine
(DBM)
• DMA – Database Machine Administrator
3

4. iDB
CPU iDB
Memory
Network
I/O Controller
Disk
4

5. Storage Cell Server
iDB • Cells are Sun Blades
• Run Oracle Enterprise
Linux
• Software called Exadata
Storage Server (ESS)
which understands iDB
Disk1 Disk2 Disk3
5

6. Storage Indexes
Storage Indexes store in memory
of the Cell Server the areas on
the disk and the MIN/MAX value
of the column and whether NULL Disk1 Disk2 Disk3
exists. They eliminate disk I/O.
SELECT … MIN = 3 MIN = 4 MIN = 1 MIN = 3
FROM TABLE MAX = 5 MAX = 5 MAX = 2 MAX = 5
WHERE COL1 = 1
Storage Index Disk4
6

7. Put Together: One Full Rack
RAC Cluster
Database Database
Node 1 Node 8
Database
Node 7 Clients
InfiniBand Network Switch connect to the
Switch database
nodes.
Cell 1 Cell 14
Cell 1
Cell 1
Cell 1
7

8. Disk Layout • Disks (hard and flash) are
connected to the cells.
• The disks are partitioned at the
Compute
Nodes cell
• Some partitions are presented
as filesystems
Storage
• The rest are used for ASM
Cell diskgroups
• All these disks/partitions are
presented to the compute nodes
8

9. Disk Presentation
Node
filesystem
filesystem
Cell
9

10. Disk Failures
Datafile
block1
Cell 1 Cell 2
block1 block1
10

11. One Cluster?
One Cluster
QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1
QA1 QA2 Prod4 Prod1 Prod2 Prod3 Dev1 Int1
QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1
Dev2
11

12. Many Clusters?
QA Cluster Prod Cluster
Dev Int
QA1 QA2 QA3 Prod1 Prod2 Prod3 Dev1 Int1
QA Cluster Prod Cluster
Dev Int
12

13. Playing Nice
• Database Resource Manager
• I/O Resource Manager
• Cell Fencing
Compute
QA Prod Nodes
Storage Cells
13

14. Component Management
Linux Commands – vmstat, mpstat, fdisk, etc.
Compute
Nodes
ASM Commands – SQL*Plus, ASMCMD, ASMCA
Database Commands – startup, alter database, etc.
Clusterware Commands – CRSCTL, SRVCTL, etc.
Storage Cell
Linux Commands – vmstat, mpstat, fdisk, etc.
CellCLI – command line tool to manage the Cell
5-part Linux Commands article series
http://bit.ly/k4mKQS
4-part Exadata Command Reference article series
http://bit.ly/lljFl0
14

15. Server Management
• Sun Blades and Oracle Enterprise Linux
• Normal Sysadmin Work
– Shutdown, fdisk, etc.
• ILOM – Integrated Lights Out Management
• KVM allows physical access
– But you can use ILOM for virtual console
• Needs Pure Linux Skills
15

16. Database, Cluster Management
• Cluster
– crsctl, srvctl
• ASM
– asmcmd, SQL*Plus
• Database
– srvctl, SQL*Plus
16

17. Storage Management
• Two ways to manage the
storage
– Traditional Storage
Administrators not quite iDB
helpful here
– Enterprise Manager
– CellCLI
Disk1 Disk2 Disk3
17

18. Network Management
• Two types of network
– Ethernet
– Infiniband
• Tools
– ibstatus
– iblinkinfo
– verify-topology
18

19. Oracle Provided Tools
• All tools are found at
/opt/oracle.SupportTools
• CheckHWnFWProfile
– to check the HW profile
• Directory ibdiagtools
/opt/MegaRAID/MegaCli/MegaCli64
# ipmitool -H prolcel01-ilom -U root chassis
power on
# imageinfo
# imagehistory
19

20. Skills for Administration
Skill Needed
System Administrator 10%
Storage Administrator 0%
Network Administrator 5%
Database Administrator 60%
Cell Administration 25%
DBA
Sys Admin
Network Admin
Cell Admin
20

21. Divide and Conquer
Database DBA
Machine System Admin
Network Network Admin
Storage ?? DBA
Sys Admin
Network Admin
Cell Admin
21

22. Combined Skills
Database Database
Machine
Administrator
Machine (DMA)
New Role
Network
Storage DBA
Sys Admin
Network Admin
Cell Admin
22

23. Resources
• My Papers
– 5-part Linux Commands article series
http://bit.ly/k4mKQS
– 4-part Exadata Command Reference article series
http://bit.ly/lljFl0
• OTN Page on Exadata
– http://www.oracle.com/technetwork/database/exadata/in
dex.html
• Tutorials
– http://www.oracle.com/technetwork/tutorials/index.html
23

24. Pros and Cons of DMA
• Pros
– Natural progression from DBA skills
– Very little non-DBA activities
– Patching becomes simpler
– Performance diagnosis becomes easier
• Cons
– Skills
– Security?
• Appliance model comes in here
24

25. Backup and DR
• No SAN connectivity
• Only NAS Exadata
– Infiniband
– Tape , Disk Pool
• DR Infiniband
– No Storage Level Replication
– Only Data Guard Backup
– Supplemental Logging Device
– Force Logging
– http://www.oracle.com/technetwork/database/features/availa
bility/maa-wp-dr-dbm-130065.pdf
• Golden Gate
25

26. ETL and Reporting OLTP
Exadata
Golden
Gate
Infiniband
Informatica MicroStrategy
Exalytics
26

27. Summary
• Exadata is an Oracle Database on RAC with a
specialized storage
• A single role helps in administration: Database
Machine Administrator (DMA)
• DBA -> DMA is the natural progression
• Use many clusters in the same Exadata frame
• Use IB for ETL, Reporting and Backups
• Data Guard only physical DR solution
27

28. Thank You!
28