My First Hands on Oracle 11gR2 Implementation.

1. - Santosh Kangane
Santosh Kangane – Oracle DBA
santoshkangane.blogspot.in
www.linkedin.com/pub/santosh-kangane/1b/78b/6b
© 2012. Santosh Kangane

2. Highlights
 System Requirement
 Oracle 11gR2 RAC overview
 Storage Option
 SAN
 NAS
 SCAN
 OCR, Voting Disk and Time Synchronisation
 Oracle ASM
 Oracle Enterprise manager for RAC

3. System Requirement
Oracle Certified Operating Systems for RAC :
 Linux :
 Red Hat Enterprise Linux : 4.7, 5.2, 5.5, 6
 SUSE Linux Enterprise Server : 10 SP2, 11
 Oracle Enterprise Linux : All versions
 Oracle Solaris ( Most popular option for production systems )
 Windows
* All Node must have same OS and Kernel Version
Memory Requirement for Linux platform:
RAM Swap Space Required
Between 1 GB and 2 GB 1.5 times the size of the RAM
Between 2 GB and 16 GB Equal to the size of the RAM
More than 16 GB 16 GB
Shared Memory : Minimum 40% of RAM; Should be grater than size of SGA

4. Oracle 11g Real Application Cluster

5. Oracle 11g RAC – PEG Lab setup
RAC Node 1 (PTS0009)
RHEL 5.0
eth0 (public IP)
eth1 (private IP) 10.88.33.23
10.88.32.102 PTS0009-vip
10.88.33.28
LAN
eth0 (public IP)
eth1 (private IP) 10.88.33.22
Storage Server 10.88.32.101 PTS0006-vip
10.88.33.27 DNS
(PTS0012)
ORARAC-SCAN
10.88.33.24
10.88.33.25
10.88.33.26
3 IP’s with round robin allocation mode
RAC Node 2 (PTS0006)
RHEL 5.0
© 2012. Santosh Kangane

6. Oracle 11g Real Application Cluster Stack
3
7
6
5
4
2
1
* Number shows the startup sequence

7. Storage Options

8. NAS and SAN

9. IP – Based Storage Area Network Configuration
Standard configuration options
- Tool provided by Storage vendors
- OpenFiler Open source DOS
- Manual Setup using iSCSI protocol
Target (Ubuntu 12.04 on PTS0012)
 Install iscsitarget, iscsitarget-source, iscsitarget-dkms package
 Format the disk and create the partitions as per your requirements. ( I have 10 GB for
OCR &Voting Disk; 61 GB for oracle storage).
 edit /etc/default/iscsitarget. Change the default value of ISCSITARGET_ENABLE=false
over to ISCSITARGET_ENABLE=true.
 Define LUNs in /etc/iet/ietd.conf
Target iqn.2012-08.in.co.persistent:storage.disk0. pts0012
LUN 0 Path=/dev/cciss/c0d1p5,Type=fileio,ScsiId=lun0,ScsiSN=lun0
Node :
― 0th LUN is mandatory.
― Make sure there is no space in LUN Definition above and every parameter is
separated by “,”
 Restart the service iscsitarget

10. IP – Based Storage Area Network
Initiator ( All RAC node RHEL5)
 Install iscsi-initiator-util package.
(Make sure that you have iscsi version >= iscsi-initiator-utils-6.2.0.868-0.7.el5)
 Edit /etc/iscsi/iscsid.conf and set node.startup = automatic
 Then restart the initiator: /etc/init.d/iscsi restart
 Make sure iscsid service is configured to start at system boot up
$ chkconfig --list iscsid
iscsid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
 Using CHAP authentication for security : Edit /etc/iscsi/iscsid.conf
Enable following entries :
node.session.auth.authmethod = CHAP
node.session.auth.username = <<sanusername>>
node.session.auth.password = <<sanpassword>>
 Discover the target using command
$ iscsiadm –m discovery –p <<Target IP>>:3260
 View detected nodes
$ iscsiadm –m node

11. IP – Based Storage Area Network
Initiator ( All RAC node RHEL5) …cont
 Login to the LUNs ( use same sequence to login on all nodes so that the attached
devices names will be identical )
$iscsiadm -m node --targetname “<<inq name of first LUN>>" --portal
“<< Target IP >>:3260" --login
 Verify the sessions
$ iscsiadm -m session -P3 | less
 Now in $fdisk –l all newly attached iscsi drives will be shown ( make sure every
LUN of storage has got same Device name on all RAC nodes)
 Format newly added drives from any ONE node and use $partprobe to push
partition table on all other nodes
 Verify on all nodes if newly attached drives are showing valid portions using
fdisk -l
For more details on Storage Initiator setup and RAC Ethernet tuning refer :
(http://www.scribd.com/doc/46183679/Red-Hat-Linux-v5-x-Software-iSCSI-Initiator-
Configuration-MPIO-and-Tuning-Guide#outer_page_5)

12. Oracle RAC 11gR2 – Single Client Access Name (SCAN)
How do I connect to a cluster ?
How to configure Load balancing?
How to handle Node failure?
Who to isolate node addition/removal without affecting client
connections and downtime ?
 Single client access name (SCAN) is the virtual hostname (of 1-15 Char) to provide for all
clients connecting to the cluster and It should be unique across the network domain.
 SCAN is registered with DNS (or Grid Naming Service) with at least one and up to three
IP addresses from the same subnet as that of Public and VIP address.
 These IP addresses should be allocated in round robin fashion

13. Oracle RAC 11gR2 – Scan Single Client Access Name (SCAN)
1. On each Node there is pair of SCAN
Listener & Local Listener
4 3 SCAN IP address will be mapped as:
3 • 3 Node : Each node will be mapped
PEGLAB1 with one SCAN IP
• More than 3 node : only 3 nodes will
be mapped to SCAN IPs
• 2 Node : One node will have double
2 SCAN IP & SCAN Listener and other
node will have single IP and SCAN
5 Listener
2. PMON process of each instance report
SQL> conn PEGLAB2 the current work load to SCAN listener
user/pwd@orarac-
Service (specified in
scan/peglab
the REMOTE_LISTENER database
parameter.)
6
3. On connect request DNS resolves the orarac-scan and returns list of 3 SCAN IP addresses
4. Client select the first IP from the list and connect to the RAC; if it fails then request again with next IP.
5. Depending on the Load SCAN listener re-direct the request to Local listener of lightly loaded node. ( Note here in case
of More than 3 node cluster, this is how request can still be given to lightly loaded node via SCAN listener service, even
if it does not have SCAN IP mapped )
6. Now all further communication happens through local listener directly.

14. OCR And Voting Disk
Oracle Cluster Registry:
The OCR maintains cluster configuration information that is used by each node
of the cluster to determine the state of the cluster.
OCR also maintains information about cluster resources:
• Databases
• Instance
• Services
Each node in the cluster maintains a copy of the OCR in memory for better performance
and also responsible for updating the OCR as required in shared storage.
Voting Disk:
The Voting Disk Files are used by Oracle Clusterware to determine which nodes
are currently members of the cluster. The voting disk files are also used in concert with
other Cluster components such as CRS to maintain the clusters integrity.

15. Auto Segment Management : Architecture
PEGLAB1 RDBMS Instances PEGLAB2
+ASM1 ASM +ASM2
Instances
CLUSTERWARE Clusterware CLUSTERWARE
PEGLAB
Database Files

16. ASM Disk Groups, Disks and Database Files
Disk Disk Disk
Group 1 Group 2 Group 3
File 3 File 2
File
1
File 4 File 5
File 5 File 6
Disk 3
Disk 1 Disk 4
Disk 2 Disk 5 Disk 6 Disk 7

17. ASM Failure Groups - External Redundancy
Disk Group
Disk 1 Disk 2 Disk 3

18. ASM Failure Groups – Normal Redundancy
Disk Group
Failure Group 1 Failure Group 2
Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6

19. ASM Failure Groups – High Redundancy
Disk Group
Failure Group 1 Failure Group 2 Failure Group 3
Disk 1 Disk 2 Disk 1 Disk 2 Disk 1 Disk 2

20. Extent Distribution – External Redundancy – 1 Disk
Metadata
0
1
2
3
4
5
6
7
Disk 0
Metadata Data

21. Extent Distribution – External Redundancy – 2 Disk
1
3
5
Metadata 7
9
11
13
0 15
2
4
6
8
10
12
14
Disk 0 Disk 1
Metadata Data

22. Extent Distribution – Normal Redundancy – 2 Disk
Failure Group 1 Failure Group 2
0 1 0 1
3 2 3 2
4 5 4 5
7 6 7 6
0 1 0 1
3 2 3 2
4 5 4 5
7 6 7 6
8 9 8 9
11 10 11 10
12 13 12 13
15 14 15 14
16 17 16 17
19 18 19 18
20 21 20 21
23 22 23 22
Disk 0 Disk 1 Disk 2 Disk 3
Metadata Primary Secondary
Data Primary Secondary

23. Extent Distribution – High Redundancy – 1 Disk
Failure Group 1 Failure Group 2 Failure Group 3
0 0 0
1 1 1
2 2 2
3 3 3
4 4 4
5 5 5
0 0 0
1 1 1
2 2 2
3 3 3
4 4 4
5 5 5
6 6 6
7 7 7
8 8 8
9 9 9
Disk 0 Disk 1 Disk 2
Metadata Primary Secondary Tertiary
Data Primary Secondary Tertiary

24. RAC Implementation Step By Step
Document Explains the step by steps executions and
seen dumps for
- Pre-requisite Linux packages installation
- Oracle 11gR2 Clusterware installation
- Oracle 11gR2 Database software installation
- Creation of ASM Disk group
- Creation of Oracle 11g Database on Cluster

25. References
1. http://docs.oracle.com/cd/E11882_01/install.112/e24326/toc.htm
2. http://www.oracle.com/technetwork/products/clustering/overview/scan-
129069.pdf
3. https://support.oracle.com
4. http://www.scribd.com/doc/46183679/Red-Hat-Linux-v5-x-Software-iSCSI-
Initiator-Configuration-MPIO-and-Tuning-Guide#outer_page_5
5. http://www.oracledba.org/11gR2/Pre_Install_11gR2.htm
6. http://www.toadworld.com/KNOWLEDGE/KnowledgeXpertforOracle/tabid/6
48/TopicID/RACR2ARC6/Default.aspx
7. http://juliandyke.com/Presentations/Presentations.html
25

26. santoshkangane.blogspot.com