1. Luci, Ricci and the RAC
...or Clustering on Centos 5 using
Conga for an Oracle RAC install
Drawn up by Fauz Ghauri (@fauzg) for barcamp

2. Introduction
• Hardware
• Installation
• Custom udev rules for iscsi
• Cluster
• Heartbeat and fencing
• RAC Other things to consider for RAC.
• Acknowledgements

3. Hardware
• Network Cards
– DRAC (Dell Remote Access Card)
– ILO (Integrated Lights Out – HP kit)
– VMware Fence (Built into ESX – maybe server?)
– Etc.
• SAN – Are you using iSCSI?
• VMware – which version of ESX/server do you
run?

4. Installation
• Mainly Defaults used for centos 5 install
• A few extra packages are required outside of
the base install.

5. Installation

6. Installation

7. Installation

8. Installation

9. Installation

10. Installation

11. Installation

12. Installation – Conga Components

13. Installation – Conga Components

14. Installation – Luci

15. Installation – Luci

16. Custom udev rules for iSCSI
• If you’re using iSCSI – you’ll need to make sure
that your drives are mapped.
# iscsiadm -m discovery -t sendtargets –p
<sanIP.mydomain (or maybe ip address)>
192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm1
192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm2
192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm3
192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm4
192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.crs

17. Custom udev rules for iSCSI
• Manually log onto targets
– # iscsiadm -m node -T iqn.2006-
01.com.openfiler:racdb.asm1 -p 192.168.2.195 -l
• Configure auto login
– # iscsiadm -m node -T iqn.2006-
01.com.openfiler:racdb.asm1 -p 192.168.2.195 --
op update -n node.startup -v automatic

18. Custom udev rules for iSCSI
• # (cd /dev/disk/by-path; ls -l *sanname* | awk '{FS="
"; print $9 " " $10 " " $11}')
• Returns:
– ip-192.168.2.195:3260-iscsi-iqn.2006-
01.com.openfiler:racdb.asm1-lun-0 -> ../../sda
– Etc.
– Make a note!
– scsi_id –g –u –s /block/sdX – find out which letters map to
which iSCSI device (where X is a, b, c, etc)
– Note iSCSI strings.
– Create a new rules file in /etc/udev/rules.d/22-
randomnumberandname.rules

19. Custom udev rules for iSCSI
• ACTION=="add", KERNEL=="sd*",
PROGRAM=="/sbin/scsi_id -g -u -s %p",
RESULT==“<iscsi string goes here>",
RUN+="/bin/raw /dev/raw/raw1 %N“
• ACTION=="add", KERNEL=="raw1",
OWNER=="root", GROUP=="dba",
MODE=="0660", SYMLINK+="oracle_ocr"

20. Cluster

21. Cluster

22. Cluster

23. Cluster

24. Cluster

25. Cluster

26. Cluster

27. Cluster
This turns
yellow as
there’s an
issue.

28. Cluster

29. Cluster

30. Cluster

31. Cluster

32. Heartbeat & Fencing
• Heartbeat – saying “hi! I’m here” to the cluster.
• Fencing – fencing off a dodgy node
• Fencing on ESX means the creation of a user for
fencing.
– useradd and groupadd – insufficient permissions.
– Get it working on ESX before trying to fence remotely.
– Adding permissions can be done by doing the
following:

33. Heartbeat & Fencing
– vmware-vim-cmd vimsvc/auth/role_add groupX
VirtualMachine.Interact.PowerOn
– vmware-vim-cmd
vimsvc/auth/entity_permission_add
vim.Folder:ha-folder-root userX users groupX
true
• Test it out with the command fence_vmware
(syntax in wiki in ackowledgements section)

34. Things to consider with RAC
• Don’t reboot all nodes at the same time –
unless you feel like losing your entire
database!
• Bear in mind that the /udev rules should map
to some raw disks if you’re using ASM.
• SSH key equivalency (sharing keys).

35. Acknowledgements
• Thanks to:
– [redacted] for getting me going
– [redacted] for helping me figure out that nasty little issue
with GSSAPI timing out my SSH connections
• Sources:
– http://www.oracle.com/technology/pub/articles/hunter_r
ac10gr2_iscsi.html
– http://www.redhat.com/docs/en-
US/Red_Hat_Enterprise_Linux/4.8/html/Cluster_Administr
ation/s1-start-luci-ricci-conga-CA.html
– http://sources.redhat.com/cluster/wiki/VMware_FencingC
onfig
– http://www.vroem.co.za/?p=7

36. Bonus Slides!

37. Bonus Slides!

38. Bonus Slides!

39. Bonus Slides!

40. Bonus Slides!