Mike Gangler presented on how Secure-24 leverages Oracle ZFS Storage, Oracle Intelligent Storage Protocol (OISP), and Oracle 12C RAC to implement a private cloud for Oracle 12C RAC applications. They paired Oracle databases with ZFS storage through Oracle's VCA, which enables automatic and dynamic tuning of storage settings through OISP. This simplifies management and improves performance by optimizing record sizes based on database queries.

1. Session ID:
Prepared by:
Optimize and Simplify Oracle 12C RAC
using dNFS, ZFS and OISP
1197
@jmjgangler
Mike Gangler – Senior Database Specialist, DBA Team Lead
Secure-24 - @mjgangler
Mjgangler@yahoo.com
Palm D
Thu, Apr 14, 2016
08:30 AM - 09:30 AM

2. About Mike Gangler
• Oracle ACE with robust database credentials
• DBA for over 28 years, working with Oracle since
version 4
• Team Lead and Senior Database Specialist at Secure-24
• Currently serving on the board of the Southeast Michigan
Oracle Professionals (SEMOP) group – www.meetup.com
• Charter member of the Board of Directors for the
International Oracle Users Group (IOUG) – www.ioug.org
• Follow me on my Blog http://mjgangler.wordpress.com and
on twitter! @mjgangler
2

3. About Secure-24
3
FOUNDED
HEADQUARTERS
GLOBAL
OPERATION
CENTERS
DATA CENTERS
Secure-24 was
founded in 2001 and
since then has grown
to 500+ employees
and has received
recogniPon as one of
Computerworld’s Best
Places to Work in IT,
3-years running.
Secure-24 is
headquartered in
Southfield, MI
Serving customers
around the globe,
Secure-24 has two (2)
OperaPon Centers in
Michigan, one (1) in
Nevada and one (1) in
Hyderabad India.
Secure-24 has three
(3) data centers in
Michigan, one (1) in
Nevada, plus several
global partnerships.
We only choose the
safest locaPons for
our data centers.
Secure-24 has 15 years of experience delivering managed IT operaPons, applicaPon hosPng and cloud services to
enterprises worldwide. We manage SAP, Hyperion, PeopleSo^, JD Edwards, Oracle E-Business Suite and other
mission criPcal applicaPons across all industries for businesses of every size.

4. Communi'es Educa'on
Join for as low as $150
SELECT Journal Resource Center IOUG Press Webinars & Podcasts IOUG Forum 5 Minute Briefing
Plus get access to IOUG’s content library, peer-to-peer networking, and more!
Corporate options also available!

5. Oracle Conferences in Detroit Area
Southeast Michigan Oracle Professionals
http://www.meetup.com/SouthEast-Michigan-Oracle-Professionals/
Meet monthly – 2nd Tuesday of the month
Michigan Oracle User Summit
November 3, 2016
http://www.mous.us

6. Great Lakes Oracle Conference
• 2016 Great Lakes Oracle Conference (GLOC)
• May 18 & 19, 2016
Cleveland Public Auditorium
Cleveland, OH
https://www.neooug.org/gloc/

7. Today’s Discussion
7
Learn how Secure-24 leverages Oracle ZFS Storage, Oracle Intelligent
Storage Protocol (OISP), Templates and Oracle 12C RAC to implement a
Private Cloud for Oracle 12c RAC applications. This pairing of Database and
Storage hardware through Oracle’s VCA has enabled Secure-24 to take
advantage of the unique integration capabilities such as OISP which enables
automatic and dynamic tuning of storage settings such as record size.
Secure-24 Confidential – Internal – Copyright © 2015 Secure-24 and/or its affiliates. All rights reserved.

8. • Simplicity
– Simplify the environment with new technology
• Scalability
– Technology should be able to grow and scale as the company grows.
• Automation
– Technology should have the ability to be automated and repeatable process
• Overall reduction of cost
– Technology should drive down the overall cost
• Solution driven by Customer requirements and demands
– This is the reason for looking at the technology
Secure-24 Technology Goals

9. Business Case
• Simplify Oracle 12c RAC deployments
• Eliminate complexity around storage tuning
• Decrease support time through transparency and
simplicity
• Fully Oracle Support technology stack

10. 10
Hardware Specifications Overview
Oracle Private Cloud Appliance
Oracle X86 Servers
Oracle ZFS Storage Appliance
• Redundant controllers, high performance virtual storage
Network Infrastructure
• Oracle Fabric Interconnect F1-15 (Virtual Network)
• Oracle Switches
• Fully configured fabric networking, no setup required
• Integrates with exis'ng Ethernet and storage networks
Fabric
Interconnect
Fabric
Interconnect
Compute
Nodes
Compute
Nodes
Mgmt Nodes
Management
Storage
InfiniBand Expansion
Ethernet Switch
Oracle X86 Servers
• Up to 25 nodes
• CPU: Up to 900 cores
• Memory: Up to 6.4TB aggregate
• Supports any Linux, Solaris and Windows applica'ons
NetworkingManagement
40Gb/s Server-to-Server
InfiniBand Expansion
• High speed InfiniBand
network – mul'ple 40
Gbps
• InfiniBand connected
storage
• Dedicated management
systems included
• Oracle VM and Oracle
Linux included
• Tested and Cer'fied by
Oracle
• So^ware Defined
networking through
Fabric Interconnect

11. • Fast “Power-on to Production” - Automation
– 2 hours vs. 1 month on legacy infrastructure
• Mixed workloads - Scalable
– Oracle DB, Oracle Applications, Microsoft, Custom apps
• More efficient Licensing – Reduction of cost
– Trusted Partitions Licensing
• Rapid Application Deployment – Simplicity and
automation
– Pre-built Oracle VM templates
• Simplified Support - Simplicity
– One Hand to Shake (One Throat to Choke)
• Lower Acquisition and Deployment Costs
• Impressive performance – Customer requirements
Why We Selected Private Cloud Appliance

12. Oracle 12C RAC Design Overview
• Oracle PCA and ZS3-2 Storage
• Single Node or multiple nodes with RAC
• Oracle 12c Enterprise Edition
• Oracle ZFS storage array with OS 8.3+
• Database connections made from Oracle
using Direct NFS
• ZFS Storage – Flash enabled performance
storage
• OISP (Oracle Intelligent Storage Protocol)
for storage optimization
• No “ASM”
X3-2
VIRTUALCOMPUTEAPPLIANCE
S P
REAR
PS
MAJ OR
MINOR
US ER
C RITIC AL
ZFS S TORAGE
ZS 3-2
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S TORAGE
DE2-24P
HDD
MAP
12-23
0-11
S P
REAR
PS
MAJ OR
MINOR
US ER
C RITIC AL
ZFS S TORAGE
ZS 3-2

13. Design Overview
• Local OS disks Stored
on Oracle VM data
repository
• Data, Logs, and Voting
disks presented via NFS
and dNFS
VM 2
dNFS
VM
NAS, SAN, iSCSI
Oracle VM Servers
Oracle VM Server Pool
VM VMVM
SI
VM 1
RAC RAC
ZFS Storage

14. ZFS Powered
• OISP MOS Doc ID 1943618.1
• ZFS Admin Guide http://
docs.oracle.com/cd/
E51475_01/pdf/E52872.pdf
• ZFS Hybrid Array with 3 types of caching
• DRAM, SLC Write FLASH,
MLC Read FLASH
• OISP Op'mal Record Size & Latency
• The Oracle dNFS client passes the op'mal
record size to the ZFS Storage Appliance for
each NFSv4 write request
• The Oracle dNFS client passes the op'mal
write bias to the ZFS Storage Appliance for
each NFSv4 write request..

15. ZFS Setup
• Create our 3 shares
• OCR Voting Disk
– This will be raw NFS v3
– https://docs.oracle.com/database/121/CWADD/
votocr.htm#CWADD92368
– We leave Write Bias as Latency and Record Size
as 128K (Generic)
• Data and Logs files
– This is where the magic of OISP comes in.
– Instead of having to create multiple different
volumes for redo, archive, control, data, temp etc.
We simple create 2 volumes for Logs and Data
– Write Bias and Record Size will be set via
database cue’s through OISP and optimized for
the database.

16. STANDARD NFS ARCHITECTURE

19. dNFS and OISP

20. Linux Configuration Setup
Direct NFS
– Fairly simple example
• # cat oranfstab.example
• server: 10.10.10.10
• nfs_version: nfsv4
• export: /export/U02 mount: /u02
• export: /export/U03 mount: /u03
– dNFS is fairly straight forward in 12c
setup guide
• https://docs.oracle.com/database/
121/CWLIN/
storage.htm#CWLIN279
– This is where NFSv4 is setup
– Many other options are
possible, but simple is
possible too
• Old paper, still valid - http://
www.oracle.com/
technetwork/articles/
directnfsclient-11gr1-
twp-129785.pdf
– Most dNFS guides are for 11g,
12c is the same except dNFS
is Natively part of 12c
– Troubleshooting
• (Doc ID 1464567.1)

21. dNFS Linux Configuration Setup
dNFS (11G) – Add 12C dnfs_on
Enable:
Ln –s $ORACLE_HOME/lib/libnfsodm11.so $ORACLE_HOME/lib/libodm11.so
Disable
Ln –s $ORACLE_HOME/lib/libodmd11.so $ORACLE_HOME/lib/libodm11.so

22. dNFS Linux Configuration Setup
dNFS (12C)
Enable:
Change directory to $ORACLE_HOME/lib
>>make –f ins_rdbms.mk dnfs_on
Disable
Change directory to $ORACLE_HOME/lib
>>make –f ins_rdbms.mk dnfs_off

23. dNFS Linux Configuration Setup
dNFS Verify
• bash-3.2$ ps -ef | grep dbw
oracle 23126 1 0 06:53 ? 00:00:00 ora_dbw0_MYINST1
oracle 23128 1 0 06:53 ? 00:00:00 ora_dbw1_MYINST1
oracle 23246 22377 0 07:04 pts/0 00:00:00 grep dbw
• bash-3.2$ /usr/sbin/lsof -p 23126
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
oracle 23126 oracle cwd DIR 104,3 4096 14811513 /u02/app/oracle/product/12.1.0/dbs
oracle 23126 oracle rtd DIR 104,3 4096 2 /
oracle 23126 oracle txt REG 104,3 210824714 15892486 /u02/app/oracle/product/12.1.0/bin/oracle
• ...
oracle 23126 oracle 32u IPv4 322388 TCP mydbhost:64553->mynfshost:nfs (ESTABLISHED)
oracle 23126 oracle 33u IPv4 322389 TCP mydbhost:21549->mynfshost:nfs (ESTABLISHED)
-- dNFS setup guide
• https://docs.oracle.com/database/121/CWLIN/storage.htm#CWLIN279

24. Linux Configuration Setup
Oracle Linux 6 – uek Kernel
– Utilize dtrace in Linux
– Ksplice for non-disruptive kernel patching
– UEK kernel optimization for Oracle DB
Setup Mount Points
– Setup Voting disk
• Must be NFSv3
• Not Part of dNFS
• Mount options important (Doc ID
359515.1)
• rw,bg,hard,nointr,rsize=32768,
wsize=32768,tcp,noac,vers=3,timeo=600,
actimeo=0
– Setup Logs and Data
• Must be NFSv3 (dNFSv4 setup to come
later)
• Mount options important (Doc ID
359515.1)
• rw,bg,hard,nointr,rsize=32768,wsize=3276
8,tcp,actimeo=0,vers=3,timeo=600

25. Linux Configuration Setup - #1
Install Oracle Pre-Install RPM
• Optional but makes life easier
• >> yum install oracle-rdbms-server-12cR1-preinstall
• This will download all required dependencies
• create the oracle user id
• modify sysctl.conf
• modify limits.conf
• https://blogs.oracle.com/wim/entry/easily_install_oracle_rdbms_12cr1

26. Linux Configuration Setup - #2
TEST SNMP – REQUIRED FOR OISP
MOS ID – 1943618.1
# SNMPGET –v1 –c public 10.10.10.10 1.3.6.1.4.1.42.2.225.1.4.2.0
- Looking for the following:
SNMPv2- SMI::enterprises.
42.2.225.1.4.2.0 = STRING: "Sun ZFS Storage 7330"

27. Linux Configuration Setup - #3
Setup for dNFS
Oracle wants us to tune for dNFS, so we will
>> vi /etc/sysctl.conf
##7.3.4 Checking TCP Network Protocol Buffer for Direct
NFS Client
• net.ipv4.tcp_rmem = 4096 87380 4194304
• net.ipv4.tcp_wmem = 4096 16384 4194304
• https://docs.oracle.com/database/121/CWLIN/
storage.htm#CHDEFHFC

28. Linux Configuration Setup
Install Oracle Pre-Install
RPM
– Optional but makes
life easier
– # yum install oracle-
rdbms-server-12cR1-
preinstall
– This will download all
required
dependencies, create
the oracle user id,
modify sysctl.conf an
d modify limits.conf.
– https://
blogs.oracle.com/wim/
entry/
easily_install_oracle_r
dbms_12cr1
Test snmp (needed for
OISP)
– Doc ID 1943618.1
– # snmpget -v1 -c
public 10.10.10.10
1.3.6.1.4.1.42.2.225.1
.4.2.0
SNMPv2-
SMI::enterprises.
42.2.225.1.4.2.0 =
STRING: "Sun ZFS
Storage 7330"
Setup for dNFS
– Oracle wants us to
tune for dNFS, so we
will
• # vi /etc/sysctl.conf
• #
• #
https://docs.oracle.com/
database/121/CWLIN/
storage.htm#CHDEFH
FC
• #7.3.4 Checking TCP
Network Protocol Buffer
for Direct NFS Client
• #
• net.ipv4.tcp_rmem =
4096 87380
4194304
• net.ipv4.tcp_wmem =
4096 16384
4194304

29. Oracle 12c Enterprise Edition
Install Oracle 12.1.0.2
- If using a private heartbeat IP that contains .255
- add the following patch (Before root.sh)
- 19777496 (Patch Id)
- Or dont use .255 in the heartbeat network
Oracle 12c Installation - #1

30. Confirm dNFS and OISP setup
$ egrep I '"Sun ZFS"|Direct' alert_*.log
Found NAS server Sun ZFS Storage 7420
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0
Direct NFS: channel id [0] path [10.80.216.18] to filer [10.80.216.18] via local [] is UP
Direct NFS: channel id [1] path [10.80.216.19] to filer [10.80.216.18] via local [] is UP
Direct NFS: mount complete dir /export/oisp/redo on 10.80.216.18 nfsport 2049 using NFS
version 4
Direct NFS: mount complete dir /export/oisp/redo on 10.80.216.18 nfsport 2049 using NFS
version 4
u “Found NAS server” indicates SNMP has found the ZFSSA. Requirement
for OISP.
u ODM/Direct NFS indicates that dNFS has been activated using NFSv4.
Oracle 12c Installation - #2

31. Apply required patch (OISP)
Patch 19630940
Resolves issue with DB Analy'cs
Oracle 12c Installation - #3

32. Oracle 12c Enterprise Edi'on
Install Oracle 12.1.0.2
Confirm dNFS and OISP setup
$ egrep I '"Sun ZFS"|Direct' alert_*.log
Found NAS server Sun ZFS Storage 7420
Oracle instance running with ODM: Oracle Direct NFS ODM
Library Version 3.0
Direct NFS: channel id [0] path [10.80.216.18] to filer
[10.80.216.18] via local [] is UP
Direct NFS: channel id [1] path [10.80.216.19] to filer
[10.80.216.18] via local [] is UP
Direct NFS: mount complete dir /export/oisp/redo on
10.80.216.18 nfsport 2049 using NFS version 4
Direct NFS: mount complete dir /export/oisp/redo on
10.80.216.18 nfsport 2049 using NFS version 4
“Found NAS server” indicates SNMP has
found the ZFSSA. Requirement for OISP.
ODM/Direct NFS indicates that dNFS has
been ac'vated using NFSv4.
Apply required patch
Patch 19630940
Resolves issue with DB
Analy'cs
Oracle 12c Installation

33. Typical Deployment
Typical Process For Creating and Deploying New Applications
1 to 3 weeks
1-2 days 1 day 1-5 days 1-2 days 1-5 days Start

34. PCA Rapid Deployment
• Pre-made, certified virtual machine templates, including application, operating system,
and database available at https://edelivery.oracle.com
• Save Weeks/Days in Configuration Time
– Pre-built, pre-configured Templates
– Application, Middleware, and DB
– Oracle DB, JD Edwards, EBS, PeopleSoft, and many more…. 90+ templates
Start-up in
Oracle VM Pool
Import to
Oracle VM
Manager
Oracle VM
Servers
Server
Pool 2 Customize & Save
as Golden Image
E-Delivery
File
JD
Edwards
Download from
Oracle.com
File
JDE
VM 1
JDE

35. Application Templates & Assemblies
DB
WebLogic
Server
Deployment Policies
Configura'on Metadata
SOA_PROD_1
Oracle VM Assembly
Produc'on Environment: SOA_PROD_1
VM
VM
VM
VM
VM
VM
Deploy
Deploy applica'on into Oracle VM Manager as
‘Golden Image’
• Clone, ini'alize, and start-up instance
• Up and running in hours vs. days or weeks with
litle risk
Oracle VM
Servers
Storage Repository
Server
Pool 2
Deployed:
Produc'on Environment SOA_PROD_1
Guest Addi'ons
• Allows direct integra'on between guest
so^ware and virtualiza'on layer
• Assists in orchestra'on of complex, mul'-
VM deployments
OVF-based
3

36. Notes
• Oracle Database 11g RAC with ZFS Storage Appliance and dNFS on SPARC:
http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/
oracledb-rac-sparc-zfssa-2414324.pdf
• Optimizing Storage for Oracle Database 11gR2 with ZFS Storage Appliance:
http://www.oracle.com/technetwork/server-storage/sun-unified-storage/documentation/
oracle11gr2-zfssa-bestprac-2255303.pdf
• How to enable OISP on ZFS Storage Appliance for 12c:
https://support.oracle.com/epmos/faces/DocumentDisplay?id=1943618.1
ANALYST DATA
• htp://www.oracle.com/us/corporate/analystreports/industries/zs3-storage-appliance-2033590.pdf

37. Questions
JDE
Mike Gangler
Michael.gangler@secure-24.com
Blog : htps://
mjgangler.wordpress.com
@mjgangler

38. Please complete the session evaluation
Paper – 1197
Author – Mike Gangler
We appreciate your feedback and Insight
You May complete the session evaluation via the mobile app