How to bake a Customer Story with With Windows, NVM-e, Data Guard, ACFS Snapshots & Perl

How to Bake a Customer Story
With Windows, NVM-e, Data Guard,
ACFS Snapshots & Perl
Ludovico Caldara
@ludodba ukoug_tech17

About Ludovico Caldara
DB Cloning : Customer Story2 12/2/2017
■ 18 Years DBA (Not Only Oracle)
▪ I do it everywhere (even Windows)
■ RAC ATTACK Ninja & co-writer
■ President, SOUG & ITOUG Board
■ OCP (11g, 12c, MySQL) & OCE
■ Italian living in Switzerland
■ http://www.ludovicocaldara.net
■ @ludodba ludodba
■ ludovicocaldara

Our company.
Trivadis is a market leader in IT consulting, system integration, solution engineering
and the provision of IT services focusing on and
technologies in Switzerland, Germany, Austria and Denmark.
We offer our services in the following strategic business fields:
Trivadis Services takes over the interacting operation of your IT systems.
O P E R A T I O N

COPENHAGEN
MUNICH
LAUSANNE
BERN
ZURICH
BRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than
600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 million
Financially self-supporting and
sustainably profitable
Experience from more than 1,900
projects per year at over 800
customers

Customer requirements

Customer "Original Requirements"
Protect production database(s) in case of disaster
Refresh test databases on demand
Database size at that time: 50GB

Previous solution
DB_NAME
PRIMARY
SERVER B @ SITE B
DB_NAME
STANDBY
DATA
GUARD
DB_NAME
CLONE
SERVER A @ SITE A
RMAN DUPLICATE
Storage Area Network Storage Area Network

Previous solution
DB_NAME
PRIMARY
SERVER B @ SITE B
DB_NAME
STANDBY
DATA
GUARD
DB_NAME
CLONE
SERVER A @ SITE A
RMAN DUPLICATE
Storage Area Network Storage Area Network
25 minutes
cloning
process

Requirements 4 years later...
Database size now: 300GB

Requirements 4 years later...
Database size now: 300GB
2 hours
cloning
process!

Customer needs a new solution!
New hardware, not related to cloning performance
– Eventually not tied to Storage Area Network
– (Eventually NVM-e)
A magic for faster clones
– Not willing to spend for additional technologies (Delphix, Actifio...)

The new solution

Ingredients
Oracle Data Guard

Ingredients
Oracle Data Guard
NVM-e

Ingredients
Oracle Data Guard
NVM-e
ACFS Snapshots

Ingredients
Oracle Data Guard
NVM-e
ACFS Snapshots
Bash Scripts

Ingredients
Oracle Data Guard
NVM-e
ACFS Snapshots
Bash Scripts
Linux

Ingredients
Oracle Data Guard
NVM-e
ACFS Snapshots
Bash Scripts
Linux
Windows

Ingredients
Oracle Data Guard
NVM-e
ACFS Snapshots
Bash Scripts
Linux
Windows + Perl

Target Architecture Overview
NEW SERVER A @ SITE A
NVM-e
NEW SERVER B @ SITE B
NVM-e

NVM-e
GRID INFRASTRUCTURE 12cR1
E:oracleproduct12.1.0.2grid
DG +DATA
NVM-e
DG +DATA

NVM-e
DG +DATA
NVM-e
DG +DATA
ACFS
E:acfsDATA
ACFS
E:acfsFRA
ACFS
E:acfsREDO

NVM-e
DG +DATA
NVM-e
DG +DATA
SNAPSHOT
DB_NAME
ACFS
E:acfsDATA
ACFS
E:acfsFRA
ACFS
E:acfsREDO

NVM-e
DG +DATA
DB_NAME
PRIMARY
NVM-e
DG +DATA
ACFS
E:acfsDATA
ACFS
E:acfsFRA
ACFS
E:acfsREDO
DB_NAME
STANDBY
SNAPSHOT
DB_NAME
DATA
GUARD

NVM-e
DG +DATA
DB_NAME
PRIMARY
NVM-e
DG +DATA
ACFS
E:acfsDATA
ACFS
E:acfsFRA
ACFS
E:acfsREDO
DB_NAME
STANDBY
SNAPSHOT
DB_NAME
DATA
GUARD
SNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPY

NVM-e
DG +DATA
DB_NAME
PRIMARY
NVM-e
DG +DATA
ACFS
E:acfsDATA
ACFS
E:acfsFRA
ACFS
E:acfsREDO
DB_NAME
STANDBY
SNAPSHOT
DB_NAME
DATA
GUARD
DATABASE
CLONE
SNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPY

NVM-e
DG +DATA
DB_NAME
PRIMARY
NVM-e
DG +DATA
ACFS
E:acfsDATA
ACFS
E:acfsFRA
ACFS
E:acfsREDO
DB_NAME
STANDBY
SNAPSHOT
DB_NAME
DATA
GUARD
DATABASE
CLONE
DB_NAME
CLONE
SNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPY

NVM-e
DG +DATA
DB_NAME
PRIMARY
NVM-e
DG +DATA
ACFS
E:acfsDATA
ACFS
E:acfsFRA
ACFS
E:acfsREDO
DB_NAME
STANDBY
SNAPSHOT
DB_NAME
DATA
GUARD
DATABASE
CLONE
DB_NAME
CLONE
SNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPYSNAPSHOT
COPY
SNAPSHOTS
ON DATA
NO
SNAPSHOTS
ON FRA/REDO

Thoughts about ACFS

Snapshot on Snapshot: Why?
Multiple databases!

Multiple databases!
I want to clone RED database

Multiple databases!
But I get copy on write for ALL of them!

Multiple databases!

Multiple databases!
Snapshot space usage will grow much faster

Snapshots on snapshots are better!

Not Supported

ACFS and Oracle Restart
From the documentation:
Oracle Restart does not support root-based Oracle ACFS resources for this release:
Loading Oracle ACFS drivers
Mounting Oracle ACFS file systems listed in the Oracle ACFS mount registry
Mounting resource-based Oracle ACFS database home file systems
And:
Oracle ACFS does not support data files or redo logs in an Oracle Restart configuration.

Assume the risk…
Primary: Oracle Restart with DB on ASM Only
Standby and Test: Oracle Restart with DB on ACFS

GI + ACFS Implementation

GI Installation and Patching… out of scope?
Meet system requirements (e.g. Windows object access auditing disabled)
Partition Disks and stamp the partitions with asmtool.exe
Install Oracle Grid Infrastructure
Apply the latest bundle patch
At that time: BP 12.1.0.2.171017. Now: BP 12.1.0.2.171130

Patching "sometimes" leaves the old drivers
Solution?
– crsctl stop has; acfsload stop
– acfsinstall /i /a ; acfsinstall /i /o ; acfsinstall /i /l
Directory of C:WindowsSystem32drivers
12/20/2016 05:15 AM 3,461,008 oracleacfs.sys
12/20/2016 05:02 AM 841,104 oracleadvm.sys
12/20/2016 04:53 AM 478,608 oracleoks.sys
Directory of C:WindowsSystem32drivers
06/21/2017 11:42 AM 3,461,008 oracleacfs.sys
06/21/2017 11:29 AM 841,104 oracleadvm.sys
06/21/2017 11:19 AM 478,608 oracleoks.sys
– acfsload start -s ; crsctl start has

ACFS filesystems with Oracle Restart...
ACFS Drivers need to be loaded at boot
set ORACLE_BASE=E:oracle
set ORACLE_HOME=E:oracleproduct12.1.0.2grid
set PATH=...
%ORACLE_HOME%binacfsload start -s

The ACFS filesystem resource does not exist on Oracle Restart configurations
Must use action script / custom type / resource!

crsctl add type acfs_fs.type -basetype local_resource -attr
"ATTRIBUTE=ACTION_SCRIPT,TYPE=string,DEFAULT_VALUE=E:oracleTrivadisscri
ptsstop_startacfs_actionscript.bat,ATTRIBUTE=AUTO_START,TYPE=string,DEFA
ULT_VALUE=always,ATTRIBUTE=START_TIMEOUT,TYPE=int,DEFAULT_VALUE=100,ATTRIB
UTE=STOP_TIMEOUT,TYPE=int,DEFAULT_VALUE=100,ATTRIBUTE=CHECK_INTERVAL,TYPE=
int,DEFAULT_VALUE=10,ATTRIBUTE=START_DEPENDENCIES,TYPE=string,DEFAULT_VALU
E=hard(ora.DATA.dg)pullup(ora.DATA.DG),ATTRIBUTE=STOP_DEPENDENCIES,TYPE=st
ring,DEFAULT_VALUE=hard(ora.DATA.dg),ATTRIBUTE=ACL,TYPE=string,DEFAULT_VAL
UE='owner:nt
authoritysystem:rwx,pgrp:ORA_ASMDBA:rwx',ATTRIBUTE=TYPE_ACL,TYPE=string,D
EFAULT_VALUE='owner:nt authoritysystem:rwx,pgrp:ORA_ASMDBA:rwx,other::r--
'"

crsctl add resource FRA.acfs -type acfs_fs.type -attr "ACL='owner:nt
authoritysystem:rwx,pgrp:ORA_ASMDBA:rwx,other::r--'"
crsctl add resource DATA.acfs -type acfs_fs.type -attr "ACL='owner:nt
crsctl add resource REDO.acfs -type acfs_fs.type -attr "ACL='owner:nt

Database Implementation

Naming Convention is crucial!
EXAMPLES SITE1 SITE2
ORACLE_SID/DB_NAME LUDO LUDO
DB_UNIQUE_NAME LUDO_SITE1 LUDO_SITE2
CRS_RESOURCE ora.ludo_site1.db ora.ludo_site2.db
DG Configuration LUDO LUDO
DG Member ludo_site1 ludo_site2
DGConnectIdentifier ludo_site1 ludo_site2
Static Listener Entry
(GLOBAL_DBNAME)
LUDO_SITE1_DGMGRL
LUDO_SITE2_DGMGRL
Base Snapshot Name LUDO LUDO
Snapshot Copies (when applies) -
LUDO.18H.<weekday>
E.g. LUDO.18H.Thu

Configuration of new standby databases
Create the directories for controlfiles and redo log
Create the main snapshot
– acfsutil snap create LUDO E:acfsdata
Add the Database (in this case, as standby)
– srvctl add database -d LUDO_SITE2 -o
E:oracleproduct12.1.0.2db_BP171017 -p
%ORACLE_HOME%databasespfileLUDO.ora -r PHYSICAL_STANDBY -s
mount -n LUDO -i LUDO -a "DATA"
Add the dependencies with the filesystem
– crsctl modify resource ora.ludo_site2.db -attr
"START_DEPENDENCIES='hard(FRA.acfs,DATA.acfs,REDO.acfs)pullup(F
RA.acfs,DATA.acfs,REDO.acfs)',STOP_DEPENDENCIES='hard(FRA.acfs,
DATA.acfs,REDO.acfs)'" -unsupported

Data Guard implementation...
Duplicate for standby
Create broker configuration
Add standby database
... usual steps!
Oracle Managed Files because the primary is on ASM

Configuration of new clone databases
Create the service with ORADIM.exe
Add the resource with srvctl add database
– -db is the unique name, for the copies it MUST be the same as the db_name!
Add the dependency with the filesystem (same as a new master database)

Snapshots

ACFS Filesystem Layout: Source and its snapshot
E:acfsdata.ACFSsnaps<DB_NAME><DB_UNIQUE_NAME>DATAFILE...
DB_CREATE_FILE_DEST
Example of datafile:
E:acfsdata.ACFSsnapsLUDOLUDO_SITE2DATAFILE01_MF_SYSTEM_DNNMQRVC_.DBF
(Oracle Managed Files)
SOURCE DATABASE LAYOUT
E:acfsdata.ACFSsnaps<SNAP_NAME><DB_UNIQUE_NAME>DATAFILE...
DB_CREATE_FILE_DEST
Example:
E:acfsdata.ACFSsnapsLUDO.18H.THULUDO_SITE2DATAFILE...
SAME AS SOURCE
SNAPSHOT COPY LAYOUT

ACFS Filesystem Layout: Clones
Source Snapshot
E:acfsdata.ACFSsnaps<SNAP_NAME><SRC_DB_UNIQUE_NAME>
DATABASE CLONE LAYOUT
E:acfsdata.ACFSsnaps<CLONE_DB_NAME><SRC_DB_UNIQUE_NAME>
ONLY THE SNAPSHOT NAME CHANGES
Example of datafile:
E:acfsdata.ACFSsnapsTESTVICOLUDO_SITE2DATAFILE01_MF_SYSTEM_DNNMQRVC_.DBF
Clone
DB_NAME
Source
DB_UNIQUE_NAME

Detail of the cloning process
LUDO_SITE2
SNAPSHOT
LUDO
Let's give it a name:
– Primary: LUDO_SITE1
– Standby: LUDO_SITE2
MRP0

LUDO_SITE2
SNAPSHOT
LUDO
Stop Apply

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.SUN
Stop Apply
Create a Snapshot

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.SUN
Stop Apply
Create a Snapshot
Copy pfile and Controlfile to trace

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.SUN
Stop Apply
Create a Snapshot
Start Apply
MRP0

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.SUN
Stop Apply
Create a Snapshot
Start Apply
MRP0
snap_database.pl

Running the script
snap_all_database.bat
– Will overwrite the snapshot named <DB_NAME>.18H.<Weekday> for all the dbs
tvd_perl snap_database.pl -b SOURCE [-n SNAPNAME] [-s SUFFIX]
– Where SOURCE is the source DB (e.g. LUDO)
– SNAPNAME is the name of the new snapshot (defaults to <SOURCE>.18H)
– SUFFIX can be set to «weekday» to automatically add the week of the day to the
name of the snapshot
– E.g. tvd_perl snap_database.pl -b LUDO -n PREUPG will create a snapshot
PREUPG from the standby database LUDO

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.SUN
Stop Apply
Create a Snapshot
Start Apply
MRP0

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.SUN
Stop Apply
Create a Snapshot
Start Apply
MRP0
LUDO.MON

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.SUN
Stop Apply
Create a Snapshot
Start Apply
MRP0
LUDO.MON
LUDO.TUE

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.SUN
Stop Apply
Create a Snapshot
Start Apply
MRP0
LUDO.MON
LUDO.TUE
LUDO.WED
LUDO.THU
LUDO.FRI
LUDO.SAT

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.MON
LUDO.TUE
LUDO.WED
LUDO.THU
LUDO.FRI
LUDO.SAT
MRP0
LUDO.SUN

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.MON
LUDO.TUE
LUDO.WED
LUDO.THU
LUDO.FRI
LUDO.SAT
MRP0
LUDO.SUN
Delete previous clone

LUDO_SITE2
SNAPSHOT
LUDO
DATABASE
CLONE
LUDO.MON
LUDO.TUE
LUDO.WED
LUDO.THU
LUDO.FRI
LUDO.SAT
MRP0
LUDO.SUN
Create a Snapshot

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.MON
LUDO.TUE
LUDO.WED
LUDO.THU
LUDO.FRI
LUDO.SAT
MRP0
LUDO.SUN
Create a Snapshot
Replace snapshot name in pfile and ctlfile
DATABASE
CLONE

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.MON
LUDO.TUE
LUDO.WED
LUDO.THU
LUDO.FRI
LUDO.SAT
MRP0
LUDO.SUN
Create a Snapshot
Open resetlogs
DATABASE
CLONE
CLONE

LUDO_SITE2
SNAPSHOT
LUDO
LUDO.MON
LUDO.TUE
LUDO.WED
LUDO.THU
LUDO.FRI
LUDO.SAT
MRP0
LUDO.SUN
Create a Snapshot
Open resetlogs
DATABASE
CLONE
CLONE
clone_from_snap.pl

Cloning the databases: running the procedure
tvd_perl clone_from_snap.pl –b SOURCEDB –s SNAPNAME –d DESTDB
– Where SOURCEDB is the original DB_NAME of the source DB (e.g. LUDO)
– SNAPNAME is the name of the snapshot used for the clone (e.g. LUDO.18h.Thu)
– DESTDB is the name of the destination database. A new snapshot will be created with this
name (or overwritten if it already exists)
– E.g.:
tvd_perl clone_from_snap.pl –b LUDO –s LUDO.18H.Thu –d testclone
Will clone LUDO to testclone using LUDO.18H.Thu snapshot

Checking the ACFS

Checking the ACFS
Two custom aliases inside TVD-basenv
$ acfsinfo
– Gives information (including size and free space) about the ACFS filesystems
$ snaps
– Gives information about the snapshots in E:acfsdata
– It includes parent name and creation time
– The snapshot size does not apply
(Oracle copy-on-first-write works differently from other techhnologies)
Another script is integrated with customer's PRTG environment

Checking the ACFS
SQL> SELECT fs_name, type, parent, snap_name, to_char(create_time,'YYYY-MM-DD
HH24:MI:SS') as CREATE_TIME FROM v$asm_acfssnapshots ORDER BY create_time;
FS_NAME TYPE PARENT SNAP_NAME CREATE_TIME
------------ ---- -------------- -------------- --------------------
E:acfsDATA RW NULL LUDO 2017-06-21 12:48:22
E:acfsDATA RW LUDO CLONE002 2017-07-06 15:50:49
E:acfsDATA RW LUDO LUDO.18H.SAT 2017-07-08 18:00:56
E:acfsDATA RW LUDO LUDO.18H.SUN 2017-07-09 18:00:57
E:acfsDATA RW LUDO LUDO.18H.MON 2017-07-10 18:00:47
E:acfsDATA RW LUDO LUDO.18H.TUE 2017-07-11 18:00:28
E:acfsDATA RW LUDO LUDO.18H.WED 2017-07-12 18:00:32
E:acfsDATA RW LUDO.18H.WED CLONE001 2017-07-14 10:20:31
E:acfsDATA RW LUDO.18H.WED CLONE003 2017-07-14 10:58:52
E:acfsDATA RW LUDO LUDO.18H.FRI 2017-07-14 11:04:09
E:acfsDATA RW LUDO PREUPG 2017-07-13 11:42:28
E:acfsDATA RW PREUPG CLONE004 2017-07-13 11:47:24

Checking the ACFS
ludo@winsrv:E:oraclelocaldbaetc [OraDb11g_home1] acfsinfo
FS_NAME STATE CORRUPT TOTAL_SIZE TOTAL_FREE SNAP_USAGE PCT_USAGE
-------------- ------------- ------- ---------- ---------- ---------- ----------
E:acfsREDO AVAILABLE FALSE 51200 32200.07 0 37.11
E:acfsDATA AVAILABLE FALSE 1331200 517929.54 825515.75 61.09
E:acfsFRA AVAILABLE FALSE 122880 36200.09 0 70.54
SELECT fs_name, state, corrupt
, ROUND(total_size,2) as total_size
, ROUND(total_free,2) as total_free
, ROUND(total_snap_space_usage,2) as snap_usage
, ROUND((100-(total_free/total_size)*100),2) as PCT_USAGE FROM v$asm_filesystem;

ACFS and Windows.
What can go wrong?

The Check inside the action script
Check was executing `acfsutil info fs`
During shapshot operations there are ACFS metadata locks
The action script was failing
Oracle Restart decided to restart the FS (indluding all the databases!)
Now a lockfile mechanism and higher script timeouts prevent the problem
if ( -f $lockFile ) {
&DoMsg ("ACFS operation in progress. Will not check.");
}

Standby reinstate: erase all?
ALL the snapshots depend on the first one containing the standby
Reinstate of the standby will increase massively the disk space
If not enough, all the clones will be destroyed and taken again

Need to switchover
The standby is on a snapshot that has snapshots
In case of switchover the performance will decrease significantly
The customer is ready to accept degraded performance in such case

Three months long service request
**************************************************************************
* *
* Bugcheck Analysis *
* *
**************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck 3B, {c0000005, fffff8012845e144, ffffd0006da60b10, 0}
*** WARNING: Unable to verify timestamp for oracleacfs.sys
*** ERROR: Module load completed but symbols could not be loaded for
oracleacfs.sys
Probably caused by : oracleacfs.sys ( oracleacfs+3d144 )
Followup: MachineOwne

Three months long service request
Hi Ludovico,
We got a development teame. They are going to include patch on
12.1.0.2.171130.
Targeting fix for Bundle Patch via this CI bug:
BUG 26994229 - CI BACKPORT OF BUG 19439577 FOR INCLUSION IN WINDOWS
DB BP 12.1.0.2.171130
I would request you to monitor the next PSU release and apply on your
environment.

Customer satisfied?

Customer satisfied?
GOOD
Clone process reduced to 3 minutes
Space consumption: -80%
Performance increase due to NVMe
BAD
Stability problems (almost fixed)
Big operational change
Clones cannot live forever (wow, that
sounds just like Blade Runner)
because of space usage

12/2/2017 DB Cloning : Customer Story102
Technology on its own won't help you.
You need to know how to use it properly.

Thank you!
@trivadis #opencompany
Ludovico Caldara
@ludodba ukoug_tech17

How to bake a Customer Story with With Windows, NVM-e, Data Guard, ACFS Snapshots & Perl

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a How to bake a Customer Story with With Windows, NVM-e, Data Guard, ACFS Snapshots & Perl

Semelhante a How to bake a Customer Story with With Windows, NVM-e, Data Guard, ACFS Snapshots & Perl (20)

Mais de Ludovico Caldara

Mais de Ludovico Caldara (6)

Último

Último (20)

How to bake a Customer Story with With Windows, NVM-e, Data Guard, ACFS Snapshots & Perl