6. Allocate Any Storage to Delphix
Allocate Storage
Any type
Pure Storage + Delphix
Better Performance for
1/10 the cost
7. One time backup of source database
Production
Supports
Instance
Instance
Instance
Database
File system
Application Stack Virtualization
Upcoming
8. DxFS (Delphix) Compress Data
Production
Instance
Instance
Instance
Database
File system
Data is
compressed
typically 1/3
size
9. Incremental forever change collection
Production
Instance
Instance
Instance
Database
Changes
Time Window
File system
• Collected incrementally forever
• Old data purged
10. Change collection works with snapshots
Snapshot 0:
Source Full Copy
Snapshot 1:
Changed blocks at
time T1
11. Snapshots immediately look like full copies as they are
mapped
onto the previous copy
Snapshot 0
Snapshot 1
14. Purging old snapshot releases any blocks that have
been since replaced, but keeps all unchanged blocks
Snapshot 2
Snapshot 1
Drop
Snapshot 0
15. Cloning : almost no storage at creation since it is all
shared. Upon usage changed blocks will add to storage
used
Production
Instance
Instance
Instance
Instance
Instance
Database
Database
Time Window
File system
16. Typical Architecture
Production
Dev, QA, UAT
Reporting
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Instance
Database
Database
Database
Database
Database
File system
File system
File system
File system
File system
File system
File system
File system
File system
File system
File system
Backup
Instance
Instance
Instance
File system
File system
29. DevOps With Delphix
1.
2.
3.
4.
5.
Efficient QA: Low cost, high utilization
Quick QA : Fast Bug Fix
Every Dev gets DB: Parallelized Dev
Full DB : Less Bugs
Fast Builds: Culture of Yes
30. Without Delphix
1. Inefficient QA: Long Build times
Build
QA Test
Build Time
96% of QA time was building environment
$.04/$1.00 actual testing vs. setup
31. : With Delphix
1. Efficient QA: Lower cost
Build
QA Test
Build Time
B
u
i
l
d
T
i
m
e
QA Test
1% of QA time was building environment
$.99/$1.00 actual testing vs. setup
38. Without Delphix
4. Subsets : cause
bugs
The Production ‘Wall’
Classic problem is that queries that
run fast on subsets hit the wall in
production.
Developers are unable to test against
all data
40. Without Delphix
5. Slow Environment Builds:
Slow Environment Builds: 3-6 Months to Deliver
Data
Developer
Asks for DB
Get Access
Manager
DBA
System
Admin
Storage
Admin
approves
Request
system
Setup DB
Request
storage
Setup
machine
Allocate
storage
(take snapshot)
45. 1. Forensics: Investigate Production Bugs
Development
Instance
Instance
Time Window
Anomaly on Prod
Possible code bug
At noon yesterday
Spin up VDB of Prod
as it was during
anomaly
46. 2. Testing : Rewind for patch and QA testing
Prod
Development
Instance
Instance
Time Window
Time Window
47. 2. Testing: A/B
Instance
Test A with Index 1
Instance
Instance
Time Window
• Keep tests for compare
• Production vs Virtual
– invisible index on Prod
– Creating index on virtual
• Flashback vs Virtual
Test B with Index 2
48. 3. Recovery: Surgical recover of Production
Source
Development
Instance
Instance
Spin VDB up
Before drop
Time Window
Problem on Prod
Dropped Table Accidently
49. 3. Recovery Surgical or Full Recovery on VDB
Dev1 VDB
Source
Instance
Instance
Dev2 VDB Branched
Source
Time Window
Dev1 VDB
Time Window
Instance
50. 3. Recovery: Virtual to Physical
Source
VDB
Instance
Instance
Spin VDB up
Before drop
Time Window
Corruption
51. 50 days of backup in size of
Original
3. Recovery
56. ETL and DW Refreshes
Prod
DW & BI
Instance
Instance
Data Guard – requires full refresh if used
Active Data Guard – read only, most reports don’t work
57. Fast Refreshes
• Collect only Changes
• Refresh in minutes
Prod
Instance
BI
DW
Instance
Instance
ETL
24x7
Interview Delphix blew me away. As a DBA I had to spend 50% of my time making copiesAfter joining Delphix, I banged on Delphix for 2 years. It worksBlog entryReinforce ideas we’ve already seen from a different perspectiveWork for a company called DelphixWe write software that enables Oracle and SQL Server customers toCopy their databases in 2 minutes with almost no storage overheadWe accomplish that by taking one initial copy and sharing the duplicate blocks Across all the clones
In the physical database world, 3 clones take up 3x the storage.In the virtual world 3 clones take up 1/3 the storage thanks to block sharing and compressionDelphix orchestrates access to the shared compressed blocks on the storage that Delphix manages.Delphix manages changes block access only making changed blocks visible to the database that made that block change.
Software installs an any x86 hardware uses any storage supports Oracle 9.2-12c, standard edition, enterprise edition, single instance and RAC on AIX, Sparc, HPUX, LINUX support SQL Server
EMC, Netapp, Fujitsu, Or newer flash storage likeViolin, Pure Storage, Fusion IO etc
Delphix does a one time only copy of the source database onto Delphix
Physically independent but logically correlatedCloning multiple source databases at the same time can be a daunting task
One example with our customers is InformaticaWho had a project to integrate 6 databases into one central databaseThe time of the project was estimated at 12 monthsWith much of that coming from trying to orchestratingGetting copies of the 6 databases at the same point in timeLike herding cats
Informatical had a 12 month project to integrate 6 databases.After installing Delphix they did it in 6 months.I delivered this earlyI generated more revenueI freed up money and put it into innovationwon an award with Ventana Research for this project
Developer each get a copyFast, fresh, full, frequentSelf serviceQA branch from DevelopmentFederated cloning easyForensicsA/B testingRecovery : Logical and physical Development Provision and RefreshFullFreshFrequent (Many) Source control for code, data control for the database Data version per release version Federated cloning QA fork copies off to QA QA fork copies back to Dev Instant replay – set up and run destructive tests performance A/B Upgrade patching Recovery Backup 50 days in size of 1 copy, continuous data protection (use recent slide ob backup schedules full, incr,inrc,inrc, full) Restore logical recovery on prod logical recovery on Dev Debugging debug on clone instead of prod debug on data at the time of a problem Validate physical integrity (test for physical corruption)
We talked to Presbyterian HealthcareAnd they told us that they spend 96% of their QA cycle time building the QA environmentAnd only 4% actually running the QA suiteThis happens for every QA suitemeaningFor every dollar spent on QA there was only 4 cents of actual QA value Meaning 96% cost is spent infrastructure time and overhead
Presbyterian when from 10 hour builds to 10 minute buildsTotal Investment in Test Environment: $2M/year10 QA engineersDBA, storage team dedicated to support testingApp, Oracle server, storage, backupsRestore load competes with backup jobsRequirements: fast data refresh, rollbackData delivery takes 480 out of 500 minute test cycle (4% value)$.04/$1.00 actual testing vs. setup
Because of the time required to set up QA environmentsThe actual QA tests suites lag behind the end of a sprint or code freezeMeaning that the amount of time that goes by after the introduction of a bug in code and before the bug is found increasesAnd the more time that goes by after the introduction of a bug into the codeThe more dependent is written on top of the bug Increasing the amount of code rework required after the bug is finally foundIn his seminal book that some of you may be familiar with, “Software Engineering Economics”, author Barry Boehm Introduce the computer world to the idea that the longer one delays fixing a bug in the application design lifescyleThe more expensive it is to to fix that bug and these cost rise exponentially the laterThe bug is address in the cycle
Not sure if you’ve run into this but I have personally experience the followingWhen I was talking to one group at Ebay, in that development group they Shared a single copy of the production database between the developers on that team.What this sharing of a single copy of production meant, is that whenever a Developer wanted to modified that database, they had to submit their changes to codeReview and that code review took 1 to 2 weeks.I don’t know about you, but that kind of delay would stifle my motivationAnd I have direct experience with the kind of disgruntlement it can cause.When I was last a DBA, all schema changes went through me.It took me about half a day to process schema changes. That delay was too much so it was unilaterally decided byThey developers to go to an EAV schema. Or entity attribute value schemaWhich mean that developers could add new fields without consulting me and without stepping on each others feat.It also mean that SQL code as unreadable and performance was atrocious.Besides creating developer frustration, sharing a database also makes refreshing the data difficult as it takes a while to refresh the full copyAnd it takes even longer to coordinate a time when everyone stops using the copy to make the refreshAll this means is that the copy rarely gets refreshed and the data gets old and unreliable
For example Stubhub went from 5 copies of production in development to 120Giving each developer their own copy
To circumvent the problems of sharing a single copy of productionMany shops we talk to create subsets.One company we talked to , RBS spends 50% of time copying databases have to subset because not enough storagesubsetting process constantly needs fixing modificationNow What happens when developers use subsets -- ****** -----
Stubhub (ebay) estimates that 20% of there production bugs arise from testing onSubsets instead of full database copies.
Stubhub estimated a 20% reduction in bugs that made it to production
Due to the constraints of building clone copy database environments one ends up in the “culture of no”Where developers stop asking for a copy of a production database because the answer is “no”If the developers need to debug an anomaly seen on production or if they need to write a custom module which requires a copy of production they know not to even ask and just give up.
State of Colorado has a 100 projects support 3KLA tencor can only support 2 projects of a dozen
Take for example a 9TB source databaseIt’s compressed to 3TB on Delphix.Then we still have 6TB on Delphix before we reach the size of the original database, but that 6TB is actuall 18TB of data at a 1/3 compressionWhich means we can have the original database and 18TB of changes in the size of the original databaseWe see on average about 50 days worth of back up in the size of the original databaseMultiple scripted dumps or RMAN backups are used to move data today. With application awareness, we only request change blocks—dramatically reducing production loads by as much as 80%. We also eliminate the need for DBAs to manage custom scripts, which are expensive to maintain and support over time.
Once Last Thinghttp://www.dadbm.com/wp-content/uploads/2013/01/12c_pluggable_database_vs_separate_database.png
250 pdb x 200 GB = 50 TBEMC sells 1GB$1000Dell sells 32GB $1,000.terabyte of RAM on a Dell costs around $32,000terabyte of RAM on a VMAX 40k costs around $1,000,000.
http://www.emc.com/collateral/emcwsca/master-price-list.pdf These prices obtain on pages 897/898:Storage engine for VMAX 40k with 256 GB RAM is around $393,000Storage engine for VMAX 40k with 48 GB RAM is around $200,000So, the cost of RAM here is 193,000 / 208 = $927 a gigabyte. That seems like a good deal for EMC, as Dell sells 32 GB RAM DIMMs for just over $1,000. So, a terabyte of RAM on a Dell costs around $32,000, and a terabyte of RAM on a VMAX 40k costs around $1,000,000.2) Most DBs have a buffer cache that is less than 0.5% (not 5%, 0.5%) of the datafile size.
