1. Faster Replication of
Sunrise Clinical
Manager Databases for
Running DBCC
Ziaul Mannan
-Senior Technical DBA
Howard Goldberg
-Director, Clinical Systems and Support

2. Yale-New Haven Hospital
New Haven, Connecticut
 944 Bed Tertiary Teaching Facility
 2600 Medical Staff
 7550 Employees
 100% CPOE
 Average 350,000 orders monthly
 Average Daily Census is 724
 7 Time “Most Wired” and 3 Time “Most Wireless”
Hospital by Hospitals and Health Networks

4. Future Clinical Cancer Center At
Yale-New Haven Hospital
 112 inpatient beds
 Outpatient treatment rooms
 Expanded operating rooms
 Infusion suites
 Diagnostic imaging services
 Therapeutic radiology
 Specialized Women's Cancer Center
 Yale-New Haven Breast Center/GYN Oncology Center
 Tentative Completion Date: 2009

5. Problem
• With continual growth of SCM Master DB, it just
gets longer to copy the db over to DBCC Server
for DBCC Checks.
• In our case ~ 12 Hours.
• During this long time there‟s a potential for the
Master DB to be online with corruption in it.
• Too many steps involved to backup,copy the DB
over and run DBCC checks then look for errors.

6. Goals
• Copy database over to the DBCC server and restore in
shortest possible time
• Run DBCC and scan reports for errors
• Notify DBAs via e-mails or alerts on the result of DBCC
Challenges
• Find an innovative way of duplicating the production DB
• Backup and restore production DB without jeopardizing
availability, data integrity or performance
• Detect corruption in DB before it can spread

7. YNHH Production Environment
• SCM 4.5 SP5 RU3
– Migrating to SCM4.5 with SMM
– Total Users Defined – 10,000
– Users logged on at peak hours ~450
– SCM Reports, HL7 Interfaces,CDS,Multum
– No CDR
– Total disk for data 700GB (all servers)
– Total disk for Archive 500 GB

8. YNHH Production Environment
• MS SQL Server
– SQL Server 2000 EE Build 2195: SP 4
– Master and Enterprise on their own Servers and both
clustered
– MSCS and EMC SRDF/CE used as Clustering
solution
• OS and Hardware
– Window 2000 Advance Server SP4
– Local SCSI and EMC Disks on Symmetrix

9. YNHH Production Environment
– Distributed SCM Environment
• Master Server (MSCS Cluster using EMC
SRDF/CE) ~ 125 GB DB
• Enterprise Server (MSCS+EMC SRDF/CE)
• HL7 Server (MSCS+EMC SRDF/CE)
• Reports Server (MSCS+EMC SRDF/CE)
• CDS Server (MSCS+EMC SRDF/CE)
• Multum Server (MSCS+EMC SRDF/CE)
• Compaq Servers - DL760G2,DL380G3,DL560
• 2-8 CPUs, 3-8 GB RAM

10. YNHH SCM Production
Environment
MSMQ DCs SCM Master DB
Client Workstations
Enterprise Server
XAENTER1PA
Master Active DB XAENTERP
XAMASTER1PA XAENTERCL1
XAMASTERP SCM Client SCM Client
XAMASTERCL1
YNHORG2
YNHORG4
HL7 Interface Server HL7 Interface Server
Executive Server Manager Server
MSMQ XAAPPS2P
XAHL71PA
XAHL7P
XAHL7CL1
SunriseXA Services
Notification, CDS and Order Generation Server Multum Server Report Server
XACOGNS1PA XAMULTUM1PA XAREPORT1P
XACOGNSP XAMULTUMP XAREPORTP
XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1

11. Time to Complete DBCC
(Conventional Way)
• Backup of PROD DB ~ 5hrs
• Copy over to the DBCC Server via network ~5 hrs
• Run DBCC on copy of PROD DB ~ 2 hrs
• Total time spent before a corruption is detected ~ 12 Hrs

12. Conventional DBCC Process
DBCC Server SCM Master DB
Client Workstations
Copy over the Master Active
network DB
XAMASTER1P Enterprise Server
A XAENTER1PA
Copy of XAENTERP
XAMASTERP
SCM Master XAENTERCL1
XAMASTERCL SCM Client SCM Client
Active DB
1
ERM Client
HL7 Interface Server HL7 Interface Server
Executive Server Manager Server
MSMQ XAAPPS2P
XAHL71PA
XAHL7P
XAHL7CL1
SunriseXA Services
Notification, CDS and Order Generation Server Multum Server Report Server
XACOGNS1PA XAMULTUM1PA XAREPORT1P
XACOGNSP XAMULTUMP XAREPORTP
XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1

13. Solutions/Tools
• DBCC
• VDI
• EMC ERM
• BCV
• Time Finder

14. Database Corruption
• Compromises the database for logical and physical damage
• Damages the database in its integrity and consistency
• May impair database‟s functionality
• Corruption can be caused by different reasons:
– Disk Error
– Memory Error
– Power loss
– Network

15. DBCC
• Database Consistency Checker
– Very important database maintenance tool
– Highly recommended by Eclipsys Support
– It is executed against a database to detect any corruption before
it compromises the database
• DBCC scans data in database for any corruption
• Results are logged in a text file
• Different types of DBCC commands used to repair
different kind of database corruption
• DBCC should not be run on production DB server as it
severely effects the performance of the server/DB

16. DBCC
• Eclipsys recommends running DBCC on a restored copy
of Production DB on a separate Server
• Eclipsys also recommends running DBCC on a regular
basis
• DBCC statements that Eclipsys wants us to run
– DBCC CheckTable - the integrity of the data and index pages for the
specified table
– DBCC CheckDB - the allocation and structural integrity for all tables in a
database
– DBCC CheckAlloc - The integrity of disk allocation for a specified database
– DBCC CheckCatalog - For consistency in and between system tables in the
specified database

17. VDI
• Virtual Device Interface (VDI) is the programming interface for the
Backup Restore API. It provides function calls to manipulate the
backup and restore functionality in SQL Server.
• VDI provides an alternative to using named pipes for backups
• Provides interface and allows snapshot backups
• VDI is used by several third-party tools that provide capabilities to
back up SQL Server databases.
• In SQL Server2000 information provided in VDI metadata can be
used to restore a database from existing files created by TimeFinder

18. EMC ERM
• EMC Replication Manager creates replicas of mission
critical information on disk arrays
• ERM creates point in time replicas of databases or file
systems
• It creates replicas simply, quickly and automatically.
• Automates all the important procedures related to data
replication

19. General Functionality
• Where is the Original Data
– Production Host
• Application Set (formerly know as info set)
– What is going to be Replicated - „Activities‟
• Filesystem
• SQLServer Databases
• How and When is the data going to be replicated.
– Mount host name
– Pre / Post Scripts
– Backup Utilities
– Mount Programs

20. System Interfaces
• Server Software
• Agent Software

24. BCV
• EMC technology
• Stands for Business Continuity Volumes
• Creates mirror images of active production volumes
• Disks are synchronized in the back ground transparent
to the users and application
• Can be spit and mounted to another host in point in time
with out any user outages
• Once host processes on the BCV device are complete,it
can again be mirrored
• Each BCV contains a unique host address and is
configured as a single mirror

25. Time Finder
• EMC Technology
• Creates multiple, independently addressable BVCs
• TimeFinder helps maintain viable and secure system
while ensures that each competing application receives
data they need

26. Time Finder Integration with MS
SQL Server 2000
• TimeFinder SQL Server Integration Module (TSIM)
• TSIM integrates and automates the features of Time
Finder,Symmetrix Application Program Interface (API)
and Microsoft‟s tool (VDI) supplied with MS SQL
Server2000
• TSIM provides several interfaces to back up and restore
database through SQL Server VDI
• It can be used to reduce back/recovery time

27. Replication Manager Architecture

28. Replication Manager Architecture
• Console Software
– This will let you control Replication Manager from Windows or
Unix workstation via TC/IP connection to the server
• Server Software
– Replication manager Service (IRD)
– Replication Manager Database
• Agent Software
– Replication Manager Agent (IRCCD)
– Storage Service Software
– Application Agents for each supported interface
• Database and File System Agents

29. Table: Database and File System
Agents
Agent Information Replicated
Oracle Agent Oracle information on raws devices or in
datafiles
Oracle information in RAC *
Information from other applications, that has
been organized within Solaris,Linux,HP-
UX,IBM AIX or NTFS file systems
UDB Agent UDB information on raw devices or in datafiles
UDB information in EEE clusters **
Information from other applications, as long as
data has been organized within Solaris,Linux,
HP-UX or IBM AIX file systems
SQL Server for NTFS File System Agent SQL Server 2000/2005 information
Information in MSCS clusters*
Information from other applications, as long as
data has been organized within NTFS file
systems
Exchange for NTFS File System Agent Exchange 2000/2003 information
Information in MSCS clusters*
Information from other applications, as long as
data has been organized within NTFS file
systems
*Data stored in clusters can be mounted to a nonclustered environment only
**Replication Manager UDB agent supports EEE in only a single-node instance

30. Agent Purpose and Functions
• Specify which data to replicate
• Makes sure that data can be replicated safely
• Quiesce the database
• Returns db to normal operation
• Recover db during mounting operation
• Shutdown and/or unmount database during restore
operation

31. Determining Replication Goals
• Repurposing
– Replica can be used carry out other functions; such as
reporting,analysis, data mining without impacting production
• Backup and Recovery
– Replica can be used to backup the data to tape without effecting
production performance
• Disaster Restart
– After a disaster the replica can be used to restart the application.
Point in time data recovery is possible using roll forward
• Copies of Replica
– SAN copy can be used to copy the replica for multi purpose use

32. Data Management with Replication
Manager
• Understanding storage options
– RM can create replicas of data for Symmetrix DMX, Clarion
• Selecting storage types
– Flexible to store replicated data
• Symmetrix BCVs,Clones,VDEV,Clarion Clone and snapshot
• Symmetrix storage option
– For Symmetrix DMX - replicas can be created on
STDs,BCVs,VDEVs
• Clarion storage option
– Can use both clone and snapshot functionality

33. ERM in YNHH SCM Production
Environment
SCM Master DB
Client Workstations
Master Active
ERM Server DB
XAMASTER1P Enterprise Server
A XAENTER1PA
XAMASTERP XAENTERP
XAMASTERCL XAENTERCL1
SCM Client SCM Client
1
ERM Client
ERM Copy
DBCC Server
ERM SQL
Server Agent
Copy of
SCM Master
Active DB
HL7 Interface Server HL7 Interface Server
Executive Server Manager Server
MSMQ XAAPPS2P
ERM Client
XAHL71PA
XAHL7P
XAHL7CL1
SunriseXA Services
Notification, CDS and Order Generation Server Multum Server Report Server
XACOGNS1PA XAMULTUM1PA XAREPORT1P
XACOGNSP XAMULTUMP XAREPORTP
XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1

34. Step by Step SQL DB Copy (ERM)

35. Time to Complete DBCC
(ERM Way)
• Replication of PROD DB and Copy over to the DBCC
Server via timefinder ~ 40 min
• Database in Quiesce State ~ 20 Sec
• Mount the replicas and open DB ~ 40 Sec
• Run DBCC on copy of PROD DB ~ 1 hr 40 min
• Total time spent before a corruption is detected ~ 2.5 Hrs
• Total time saved using ERM ~ 10 hrs

36. Benefits
• Faster Database copy
• Copy, restore and DBCC process were fully automated
• Very helpful during migration to SMC 4.5XA on new
Servers
• Standard process makes it easy for support staff to use it
• Could be setup as an alternative DR System

37. Disadvantages
• Cost
• Complex Setup
• Requires extra disks
• Version compatibility issue with Symmetrix micro code
• Physical replication as oppose to logical replication
• Only usable on EMC H/W
• Not portable

38. Conclusion
• Minimized time required for the DBBC operation
– From 12 hours to 2.5 hrs
• Faster detection of corruption (if any)
• Easy and frequent replication made possible
• Create copy of Production DB on-demand as per request
from our analysts to aid in their debugging

39. References
• SunriseXA Monitoring,Maintenance and Troubleshooting
Guide,Course2052,April2003 by Eclipsys Corporation
• EMC Replication Manager V4.0.0 Product Guide P/N 300-003-099
REV A01 by Eclipsys Corporation, Hopkinton, MA 01748-9103,
2006
• EMC TimeFinder Integration with Microsoft SQL Server2000- Using
TimeFinder to Build a Highly Available Microsoft SQL Server 2000
Environment, Eclipsys Corporation, April 2002
Contact Information
Ziaul Mannan : Ziaul.Mannan@ynhh.org
Howard Goldberg: Howard.Goldberg@ynhh.org

40. THANK YOU !
Questions?