In the event of a disaster, you need to be able to recover lost data quickly to ensure business continuity. For critical applications, keeping your time to recover and data loss to a minimum as well as optimizing your overall capital expense can be challenging. This session presents AWS features and services along with Disaster Recovery architectures that you can leverage when building highly available and disaster resilient applications. We will provide recommendations on how to improve your Disaster Recovery plan and discuss example scenarios showing how to recover from a disaster.

1. © 2017 Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sajee Mathew, Principal Solutions Architect
August 14, 2017
Deploying a Disaster Recovery Site
on AWS
Minimal Cost with Maximum Efficiency

2. What do I expect you to know?
• 300-level talk
• Have some prior experience with AWS
• Understand/know basic terms, like Amazon EBS, Amazon S3, and
Amazon EC2
"I get it and want to learn more."
• Sessions will dive deeper into the topic. Presenters assume the
audience is familiar with the topic but may or may not have direct
experience implementing a similar solution. Code may be shared but
will not be the primary focus of the session.

3. We are all historians
…even if you don’t know it.

7. "Shut up, shut up! I am
busy…”
- Jack Phillips, RMS Titanic telegraph operator

8. Recovery point

10. Disaster
Recovery point
Data loss

11. "SOS, SOS,CQD,CQD
MGY. We are sinking
fast - passengers are
being put into boats..
MGY”
- Titanic last radio call

12. Disaster
Recovery point Recovery time
Data loss Down time

13. Lessons learned?
Failure is not one thing…

14. Disaster recovery as
an event
aka watch this...

16. Pi
MQ-7

17. AWS cloudCorporate data center
AWS
Lambda
AWS
CloudFormation
Amazon
Kinesis
Amazon
Route 53

18. AWS cloud
Corporate data center
CloudFormation
Amazon
Route 53
Auto Scaling group
Availability Zone #1
Security group
security group
Root volume
Data volume
EC2 instance
Web app
server

19. What did it cost?
MQ-7 sensor R-PI 3 Route 53 S3
$3.81 $38.83 $0.53/zone $ 0.023/GB
Service Cost
S3 bootstrap
script repo
$.023
Hardware $42.64
Route 53 DNS $0.53
Total $43.19
Shown: us-west-2

20. How can I do this?

21. What are we really planning for?

25. Techniques:
Back up and disaster recovery

26. Pilot light

27. Subordinate
database
server
Pilot light–prep
www.example.com
Data mirroring
replication
Not running
Pilot light system
Reverse
proxy/
caching
server
Data
volume
Application
server
Corporate data center
Reverse
proxy/
caching
server
Application
server
Master
Database
server

28. Database
server
Pilot light–recovery
www.example.com
Start in minutes
Add additional
capacity,
if needed
Reverse
proxy/
caching
server
Data
volume
Application
server
Corporate data center
Reverse
proxy/
caching
server
Application
server
Master
Database
server

29. Warm standby

30. Warm standby–prep
Mirroring /replication
Application
data source
cut over
Elastic
load
balancerActive
Not active for
production traffic
Route 53
www.example.com
Scaled down
standbyCorporate data center
Data
volume
Application
server
Subordinate
database
server
Reverse
proxy/
caching
server
AWS region
Reverse
proxy/
caching
server
Application
server
Master
Database
server

31. Warm standby–recovery
Elastic
load
balancer
Active
Route 53
www.example.com
Scaled-up
production
Corporate data center
Data
volume
Application
server
Database
server
Reverse
proxy/
caching
server
AWS region
Reverse
proxy/
caching
server
Application
Server
Master
Database
server

32. Hot site

33. Hot site–prep
Mirroring /replication
Application
data source
cut over
Elastic
load
balancerActive
Route 53
www.example.com
Corporate data center
Data
volume
Application
server
Subordinate
database
server
Reverse
proxy/
caching
server
AWS region
Reverse
proxy/
caching
server
Application
server
Master
Database
server
Active

34. Hot site–recovery
Elastic
load
balancer
Route 53
www.example.com
Corporate data center
Data
volume
Application
server
Database
server
Reverse
proxy/
caching
server
AWS region
Reverse
proxy/
caching
server
Application
server
Master
Database
server
Active
Scaled up
for production
use

35. Use case 1
Basic backup and recovery use case

36. AWS CLI-based backup
$ aws s3 sync /backups s3://mybucket
;Back up and sync the backup folder
$ aws s3 sync /backups s3://mybucket --delete
;Like the preceding, but now delete files not present
$ aws s3 sync /backups s3://mybucket --delete –storage-
class STANDARD_IA
;Like the preceding, but now leverages Infrequent access

37. What does it look like?
Amazon
S3
Amazon
Glacier
S3 bucket
Remote location
/mybucket
S3
STANDARD_IA
1
2
Lifecycle
policy

38. What does a recovery look like?
Remote location
2
AWS DR Region
Amazon
EC2
Amazon
S3
Amazon
Glacier
S3 bucket
/mybucket
S3
STANDARD_IA
1
Lifecycle
policy

39. What did it cost?
S3
STANDARD_IA
S3 Amazon Glacier
$ 0.0125/GB $ 0.023/GB $ 0.004/GB
Service Cost
S3 10 GB images $0.23
S3–IA 100 GB of data $1.25
Amazon Glacier–1 TB archives $4.10
Total $5.58/mo
Shown: us-west-2

40. Use case 2
Large data archive and recovery

41. Large data set
AWS cloud
Corporate data center
NGS
On-premises
compute /cluster
Sequence data
Flowcell-ID
Amazon Glacier
2 3
AWS
Snowball device
AWS CLI
1
AWS Snowball

42. Large data set
AWS cloud
Corporate data center
NGS
On-premises
Compute / cluster
Virtual server
iSCSI
Cached
volume
1
2
virtual tape
library
AWS
Storage Gateway
Amazon
Glacier
Amazon
S3

43. Large data set
Corporate data center
NGS
On-premises
Compute / cluster
AWS
File Gateway
Virtual server
NFS
AWS us-west-2
Amazon
S3
S3
bucket
Lifecycle
policy
AWS us-east-1
Amazon
S3
S3
bucket

44. Large data set–recovery
AWS DR Region
Sequence data
Flowcell-ID
Amazon Glacier
Corporate DR facility
Server infrastructure
1
AWS Snowball
S3 VPC endpoint
AWS DR Region
2
Amazon
EC2

45. Large data set–recovery
AWS DR Region
Corporate data center
NGS
On-premises
Compute / cluster
AWS
Storage Gateway
Virtual server
ISCI
Cached
volume
1
Amazon
Glacier
Amazon
S3
instance
2
AWS DR Region
snapshot
virtual tape
library
AWS DR Region
instance
AMI
Amazon EBS

46. What did it cost? – scenario gateways
File Storage Volume Storage VTL - Archived
$ 0.023/GB $0.023/GB $ 0.004/GB
Service Cost
File Gateway - 10 TB $235.40
Storage Gateway – 32 TB $736
Storage Gateway VTL – 250TB $1,000
Total $1,971.40
Shown: us-west-2

47. What did it cost? – scenario SnowBall
S3 Snowball Amazon Glacier
$ 0.023/GB $250/80TB $ 0.004/GB
Service Cost
AWS SnowBall * 13 $3,250
Amazon Glacier archive 1 PB $4194.31
Total $ 7,444.31
$4,194.31 /mo
Shown: us-west-2

48. Massive data move

49. Use case 3
Onsite virtualization replication,
backup, and failover

50. Corporate data center AWS cloud
Storage
Gateway
iSCSI from
Storage
Gateway to
ESX Hosts
VMFS
VMware ESX VMware ESXi
Replication
Appliance
Onsite VM replication/convert
Stored
volume

51. Corporate data center AWS cloud
iSCSI from
AWS Storage
Gateway to
ESX Hosts
VMFS
VMware ESX VMware ESXi
AMI
EBS
VM
Import
Onsite VM replication/convert – recovery
Storage
Gateway
Stored
volume
Amazon
S3
Replication
Appliance

52. What did it cost?
S3 Storage Gateway
$ 0.023/GB $0.00
Service Cost
Storage Gateway archive (32 TB) $753.67
Storage Gateway VM cost $0
Total $753.67/mo
Shown: us-west-2

53. VMware Cloud on AWS

54. Deploy and consume native AWS services
Customer VPC VMware Cloud VPC
Customer Data Center
AWS Direct
Connect
VMware Cloud
Endpoints
VPC subnet VPC subnet
VPC subnet
Private Managed
AWS ServicesCustomer Instances
vSphere Environment
VMware
Endpoints
Non-vSphere Environment
ESXi
ESXi
Amazon EC2
Internet
Private
VIF
Public
VIF
Regional AWS Services
AWS
Lambda
Amazon
S3
Amazon
CloudFront
Etc…

55. Use case 4
Multisite replication and failover

56. Corporate data center
AWS
Direct Connect
VPN
us-east-1
us-west-2
Server
Server
Availability Zone Availability Zone
Failback
Server
Multisite failover
customer
gateway
users Equinix DA1

57. Corporate data center
Direct Connect
VPN
us-east-1
us-west-2
Failback
AWS
CloudFormation
Server
Availability Zone Availability Zone
Server
Multisite failover
Server
users
customer
gateway
Equinix SE2

58. What did it cost? (30 days)–remote site
VPC VPN EC2 *
(m4.xlarge)
1 Gbps
Direct Connect
EBS Region data
transfer fee
$ 0.05/hr $ 0.215 $ 0.30/hr $ 0.10/GB $ 0.02/GB
Service Cost
1 GBps Direct Connect $216
VPN Fallback Connection $36
(2) EC2 instances $362.88
(2) EBS 60 GB volumes $12
(1) AMI copy to us-west-2 $1.20
Total $628.08*us-west-2, Amazon Linux AMI

59. Use case 5
Knowledge worker DR site

61. WorkSpaces for worker DR
Customer VPC
Active
Directory
Corp servers
Direct Connect
or VPN
Customer
corporate
network
Customer AD
RO/Replica
EC2
corporate
servers
MFA (RADIUS) (Optional)
Archives
Amazon S3

62. Internet
Users
Standalone
Network
Customer VPC
AD
RO / Replica
Public AWS
endpoint
Streaming
Gateway
9
Amazon WorkSpaces
(network entry point)
Amazon EC2
corporate
servers
Customer
corporate network
Amazon S3
WorkSpaces for worker DR

63. What did it cost? – 30 days
WorkSpaces
(Standard)
EC2 *
(m4.xlarge)
EBS
(GP SSD)
S3 VPC VPN
$ 35 /
Workspace
$ 0.404 / Hour $ 0.10 $ 0.023 / GB $ 0.05 / Hour
Service Cost
WorkSpaces for 25 users $875
(2) AD EC2 + 100GB EBS $608.46
VPC VPN $36
S3 file backups (500GB) $15
(1) Share point W/ 2TB EBS $492.73
Total $2027.19/MoShown: us-west-2, Windows Standard AMI

64. Use case 6
Windows SMB backed by Storage Gateway

65. Data center SMB server with SGW backend
SMB hosted onsite, blocks stored durably in Amazon S3
Private data center
Storage
Gateway VM
us-west-2
Storage
Gateway
Volume
Windows
Clients
Windows Server
HTTPSiSCSISMB

66. High Availability Windows Server Storage
Local HA with remote DR
us-east-1
Data Center
Windows
Clients Storage
Gateway VMMicrosoft DFS
HTTPSiSCSI
HTTPSiSCSI
SMB
Storage Gateway Service
us-west-1
Microsoft DFS
HTTPSiSCSI
Storage Gateway Service

67. CLOUD LIGHTNING ROUND!!!

68. Use case 7a
All in on AWS : EC2 backup

69. “ I want to copy my EC2
instance data via EBS
snapshot…”

70. “… but not lose data.”

71. http://amzn.to/2nAz7n6

72. Leverage the power of systems manager
#!/bin/bash
mysql -u backup -h localhost -e 'FLUSH TABLES WITH READ LOCK;’
sync
for target in $(findmnt -nlo TARGET -t ext4); do fsfreeze -f $target; done
instance=`curl -s http://169.254.169.254/latest/meta-data/instance-id` region=`curl -s
169.254.169.254/latest/meta-data/placement/availability-zone` region=${region::-1} volumes=`aws
ec2 describe-instance-attribute --instance-id $instance --attribute blockDeviceMapping --output
text --query BlockDeviceMappings[*].Ebs.VolumeId --region $region`
for volume in $(echo $volumes | tr " " "n") do aws ec2 create-snapshot --volume-id $volume --
description 'Consistent snapshot of MySQL on Amazon Linux' --region $region > /dev/null 2>&1
done
for target in $(findmnt -nlo TARGET -t ext4);
do fsfreeze -u $target; done
mysql -u backup -h localhost -e 'UNLOCK TABLES;'

73. Use case 7b
All in on AWS : EC2 instance crashed!

74. Amazon EC2
Auto Recovery
Set your failed check threshold
Choose 1-minute period
and statistic minimum
Choose recover action
Metric = StatusCheckFailed_System

75. Amazon EC2
Auto Reboot
Choose reboot action
Metric = StatusCheckFailed_Instance

76. Use case 7c
All in on AWS : S3 data loss!

77. “I’m worried about losing data from S3!”
• S3 is built for 11 9’s of durability
• If you store 10,000 objects, you can on average expect to
incur a loss of a single object once every 10,000,000 years.
• S3 supports cross region replication
• S3 supports versioning
• S3 supports MFA delete
• IAM roles can also be used to limit access to S3

78. Use case 7d
All in on AWS : RDS replication

79. RDS database
• RDS automatic backup/snapshots
• RDS supports cross region read replicas for MySQL,
PostgreSQL, Aurora DB, or MariaDB.

80. Use case 7e
All in on AWS : Database migration service

81. AWS Database Migration Service
• Continuous or one-time DB replication to EC2 or RDS
• Leverage DMS to replicate your database to AWS or
even change your schema from one engine to another.

82. Conclusion

83. Lessons from history
Plan for more than just what
you expect to happen.

84. Lessons from history
Test your execution plan
before you think you can
implement it.

85. Lessons from history
Knowledge is critical. Know
how to interpret an alarm on
events.

86. Words of advice
People generally don’t do well under pressure
• Automate as much as you can
• Tabletop exercises can really help you understand roles
and responsibility
• Not all services have to require the same RTO/RPO
• Seriously, automate as much as you can ahead of time.
• If you don’t have a run book, it’s time to make one
• If you have one, have you tested it?

87. Remember to complete
your evaluations!

88. Thank you!