The talks covers various design patterns to make services more resilient. For more detail, please also follow https://twitter.com/adhorn who created the deck.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
B E R L I N
25.10.19
Resiliency and availability design
patterns for the cloud
Cobus Bernard
Senior Technical Evangelist
Amazon Web Services
@cobusbernard
cobusbernard
cobusbernard
B A R 3

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Can you guess whatwillhappen?

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Distributed Systems are hard

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failures areagivenand
everythingwilleventuallyfail
over time.
Werner Vogels
CTO – Amazon.com
“ “

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Resiliency:Abilityfor asystemtohandle and
eventuallyrecover from unexpected conditions
conditions

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Partialfailure mode

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why do we build resilient software systems?

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thecostof downtime
Annual Fortune
1000 application
downtime costs
(IDC)
$1.25 to
$2.5B
Average cost of a
data breach
(Ponemon
Institute)
$3.6M
Cost/hr of a
critical
application
failure (IDC)
$500K
to $1M
Average cost/hr
of downtime
(Ponemon
Institute)
$474K
Average cost per
lost or stolen
record (Ponemon
Institute)
$141

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do we build resilient software systems?

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
People
Application
Network & Data
Infrastructure

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk aboutAvailability

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Systemavailability
Availability =
Normal Operation Time
Total Time
MTBF**
MTBF** + MTTR*
=
* Mean Time To Repair (MTTR)
**Mean Time Between Failure (MTBF)

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Reading homework

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availabilityinparallel
A = 1 – (1 – Ax)2
Part X
Part X

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availabilityinparallel
Component Availability Downtime
X 99% (2-nines) 3 days 15 hours
Two X in parallel 99.99% (4-nines) 52 minutes
Three X in parallel 99.9999% (6-nines) 31 seconds

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Componentredundancyincreases availability
significantly!

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Global Infrastructure
• 22 Regions with 69 Availability Zones
• 3 Regions coming soon: Cape Town
Jakarta and Milan
• 100Gbps redundant network
• 99.99% SLA

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fully-scaledAvailabilityZone

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Highlyredundant regional network

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Region and availabilityzones
Region
Availability zone a Availability zone b Availability zone c
data center
data center
data center
1 or more data centers per AZ
2 or more AZs per region (new regions min 3)
data center
data center
data center
data center
data center
data center

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How about a global architecture?

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Once upon a time …
Origin

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
And Now …
Origin
~300ms

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Origin
Improve latency for end-users
Origin

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Improve availabilityand disasterrecovery
Applications in
US West
Applications in
US East
Users from
San Francisco
Users from
New York
Service 1
Service 2
Service 3
Service 4
Service 1
Service 2
Service 3
Service 4

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
So should we go for a global architecture?

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Perfect your regional architecture first!

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about Multi-AZ

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
new master
Elastic Load
Balancing (ELB)

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi-AZ architecture
• Enables fault-tolerant applications
• AWS regional services designed to
withstand AZ failures
• Leveraged by AWS regional
services such as Amazon S3,
Amazon DynamoDB, Amazon
Aurora, Amazon ELBs, etc.
Region
Availability zone a Availability zone b Availability zone c
Instances Instances Instances
DB Instance DB instance
standby
Elastic Load
Balancing (ELB)

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about auto scaling

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto-Scaling
FixedVariable

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Availability zone 1
Auto Scaling group
AWS Region
Availability zone 2
Auto-scaling for self-healing
Elastic Load
Balancing (ELB)
X

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about the AWS responsibility
models

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWSoperationalresponsibilitymodels
On-Premises Cloud
Less More
Compute Virtual Machine
EC2 Elastic Beanstalk AWS LambdaFargate
Databases MySQL MySQL on EC2
RDS MySQL RDS Aurora Aurora Serverless DynamoDB
Storage Storage
S3
Messaging ESBs
Amazon MQ Kinesis SQS / SNS
Analytics
Hadoop Hadoop on EC2 EMR Elasticsearch Service Athena
Firehose

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about databases

40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Common resiliencyissueswithDatabases??
REPLICATION BACKUPSSCALING

41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
AZ 1
AZ 2
AZ 3
Network
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
Storage
Node
Leader
PutItem

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
AZ 1
AZ 2
AZ 3
RR RR RR RR RR
RR RR RR RR RR
RR RR RR RR RR
Storage
Node
Leader
GetItem
Network

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon.com, Nike, Netflix, Duolingo, Lyft, Airbnb, Samsung,
Toyota, and Capital One depend on the scale and performance of
DynamoDB to support their workloads.
10 trillion requests
per day
20 million
requests per second

45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Purpose-built databases
Relational Key-value Document In-memory Graph Time-series Ledger
DynamoDB NeptuneAmazon RDS
Aurora CommercialCommunity
Timestream QLDBElastiCacheDocumentDB

46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Read-Writeseparation
Master Read Replica Read Replica Read Replica
Instance InstanceInstance

47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DatabaseFederation
Users
DB
Products
DB
Master
(Read) Replica
Master
(Read) Replica
Instance InstanceInstance

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
DatabaseSharding User ShardID
002345 A
002346 B
002347 C
002348 B
002349 A
CBA
Master
(Read) Replica
Master
(Read) Replica
Master
(Read) Replica
Instance InstanceInstance

49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about backups

50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Backup service

52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PreventingAccidentalTableDeletion
https://aws.amazon.com/blogs/database/preventing-accidental-table-deletion-in-dynamodb/

53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PreventingAccidentalTableDeletion (sql)

54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Practice and testrecoveryfrom your backups!!

55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about timeouts, backoff &
retries!

56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Users
App
DB
Conn
Pool
INSERT
INSERT
INSERT
INSERT
What happens if the DB “slows down”?
Timeout client side Timeout backend side ??

57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User 1
App
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = default = Infinite
Retry INSERT
Retry INSERT
ERROR: Failed to get connection from pool
Retry

58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.timeout

59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html

61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
@timeout_decorator.timeout(5, timeout_exception=StopIteration)
def timed_get(url):
return requests.get(url)
https://pypi.org/project/timeout-decorator/

62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How elsecould wehaveprevented theerror?
User 1
DB
Conn
Pool
INSERT
Retry INSERT
Retry INSERT
Retry
ERROR: Failed to get connection from pool

63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User 1
DB
Conn
Pool
INSERT
Timeout client side = 10s Timeout backend side = 10s
Wait 2s before Retry
INSERT
INSERT
Wait 4s before Retry
Wait 8s before Retry
Wait 16s before Retry
Backing off betweenretries
Releasing connectionsBackoff

64. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
No jitter With jitter
https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
SimpleExponentialBackoffisnotenough:AddJitter

65. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example: add jitter0-1000ms
def get_item(self, url, n=1):
MAX_TRIES = 12
try:
res = requests.get(url)
except:
if n > MAX_TRIES:
return None
n += 1
time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0))
return self.get_item(url, n)
else:
return res

66. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Idempotent operation
No additional effect if it is called more than
once with the same input parameters.

67. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about health checking!

68. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Auto Scaling group
Service A
Availability zone 1
Auto Scaling group
AWS Region
Service A
Availability zone 2
Service BService B
database Email
Probing for health
Cluster

69. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shallowhealthcheck
Instance
Cache node
Email
database
Cluster
Are you healthy?
yes

70. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Shallowhealthcheck
Instance
Cache node
Email
database
Cluster
Are you healthy?
yes

71. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep healthcheck
Instance
Cache node
Email
database
Cluster
Are you healthy?
yes
Are you healthy?
yes
yes
yes
yes

72. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep healthcheck
Instance
Cache node
Email
database
Cluster
Are you healthy?
no
Are you healthy?
no
yes
yes
yes

73. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prioritize shallow health checks during
hard times.
Cache.

74. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk about load shedding.

75. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

76. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

77. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cheaply reject excess work

78. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

79. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Be careful when selecting the right metric

80. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Don’tbeoverlyoptimisticandtakeonmorethanyoucan.
Findanoperationalmetrictorejectwhatyoucannottakein.
Favorcachedandstaticcontent
PrioritizeELBhealthcheck(shallow)pings
Inanoverloadsituationyouhavepreciousresources,donotletany
ofitgotowaste.
Load Shedding

81. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Let’s talk aboutresiliency (chaos) engineering

82. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fire Drills

83. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GameDay atAmazon
CreatingResiliencyThroughDestruction
https://www.youtube.com/watch?v=zoz0ZjfrQ9s

84. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Chaosengineering
https://github.com/Netflix/SimianArmy

85. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Chaos Engineeringis the discipline of
experimentingon a distributedsystem
in orderto buildconfidence in the system’s
capabilitytowithstand turbulentconditionsin
production.”
http://principlesofchaos.org

86. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Failureinjection
• Start small & build confidence
• Application level
• Host failure
• Resource attacks (CPU, memory, …)
• Network attacks (dependencies, latency, …)
• Region attacks
• “Paul” attack
https://www.gremlin.comhttps://github.com/Netflix/SimianArmy https://chaostoolkit.org

87. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
STEADY
STATE
HYPOTHESIS
RUN
EXPERIMENT
VERIFY
FIX!
PhasesofChaosEngineering

88. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
https://aws.amazon.com/wellarchitected

89. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
@cobusbernard
cobusbernard
cobusbernard