Chaos engineering is a discipline that focuses on improving system resilience through experiments that expose the inherent chaos and failure modes in our system, in a controlled fashion, before these failure modes manifest themselves like a wildfire in production and impact our users. Netflix is undoubtedly the leader in this field, but much of the publicised tools and articles focus on killing EC2 instances, and the efforts in the serverless community has been largely limited to moving those tools into AWS Lambda functions. But how can we apply the same principles of chaos to a serverless architecture built around AWS Lambda functions? These serverless architectures have more inherent chaos and complexity than their serverful counterparts, and, we have less control over their runtime behaviour. In short, there are far more unknown unknowns with these systems. Can we adapt existing practices to expose the inherent chaos in these systems? What are the limitations and new challenges that we need to consider?

1. APPLYING PRINCIPLES
to SERVERLESSt
a
b
chaos engineering
of
A
E
S
of

2. what is chaos engineering?

3. Chaos Engineering is the discipline of experimenting on a distributed system
in order to build confidence in the system’s capability
to withstand turbulent conditions in production.
- principlesofchaos.org

4. history of Smallpox
est. 400K deaths per year in 18th Century Europe.
earliest evidence of disease in 3rd Century BC Egyptian Mummy

5. history of Smallpox
est. 400K deaths per year in 18th Century Europe.
earliest evidence of disease in 3rd Century BC Egyptian Mummy
1798
first vaccine developed
Edward Jenner

6. 1798
first vaccine developed
1980
history of Smallpox
Edward Jenner
WHO certified
global eradication
est. 400K deaths per year in 18th Century Europe.
earliest evidence of disease in 3rd Century BC Egyptian Mummy

8. Vaccination is the most effective method of
preventing infectious diseases

9. stimulates the immune system to recognize
and destroy the disease before contracting
the disease for real

10. Chaos Engineering
controlled experiments to help us learn about
our system’s behaviour and build confidence
in its ability to withstand turbulent conditions

11. chaos engineering is the vaccine to frailties in
modern software

12. Yan Cui
http://theburningmonk.com
@theburningmonk
Principal Engineer @

13. Yan Cui
http://theburningmonk.com
@theburningmonk
Principal Engineer @

14. “Netflix for sports”
offices in London, Leeds, Katowice and Amsterdam

18. available in 7 countries, 30+ platforms

19. ~1,000,000 concurrent viewers

20. “Netflix for sports”
offices in London, Leeds, Katowice and Amsterdam
We’re hiring! Visit
engineering.dazn.com to
learn more.
follow @DAZN_ngnrs for
updates about the
engineering team.
WE’RE HIRING!

21. chaos engineering has an
image problem

26. Why did you break
production?

27. Because I can!

28. it’s about building confidence,
NOT breaking things

30. http://principlesofchaos.org

31. STEP 1. define “Steady State”
aka. what does normal, working
condition looks like?

32. this is not a
steady state

33. STEP 2.
hypothesize steady state will
continue in both control group
& the experiment group
ie. you should have a reasonable degree of
confidence the system would handle the failure
before you proceed with the experiment

34. explore unknown unknowns
away from production

35. treat production with the
care it deserves

36. the goal is NOT,
to actually hurt production

37. If you know the system would break,
and you did it anyway…
then it’s NOT a chaos experiment.
It’s called being IRRESPONSIBLE.

39. STEP 3.
inject realistic failures
e.g. server crash, network error,
HD malfunction, etc.

40. https://github.com/Netflix/SimianArmy

41. https://github.com/Netflix/SimianArmy http://oreil.ly/2tZU1Sn

43. STEP 4.
disprove hypothesis
i.e. look for difference with steady state

44. if a WEAkNESS is uncovered,
IMPROVE it before the behaviour
manifests in the system at large

45. Chaos Engineering
controlled experiments to help us learn about
our system’s behaviour and build confidence
in its ability to withstand turbulent conditions

46. Chaos Engineering
controlled experiments to help us learn about
our system’s behaviour and build confidence
in its ability to withstand turbulent conditions

47. containment and blast radius should
be front and centre of your thinking

48. communication

49. ensure everyone knows what you’re doing

50. ensure everyone knows what you’re doing
NO surprises!

51. communication
Timing

52. run experiments during office hours

53. AVOID important dates

54. communication
Timing
contain Blast radius

55. smallest change that allows
you to detect a signal that
steady state is disrupted

56. rollback at the first sign of
TROUBLE!

57. communication
Timing
contain Blast radius

58. don’t try to run before you
know how to walk.

59. by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

60. chaos monkey kills an
EC2 instance
latency monkey induces
artificial delay in APIs
chaos gorilla kills an
AWS Availability Zone
chaos kong kills an
entire AWS region

63. there is no server…

64. there is no server…
that you can kill

67. there are more inherent chaos and
complexity in a Serverless architecture

68. smaller units of deployment
but A LOT more of them!

69. more difficult to harden
around boundaries
serverful
serverless

70. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES

71. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES
more intermediary services,
and greater variety too

72. ?
SNS
Kinesis
CloudWatch
Events
CloudWatch
LogsIoT
DynamoDB
S3 SES
more intermediary services,
and greater variety too
each with its own set of
failure modes

73. serverful
serverless
more configurations,
more opportunities for misconfiguration

74. more unknown failure modes in
infrastructure that we don’t control

75. often there’s little we can do when an
outage occurs in the platform

76. improperly tuned timeouts

77. missing error handling

78. missing fallback when downstream is unavailable

79. LATENCY INJECTION

80. STEP 1. define “Steady State”
aka. what does normal, working
condition looks like?

81. what metrics do you monitor?

82. 9X-percentile latency
error count
yield (% of requests completed)
harvest (completeness of results)

83. STEP 2.
hypothesize steady state will
continue in both control group
& the experiment group
ie. you should have a reasonable degree of
confidence the system would handle the failure
before you proceed with the experiment

84. API Gateway

85. consider the effect of cold-starts
& API Gateway overhead

86. use short timeout for API calls

87. the goal of a timeout strategy is to give HTTP
requests the best chance to succeed,
provided that doing so does not cause the
calling function itself to err

88. fixed timeout are tricky to get right…

89. fixed timeout are tricky to get right…
too short and you don’t
give requests the best
chance to succeed

90. fixed timeout are tricky to get right…
too long and you run the
risk of letting the request
timeout the calling function

91. and it gets worse when you make multiple
API calls in one function…

94. set the request timeout based on the
amount of invocation time left

99. log the timeout incident with
as much context as possible
e.g. timeout value, correlation IDs,
request object, …

100. report custom metrics

102. trade harvest (completeness of response)
for yield (availability of response)

105. be mindful when you sacrifice precision for
availability, user experience is the king

106. STEP 3.
inject realistic failures
e.g. server crash, network error,
HD malfunction, etc.

107. where to inject latency?

108. hypothesis:
function has appropriate timeout on its HTTP
communications and can degrade gracefully
when these requests time out

110. should also be applied to 3rd parties
services we depend on, e.g. DynamoDB

112. what’s the blast radius?

113. http client
public-api-a
http client
public-api-b
internal-api

114. hypothesis:
all functions have appropriate timeout on
their HTTP communications to this internal
API, and can degrade gracefully when
requests are timed out

117. large blast radius, risky..

119. could be effective when used away from
production environment, to weed out
weaknesses quickly

120. not priming developers to
build more resilient systems

121. development

122. development
production

123. Priming (psychology):
Priming is a technique whereby exposure to one
stimulus influences a response to a subsequent
stimulus, without conscious guidance or intention.
It is a technique in psychology used to train a
person's memory both in positive and negative ways.

126. make dev environments better resemble the
turbulent conditions you should realistically
expect your system to survive in production

127. hypothesis:
the client app has appropriate timeout on
their HTTP communication with the server,
and can degrade gracefully when requests
are timed out

131. STEP 4.
disprove hypothesis
i.e. look for difference with steady state

132. how to inject latency?

133. static weaver (e.g. AspectJ, PostSharp),
or dynamic proxies

134. https://theburningmonk.com/2015/04/design-for-latency-issues/

135. manually crafted wrapper library

141. configured in SSM Parameter Store

143. no injected latency

145. with injected latency

147. factory wrapper function
(think bluebird’s promisifyAll function)

153. ERROR INJECTION

154. common errors
HTTP 5xx
DynamoDB throughput exceeded
throttled Lambda executions

155. hypothesis:
Function has appropriate error handling on its
HTTP communications and can degrade
gracefully when downstream dependencies fail

157. hypothesis:
Function has appropriate error handling on
DynamoDB operations and can degrade gracefully
when DynamoDB throughputs are exceeded

159. Induce Lambda throttling by temporarily setting reserved concurrency.

160. failures are INEVITABLE

161. the only way to truly know your system’s
resilience against failures is to test it
through controlled experiments

165. vaccinate your serverless
architecture against failures

166. @theburningmonk
theburningmonk.com
github.com/theburningmonk

167. API Gateway and Kinesis
Authentication & authorisation (IAM, Cognito)
Testing
Running & Debugging functions locally
Log aggregation
Monitoring & Alerting
X-Ray
Correlation IDs
CI/CD
Performance and Cost optimisation
Error Handling
Configuration management
VPC
Security
Leading practices (API Gateway, Kinesis, Lambda)
Canary deployments
http://bit.ly/prod-ready-serverless
get 40% off
with: ytcui