63. @theburningmonk theburningmonk.com
The Saga pattern
Begin transaction
Start book hotel request
End book hotel request
Start book flight request
End book flight request
Start book car rental request
End book car rental request
End transaction
104. @theburningmonk theburningmonk.com
circuit breaker pattern
When circuit is open, fail fast
but, allow 1 request through every Y mins
If request succeeds, close the circuit
After X consecutive timeouts, trip the circuit
119. @theburningmonk theburningmonk.com
Lambda autoscaling
Burst concurrency limits:
3000 – US West (Oregon), US East (N.
Virginia), Europe (Ireland), 1000 – Asia Pacific
(Tokyo), Europe (Frankfurt), 500 – Other
Regions
Burst: 500 new instances / each minute
120. @theburningmonk theburningmonk.com
Lambda autoscaling
Burst concurrency limits:
3000 – US West (Oregon), US East (N.
Virginia), Europe (Ireland), 1000 – Asia Pacific
(Tokyo), Europe (Frankfurt), 500 – Other
Regions
Burst: 500 new instances / each minute
Standard burst concurrency limits when over the
provisioned capacity
121. @theburningmonk theburningmonk.com
Lambda autoscaling
Burst concurrency limits:
3000 – US West (Oregon), US East (N.
Virginia), Europe (Ireland), 1000 – Asia Pacific
(Tokyo), Europe (Frankfurt), 500 – Other
Regions
Burst: 500 new instances / each minute
Adjustable provisioned capacity based on
CloudWatch metrics
Standard burst concurrency limits when over the
provisioned capacity
122. @theburningmonk theburningmonk.com
Lambda limitations & throttling
Concurrent executions: 1000*
Timeout: 15 minutes
Burst concurrency: 500 - 3000
Burst: 500 new instances / minute
* Can be increased with support ticket
123. @theburningmonk theburningmonk.com
Lambda limitations & throttling
good for spikey
traffic, up to a point
Concurrent executions: 1000*
Timeout: 15 minutes
Burst concurrency: 500 - 3000
Burst: 500 new instances / minute
* Can be increased with support ticket
150. @theburningmonk theburningmonk.com
“the discipline of experimenting on a system in order to build confidence in the
system’s capability to withstand turbulent conditions in production”
principlesofchaos.org
163. @theburningmonk theburningmonk.com
“Corporation X lost millions due to a
chaos experiment went wrong and
destroyed key infrastructure,
resulting in hours of downtime and
unrecoverable data loss.”
169. CONTAINMENT
run experiments during office hours
let others know what you’re doing, no surprises
avoid important dates
make the smallest change possible
170. CONTAINMENT
run experiments during office hours
let others know what you’re doing, no surprises
avoid important dates
make the smallest change possible
have a rollback plan before you start
172. by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361
173. @theburningmonk theburningmonk.com
chaos monkey kills an
EC2 instance
latency monkey induces
artificial delay in APIs
chaos gorilla kills an AWS
Availability Zone
chaos kong kills an entire
AWS region
187. @theburningmonk theburningmonk.com
TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms
delay = Math.random() * (Math.pow(2, retryCount) * base)
this is Marc Brooker’s
fav formula!
196. @theburningmonk theburningmonk.com
TIL: most HTTP client libraries have default timeout of 60s.
API Gateway has an integration timeout of 29s.
Most Lambda functions default to timeout of 3-6s.
217. @theburningmonk theburningmonk.com
circuit breaker pattern
When circuit is open, fail fast
but, allow 1 request through every Y mins
If request succeeds, close the circuit
After X consecutive timeouts, trip the circuit
220. @theburningmonk theburningmonk.com
Lambda autoscaling
Burst concurrency limits:
3000 – US West (Oregon), US East (N.
Virginia), Europe (Ireland), 1000 – Asia Pacific
(Tokyo), Europe (Frankfurt), 500 – Other
Regions
Burst: 500 new instances / each minute
Adjustable provisioned capacity based on
CloudWatch metrics
Standard burst concurrency limits when over the
provisioned capacity
223. @theburningmonk theburningmonk.com
“the discipline of experimenting on a system in order to build confidence in the
system’s capability to withstand turbulent conditions in production”
principlesofchaos.org
224. by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361
225. by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361