A chaos experiment a day, keeping the outage away

•

0 gostou•385 visualizações

Presented at ServerlessDays Warsaw Recording: https://youtu.be/21HprKZQczs You might have heard about chaos engineering in the context of Netflix and Amazon, and how they kill EC2 servers in production at random to verify that their systems can stay up in the face of infrastructure failures. But did you know that the same ideas can be applied to serverless applications? Yes, despite not having access to the underlying servers, we can still apply principles of chaos engineering to uncover failure modes in our system (and there are plenty!) so we can build defence against them and make our serverless applications more robust and more resilient!

Tecnologia

one chaos experiment a day.
keep the outages away.
yan cui,@theburningmonk

@theburningmonk theburningmonk.com
“the discipline of experimenting on a system in order to build conﬁdence in the
system’s capability to withstand turbulent conditions in production”
principlesofchaos.org

@theburningmonk theburningmonk.com
microservices death stars circa 2015

@theburningmonk theburningmonk.com
“the capacity to recover quickly from difﬁculties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun

@theburningmonk theburningmonk.com
“the capacity to recover quickly from difﬁculties; toughness.”
resilience
/rɪˈzɪlɪəns/
noun
it’s not about
preventing failures!

@theburningmonk theburningmonk.com
“You don't choose the moment, the moment chooses you!
You only choose how prepared you are when it does.”
Fire Chief Mike Burtch

@theburningmonk theburningmonk.com
anything that can go wrong, will go wrong.
MURPHY’s LAW

@theburningmonk theburningmonk.com
identify weaknesses before they manifest in system-wide, aberrant behaviors
GOAL

@theburningmonk theburningmonk.com
learn about the system’s behavior by observing it during a controlled experiments
HOW

@theburningmonk theburningmonk.com
learn about the system’s behavior by observing it during a controlled experiments
HOW
game days
failure injection

Yan Cui
http://theburningmonk.com
@theburningmonk
AWS user for 10 years

Yan Cui
http://theburningmonk.com
@theburningmonk
http://bit.ly/yubl-serverless

Yan Cui
http://theburningmonk.com
@theburningmonk
Developer Advocate @

Yan Cui
http://theburningmonk.com
@theburningmonk
Independent Consultant
advisetraining delivery

“using serverless reduces the blast radius”
www.buzzsprout.com/877747/4615985

@theburningmonk theburningmonk.com
serverless improves resilience
as platform takes care of infrastructure failures

@theburningmonk theburningmonk.com
by Russ Miles @russmiles
source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

@theburningmonk theburningmonk.com
Shared Responsibility Model

@theburningmonk theburningmonk.com
chaos monkey kills an
EC2 instance
latency monkey induces
artiﬁcial delay in APIs
chaos gorilla kills an AWS
Availability Zone
chaos kong kills an entire
AWS region

@theburningmonk theburningmonk.com
there are no servers to kill!
SERVERLESS

@theburningmonk theburningmonk.com
improperly tuned timeouts

@theburningmonk theburningmonk.com
missing error handling

@theburningmonk theburningmonk.com
missing fallbacks

@theburningmonk theburningmonk.com
STEP 1.
deﬁne steady state
i.e. “what does normal look like”

@theburningmonk theburningmonk.com
STEP 2.
hypothesis that steady state continues in control and experimental group
e.g. “the system stays up if a server dies”

@theburningmonk theburningmonk.com
STEP 3.
inject realistic failures
e.g. “slow response from 3rd-party service”

@theburningmonk theburningmonk.com
STEP 4.
try to disprove hypothesis
i.e. “look for difference between control and experimental group”

@theburningmonk theburningmonk.com
latency inject latency to function invocation

@theburningmonk theburningmonk.com
“what if service X has elevated latency?”

@theburningmonk theburningmonk.com
API Gateway Lambda API Gateway Lambda

@theburningmonk theburningmonk.com
hypothesis: API would timeout and our try-catch
would handle it and return default response

@theburningmonk theburningmonk.com
result: function times out after 6s
(hypothesis is disproved)

@theburningmonk theburningmonk.com
API Gateway Lambda API Gateway Lambda
502
200

@theburningmonk theburningmonk.com
API Gateway Lambda API Gateway Lambda
3s timeout
6s timeout

@theburningmonk theburningmonk.com
API Gateway Lambda API Gateway Lambda
max 29s integration
max 15 mins timeout

@theburningmonk theburningmonk.com
and then there’s
cold starts…

@theburningmonk theburningmonk.com
TIL: most HTTP client libraries have default timeout of 60s.
API Gateway has an integration timeout of 29s.
Most Lambda functions default to timeout of 3-6s.
Don’t forget about the cold starts!

@theburningmonk theburningmonk.com
https://bit.ly/2Wvfort

@theburningmonk theburningmonk.com
outcome: a more resilient system

@theburningmonk theburningmonk.com
latency
exception
inject latency to function invocation
throws exception

@theburningmonk theburningmonk.com
latency
exception
statuscode
inject latency to function invocation
throws exception
return HTTP status code

@theburningmonk theburningmonk.com
latency
exception
statuscode
diskspace
inject latency to function invocation
throws exception
return HTTP status code
ﬁlls up /tmp directory

@theburningmonk theburningmonk.com
latency
exception
statuscode
diskspace
denylist
inject latency to function invocation
throws exception
return HTTP status code
ﬁlls up /tmp directory
looses network connectivity

@theburningmonk theburningmonk.com
“what if DynamoDB has an elevated error rate?”

@theburningmonk theburningmonk.com
API Gateway Lambda DynamoDB

@theburningmonk theburningmonk.com
hypothesis: the AWS SDK retries would handle it

@theburningmonk theburningmonk.com
TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms

@theburningmonk theburningmonk.com
TIL: the js DynamoDB client defaults to 10 retries
with base delay of 50ms
delay = Math.random() * (Math.pow(2, retryCount) * base)
this is Marc Brooker’s
fav formula!

@theburningmonk theburningmonk.com
action: set max retry count + fallback

https://theburningmonk.com/hire-me
AdviseTraining Delivery
“Fundamentally, Yan has improved our team by increasing our
ability to derive value from AWS and Lambda in particular.”
Nick Blair
Tech Lead

@theburningmonk
theburningmonk.com
github.com/theburningmonk

Mais conteúdo relacionado

Semelhante a A chaos experiment a day, keeping the outage away

Looking in from the outside, serverless seems so simple! And yet, many companies are struggling on their journey to serverless. In this talk, I highlight a number of mistakes companies are making when they adopt serverless. Recording: https://www.youtube.com/watch?v=4hzNqxo0yTA Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

Beware the potholes on the road to serverless

Yan Cui

Beware the potholes on the road to serverless

Yan Cui

Essential open source tools for serverless developers

Yan Cui

Patterns and practices for building resilient Serverless applications

Yan Cui

Serverless introduces a number of challenges to existing tools for observability, we need to adapt our practices to fit this new paradigm. In this talk, we will discuss how we can build observability into a serverless application. We will see how you can implement log aggregation, distributed tracing and correlation IDs through both synchronous as well as asynchronous events. Recording of this talk is available at https://www.youtube.com/watch?v=AMKUKKGPJhQ

How to build observability into a serverless application

Yan Cui

Learning by doing is hands-down the best way to learn. With serverless, the cost of experimenting (and learning!) in the cloud has reduced to zero, almost. In this talk, Yan Cui discusses the value of experimenting in the cloud and doing so in a tasteful manner (i.e. don't put your experiments into production!), and shares a number of interesting experiments that the serverless community has done, and some ideas and learning that we can take from them. Recording of the talk is available here: https://www.youtube.com/watch?v=oeomaQR1IT4

Dont try these at home

Yan Cui

Migrating existing monolith to serverless in 8 steps

Yan Cui

Stand back; I'm going to try Scientist!

Matt Eland

Mahesh talks about the buddha-nature of Erlang/OTP, pointing out how the various features of the language tie together into one seamless Fault Tolerant whole. Mahesh emphasizes that Erlang begins and ends with Fault Tolerance. Fault Tolerance is baked into the very genes of Erlang/OTP - something that ends up being amazingly useful when building any kind of system. Mahesh Paolini-Subramanya is the V.P. of R&D at Ubiquiti Networks - a manufacturer of disruptive technology platforms for emerging markets. He has spent the recent past building out Erlang-based massively concurrent Cloud Services and VoIP platforms. Mahesh was previously the CTO of Vocalocity after its merger with Aptela, where he was a founder and CTO.

Erlang - Because s**t Happens by Mahesh Paolini-Subramanya

Hakka Labs

In Green Custard's 19th Friday talk, Zoltan explores the subject of Chaos Engineering Topics covered: - What is chaos engineering? - Why would anyone do this? - Availability - Chaos engineering in practice - The four golden signals - Chaos engineering in practice - Chaos Monkey - The Simian Army Green Custard is a custom software development consultancy. To discover more about their work and the team visit www.green-custard.com.

Green Custard Friday Talk 19: Chaos Engineering

Green Custard

PagerDuty | OSCON 2016 Failure Testing

PagerDuty

testing for people who hate testing

Bram Vogelaar

Migrating existing monolith to serverless in 8 steps

Yan Cui

Refactoring a monolith to serverless can be intimidating, but there are discrete steps that you can take to simplify the process. In this talk, AWS Serverless Hero Yan Cui outlines 8 steps to successfully refactor your monolith and highlight key decision points such as language and tooling choices. Recording: https://www.youtube.com/watch?v=-HQw-UK2bHc Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

Migrating existing monolith to serverless in 8 steps

Yan Cui

Serverless observability - a hero's perspective

Yan Cui

Serverless gives you wings

Yan Cui

re:Invent 2019 Highly Available ECS Spot Architecture: Save 50%-90%

Tung Nguyen

Introduction of Plasma Chamber at EDCON 2019

YurikoNishijima

Sangam 18 - Database Development: Return of the SQL Jedi

Connor McDonald

Serverless applications are the epitome of highly distributed, microservices applications. Execution happens everywhere - both inside and outside the serverless compute environment. For example, your functions could be triggered by an external service, then execute some code within AWS Lambda, then send a request over to a database, which *then* requires AWS Lambda to perform an update in a second data store. You might be able to predict and design for certain troublesome issues but there are many, many more that you probably will not be able to easily plan for. How do you build a resilient system under these highly distributed circumstances? The answer is chaos engineering. Join us as we walk through: The unique challenges of building a highly resilient serverless app Why you need to design for problems you cannot predict and cannot easily test for How you can use chaos engineering to build a resilient serverless application How you can take advantage of out of the box and third-party observability solutions to measure the impact of chaos experiments.

Applying Chaos Engineering to build Resilient Serverless Applications - Emrah...

PROIDEA

Semelhante a A chaos experiment a day, keeping the outage away (20)

Beware the potholes on the road to serverless

Essential open source tools for serverless developers

Patterns and practices for building resilient Serverless applications

How to build observability into a serverless application

Dont try these at home

Migrating existing monolith to serverless in 8 steps

Stand back; I'm going to try Scientist!

Erlang - Because s**t Happens by Mahesh Paolini-Subramanya

Green Custard Friday Talk 19: Chaos Engineering

PagerDuty | OSCON 2016 Failure Testing

testing for people who hate testing

Migrating existing monolith to serverless in 8 steps

Serverless observability - a hero's perspective

Serverless gives you wings

re:Invent 2019 Highly Available ECS Spot Architecture: Save 50%-90%

Introduction of Plasma Chamber at EDCON 2019

Sangam 18 - Database Development: Return of the SQL Jedi

Applying Chaos Engineering to build Resilient Serverless Applications - Emrah...

Mais de Yan Cui

How to win the game of trade-offs

Yan Cui

At the heart of every event-driven architecture is a conduit for messages to flow through. AWS offers many services that can act as such conduit - EventBridge, SNS, SQS, Kinesis, DynamoDB streams, MSK, IOT Core and Amazon MQ just to name a few! These services have different characteristics and trade-offs around performance, scalability and cost. Picking the right service for your workload is not always easy. In this talk, let’s talk about how to pick the right messaging service to use in your event-driven architecture and play the game of trade-offs to your advantage.

How to choose the right messaging service

Yan Cui

How to choose the right messaging service for your workload

Yan Cui

Lambda and DynamoDB best practices

Yan Cui

Lessons from running AppSync in prod

Yan Cui

Learn all about AWS Step Functions and how to use them to model business workflows and ship customer values quickly. In this session, we will talk about what is Step Functions, how to model business workflows as state machines, real-world case studies, and design patterns. By the end of this webinar, you should have a good idea of where Step Functions fit into your application and why you should use them (and why not!) to model workflows instead of building a custom solution yourself.

How to ship customer value faster with step functions

Yan Cui

One of the key characteristics of serverless components is the pay-per-use pricing model. For example, with AWS Lambda, you don’t pay for the uptime of the underlying infrastructure but for the no. of invocations and how long your code actually runs for. This important characteristic removes the need for many premature micro-optimizations as your cost is always tightly linked to usage and minimizes waste. As a result, many applications would run at a fraction of the cost if they were moved to serverless. The pay-per-use pricing model also enables more accurate cost prediction and monitoring based on your application’s throughput. This gives rise to the notion of FinDev, where finance and development can intersect and allows optimization to be targeted to give the optimal return-on-invest on the engineering efforts. And by building your application on serverless components, you can also leverage it as a business advantage and offer a more competitive, usage-based pricing to your customers. Which is going to be crucial at a time when businesses all around the world are affected by COVID and are looking for better efficiencies. In this webinar, we will cover topics such as: - How does the cost of serverless differ from serverful applications? - How to predict and monitor cost in serverless applications? - When should you optimize for cost? - How can you leverage usage-based pricing as a business advantage?

How serverless changes the cost paradigm

Yan Cui

Why your next serverless project should use AWS AppSync

Yan Cui

Serverless technologies drastically simplify the task of building modern, scalable APIs in the cloud, and GraphQL makes it easy for frontend teams to consume these APIs and to iterate quickly on your product idea. Together, they are a perfect combination for a product-focused, full-stack team to deliver customer values quickly. In this talk, see how we built a new social network mobile app in under 4 weeks using Lambda, AppSync, DynamoDB and Algolia. How we approached CI/CD, testing, authentication and lessons we learnt along the way. Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

Build social network in 4 weeks

Yan Cui

Building a social network in under 4 weeks with Serverless and GraphQL

Yan Cui

FinDev as a business advantage in the post covid19 economy

Yan Cui

AWS has improved Lambda cold starts by leaps and bounds in the last year. But for performance-sensitive applications such as user-facing APIs, Lambda cold starts are still a thorn in one’s side, especially when working with languages such as Java and .Net Core. In this webinar, we will dive into strategies for improving cold start latency and how to mitigate them altogether with Provisioned Concurrency, and how Lumigo helps you optimize your use of Provisioned Concurrency.

How to improve lambda cold starts

Yan Cui

What can you do with lambda in 2020

Yan Cui

How to debug slow lambda response times

Yan Cui

What can you do with lambda in 2020

Yan Cui

In this talk, I'm gonna tell you all about AWS Step Functions - how it works, when to use it, and some tips on how to accelerate app development so you can ship customer values faster. Recording: coming soon Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com

How to ship customer value faster with step functions

Yan Cui

When a Lambda function times out, it’s one of the trickier problems to debug, especially if the function performs multiple IO calls. What was it doing when it timed out? And how do I identify these timeout errors quickly when Lambda lumps them together with other unhandled exceptions? In this webinar, we will show you how to identify and debug timeout errors, and strategies for mitigating timeouts and degraded performance more gracefully when there are underlying issues.

Debugging Lambda timeouts

Yan Cui

Serverless a superpower for frontend developers

Yan Cui

Debugging AWS Lambda Performance Issues

Yan Cui

Recording: https://www.youtube.com/watch?v=bnXp29kQIwU Real-world serverless podcast: https://realworldserverless.com Learn Lambda best practices: https://lambdabestpractice.com Blog: https://theburningmonk.com Consulting services: https://theburningmonk.com/hire-me Production-Ready Serverless workshop: https://productionreadyserverless.com AWS has taken over the responsibilities of patching the OS and securing the underlying physical infrastructure that runs your serverless application, so what’s left for you to secure? Quite a bit it turns out. The OWASP top 10 is as relevant to you as ever; DOS attacks are still a threat even if you can probably brute force your way through it as AWS auto-scales Lambda functions automatically; and did you know attackers can easily steal your AWS credentials via your application dependencies? In addition to the traditional threats, serverless applications have more granular deployment units and therefore there are more things to configure and secure, and the tools and practices are still catching up with this fast-changing world.

Serverless Security: Defence Against the Dark Arts

Yan Cui

Mais de Yan Cui (20)

How to win the game of trade-offs

How to choose the right messaging service

How to choose the right messaging service for your workload

Lambda and DynamoDB best practices

Lessons from running AppSync in prod

How to ship customer value faster with step functions

How serverless changes the cost paradigm

Why your next serverless project should use AWS AppSync

Build social network in 4 weeks

Building a social network in under 4 weeks with Serverless and GraphQL

FinDev as a business advantage in the post covid19 economy

How to improve lambda cold starts

What can you do with lambda in 2020

How to debug slow lambda response times

What can you do with lambda in 2020

How to ship customer value faster with step functions

Debugging Lambda timeouts

Serverless a superpower for frontend developers

Debugging AWS Lambda Performance Issues

Serverless Security: Defence Against the Dark Arts

Último

Sara Mae O’Brien Scott and Tatiana Baquero Cakici, Senior Consultants at Enterprise Knowledge (EK), presented “AI Fast Track to Search-Focused AI Solutions” at the Information Architecture Conference (IAC24) that took place on April 11, 2024 in Seattle, WA. In their presentation, O’Brien-Scott and Cakici focused on what Enterprise AI is, why it is important, and what it takes to empower organizations to get started on a search-based AI journey and stay on track. The presentation explored the complexities of enterprise search challenges and how IA principles can be leveraged to provide AI solutions through the use of a semantic layer. O’Brien-Scott and Cakici showcased a case study where a taxonomy, an ontology, and a knowledge graph were used to structure content at a healthcare workforce solutions organization, providing personalized content recommendations and increasing content findability. In this session, participants gained insights about the following: Most common types of AI categories and use cases; Recommended steps to design and implement taxonomies and ontologies, ensuring they evolve effectively and support the organization’s search objectives; Taxonomy and ontology design considerations and best practices; Real-world AI applications that illustrated the value of taxonomies, ontologies, and knowledge graphs; and Tools, roles, and skills to design and implement AI-powered search solutions.

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Enterprise Knowledge

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Histor y of HAM Radio presentation slide

vu2urc

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

CNv6 Instructor Chapter 6 Quality of Service

giselly40

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

A chaos experiment a day, keeping the outage away

1. one chaos experiment a day. keep the outages away. yan cui,@theburningmonk

2. Chaos Engineering?

3. MUST KILL SERVERS! RAWR!! RAWR!!

4. @theburningmonk theburningmonk.com “the discipline of experimenting on a system in order to build conﬁdence in the system’s capability to withstand turbulent conditions in production” principlesofchaos.org

5. @theburningmonk theburningmonk.com microservices death stars circa 2015

6. @theburningmonk theburningmonk.com

7. @theburningmonk theburningmonk.com “the capacity to recover quickly from difﬁculties; toughness.” resilience /rɪˈzɪlɪəns/ noun

8. @theburningmonk theburningmonk.com “the capacity to recover quickly from difﬁculties; toughness.” resilience /rɪˈzɪlɪəns/ noun it’s not about preventing failures!

9. everything fails, all the time

10. @theburningmonk theburningmonk.com “You don't choose the moment, the moment chooses you! You only choose how prepared you are when it does.” Fire Chief Mike Burtch

11. @theburningmonk theburningmonk.com

12. @theburningmonk theburningmonk.com anything that can go wrong, will go wrong. MURPHY’s LAW

13. @theburningmonk theburningmonk.com identify weaknesses before they manifest in system-wide, aberrant behaviors GOAL

14. @theburningmonk theburningmonk.com learn about the system’s behavior by observing it during a controlled experiments HOW

15. @theburningmonk theburningmonk.com learn about the system’s behavior by observing it during a controlled experiments HOW game days failure injection

16. Yan Cui http://theburningmonk.com @theburningmonk AWS user for 10 years

17. Yan Cui http://theburningmonk.com @theburningmonk http://bit.ly/yubl-serverless

18. Yan Cui http://theburningmonk.com @theburningmonk Developer Advocate @

19.

20. Yan Cui http://theburningmonk.com @theburningmonk Independent Consultant advisetraining delivery

21. theburningmonk.com/courses

22. theburningmonk.com/courses

23. realworldserverless.com

24. “using serverless reduces the blast radius” www.buzzsprout.com/877747/4615985

25. @theburningmonk theburningmonk.com serverless improves resilience as platform takes care of infrastructure failures

26. @theburningmonk theburningmonk.com

27. @theburningmonk theburningmonk.com by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

28. @theburningmonk theburningmonk.com by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

29. @theburningmonk theburningmonk.com Shared Responsibility Model

30. @theburningmonk theburningmonk.com by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

31. @theburningmonk theburningmonk.com chaos monkey kills an EC2 instance latency monkey induces artiﬁcial delay in APIs chaos gorilla kills an AWS Availability Zone chaos kong kills an entire AWS region

32. @theburningmonk theburningmonk.com

33. @theburningmonk theburningmonk.com

34. @theburningmonk theburningmonk.com there are no servers to kill! SERVERLESS

35. @theburningmonk theburningmonk.com improperly tuned timeouts

36. @theburningmonk theburningmonk.com missing error handling

37. @theburningmonk theburningmonk.com missing fallbacks

38. @theburningmonk theburningmonk.com

39. @theburningmonk theburningmonk.com STEP 1. deﬁne steady state i.e. “what does normal look like”

40. @theburningmonk theburningmonk.com STEP 2. hypothesis that steady state continues in control and experimental group e.g. “the system stays up if a server dies”

41. @theburningmonk theburningmonk.com STEP 3. inject realistic failures e.g. “slow response from 3rd-party service”

42. @theburningmonk theburningmonk.com STEP 4. try to disprove hypothesis i.e. “look for difference between control and experimental group”

43. @theburningmonk theburningmonk.com latency inject latency to function invocation

44. @theburningmonk theburningmonk.com “what if service X has elevated latency?”

45. @theburningmonk theburningmonk.com API Gateway Lambda API Gateway Lambda

46. @theburningmonk theburningmonk.com

47. @theburningmonk theburningmonk.com hypothesis: API would timeout and our try-catch would handle it and return default response

48. @theburningmonk theburningmonk.com

49. @theburningmonk theburningmonk.com

50. @theburningmonk theburningmonk.com result: function times out after 6s (hypothesis is disproved)

51. @theburningmonk theburningmonk.com

52. @theburningmonk theburningmonk.com API Gateway Lambda API Gateway Lambda 502 200

53. @theburningmonk theburningmonk.com API Gateway Lambda API Gateway Lambda 3s timeout 6s timeout

54. @theburningmonk theburningmonk.com API Gateway Lambda API Gateway Lambda max 29s integration max 15 mins timeout

55. @theburningmonk theburningmonk.com and then there’s cold starts…

56. @theburningmonk theburningmonk.com TIL: most HTTP client libraries have default timeout of 60s. API Gateway has an integration timeout of 29s. Most Lambda functions default to timeout of 3-6s. Don’t forget about the cold starts!

57. @theburningmonk theburningmonk.com

58. @theburningmonk theburningmonk.com

59. @theburningmonk theburningmonk.com https://bit.ly/2Wvfort

60. @theburningmonk theburningmonk.com

61. @theburningmonk theburningmonk.com

62. @theburningmonk theburningmonk.com

63. @theburningmonk theburningmonk.com

64. @theburningmonk theburningmonk.com

65. @theburningmonk theburningmonk.com

66. @theburningmonk theburningmonk.com

67. @theburningmonk theburningmonk.com outcome: a more resilient system

68. @theburningmonk theburningmonk.com latency exception inject latency to function invocation throws exception

69. @theburningmonk theburningmonk.com latency exception statuscode inject latency to function invocation throws exception return HTTP status code

70. @theburningmonk theburningmonk.com latency exception statuscode diskspace inject latency to function invocation throws exception return HTTP status code ﬁlls up /tmp directory

71. @theburningmonk theburningmonk.com latency exception statuscode diskspace denylist inject latency to function invocation throws exception return HTTP status code ﬁlls up /tmp directory looses network connectivity

72. @theburningmonk theburningmonk.com “what if DynamoDB has an elevated error rate?”

73. @theburningmonk theburningmonk.com API Gateway Lambda DynamoDB

74. @theburningmonk theburningmonk.com

75. @theburningmonk theburningmonk.com hypothesis: the AWS SDK retries would handle it

76. @theburningmonk theburningmonk.com result: function times out after 6s (hypothesis is disproved)

77. @theburningmonk theburningmonk.com TIL: the js DynamoDB client defaults to 10 retries with base delay of 50ms

78. @theburningmonk theburningmonk.com TIL: the js DynamoDB client defaults to 10 retries with base delay of 50ms delay = Math.random() * (Math.pow(2, retryCount) * base) this is Marc Brooker’s fav formula!

79. @theburningmonk theburningmonk.com

80. @theburningmonk theburningmonk.com

81. @theburningmonk theburningmonk.com action: set max retry count + fallback

82. @theburningmonk theburningmonk.com

83. @theburningmonk theburningmonk.com

84. @theburningmonk theburningmonk.com

85. @theburningmonk theburningmonk.com

86. @theburningmonk theburningmonk.com outcome: a more resilient system

87. @theburningmonk theburningmonk.com latency exception statuscode diskspace denylist inject latency to function invocation throws exception return HTTP status code ﬁlls up /tmp directory looses network connectivity

88. everything fails, all the time

89. @theburningmonk theburningmonk.com by Russ Miles @russmiles source https://medium.com/russmiles/chaos-engineering-for-the-business-17b723f26361

90. @theburningmonk theburningmonk.com

91. https://theburningmonk.com/hire-me AdviseTraining Delivery “Fundamentally, Yan has improved our team by increasing our ability to derive value from AWS and Lambda in particular.” Nick Blair Tech Lead

92. @theburningmonk theburningmonk.com github.com/theburningmonk

A chaos experiment a day, keeping the outage away

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a A chaos experiment a day, keeping the outage away

Semelhante a A chaos experiment a day, keeping the outage away (20)

Mais de Yan Cui

Mais de Yan Cui (20)

Último

Último (20)

A chaos experiment a day, keeping the outage away