SlideShare uma empresa Scribd logo
1 de 31
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
.Net Core in production
By Leonid Molotiievskyi
.NET CONFERENCE #1 IN UKRAINE
2
About me
• Hands-on software architect and technological
consultant
• Good at splitting a monolith to microservices
• Built a huge enterprise financial solution from scratch
• Technical guy who believes that right people decisions
are more important than technological ones
• Speaker and mentor
3
Spoilers about what we are going to talk
Agenda
Context overview
Environment that we used to live with
Scaling
How did we scale our services?
4
Hell for the DevOps
teamDo we solve the right problem?
Useful advices
The things that can help you to resolve
the problem
Lessons learned
How can we benefit in
future?
Q&A
Questions and answers
Context overview
Several statements about the project
6
Context overview
• Financial domain
• 25+ microservices
• Team 70+ people
• 20+ environments
• Three versions in support one in development
7
Solution overview: managing workflows
Scaling
How did we scale our services?
9
Notification service
10
Solution?
- And a set of dummy queues left after descale/redeploy appear
11
Gateway: infinite redirect
- Where do we store
keys for cookies?
12
Gateway: infinite redirect solution
Hell for the DevOps team
What technology decisions helped us to survive
14
Each morning…
• Dev/Staging/Prod cluster is down
• RabbitMq/Mongo/Consul/Prometheus is not
operational
• The fire-fighter team is on the duty
15
Greedy service
16
Queues are growing… - 1
• “TTL time is too small” or?
17
Queues are growing… - 2
• A queue has a set of consumers
• Service A consumes the message
• Service A starts processing the message
• Heath check of consumer fails due to high load of
service A/network issue/OOM killed/etc.
• Duplicated message appear in the queue
18
OOM Killed issue
• .Net Core 2.2 doesn’t respect docker limits:
https://github.com/aspnet/AspNetCore/issues/3409
https://github.com/dotnet/coreclr/issues/18971
• ” Server GC was designed with the assumption that
the process using Server GC is the dominant process
on the machine. By default it uses as many heaps as
there are # of processors on the machine.”
19
Let’s fix issue by upgrade to .Net Core 3.0?
https://github.com/mongodb/mongo-csharp-driver/pull/372/files
20
Socket file descriptor leak in HttpClient
21
Docker: no space left on the device
level=info msg="[8] System error: write
/sys/fs/cgroup/docker/01f5670fbee1f6687f58f3a943b1e1bdaec26
30197fa4da1b19cc3db7e3d3883/cgroup.procs: no space left on
device"
22
Reason:
23
Prometheus is down
Useful advices
What can prevent nasty situations
25
What can help you to find them?
Configured monitoring to track:
• Memory consumption
• CPU consumption
• Number of threads on worker node
• Number of open socket descriptors per node/pod
• Connection refused errors
• Correlation Ids in logs
• Number of messages in queues
• Number of consumers for queues
26
Use the standard health check middleware
27
Setup environment in the way…
• Infrastructure services must have HA setup
• Deploy at least two instances of each service
• Setup monitoring and alerting
• To be sure that “temporary data” disappear after
redeployment
• To not configure something manually
Lessons learned
What we get from it
29
Lessons learned
• ”Do it as simple as possible” principle
doesn’t work. “Do it in the smart way” - works
• Think about application scaling from the
begging
• Know about open issues inside your target
framework
• Do not blame DevOps team, try to help them
to find out what is the reason
30
Follow me @lmolotii on
Q&A
THANKS FOR WATCH !!!

Mais conteúdo relacionado

Semelhante a .NET Fest 2019. Леонид Молотиевский. DotNet Core in production

Network Automation Journey, A systems engineer NetOps perspective
Network Automation Journey, A systems engineer NetOps perspectiveNetwork Automation Journey, A systems engineer NetOps perspective
Network Automation Journey, A systems engineer NetOps perspective
Walid Shaari
 
Choosing the right parallel compute architecture
Choosing the right parallel compute architecture Choosing the right parallel compute architecture
Choosing the right parallel compute architecture
corehard_by
 
AOUG_11Nov2016_Challenges_with_EBS12_2
AOUG_11Nov2016_Challenges_with_EBS12_2AOUG_11Nov2016_Challenges_with_EBS12_2
AOUG_11Nov2016_Challenges_with_EBS12_2
Sean Braymen
 

Semelhante a .NET Fest 2019. Леонид Молотиевский. DotNet Core in production (20)

Ship code like a keptn
Ship code like a keptnShip code like a keptn
Ship code like a keptn
 
Architectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and ConsistentlyArchitectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and Consistently
 
Architectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and ConsistentlyArchitectural Decisions: Smoothly and Consistently
Architectural Decisions: Smoothly and Consistently
 
Bitfusion Nimbix Dev Summit Heterogeneous Architectures
Bitfusion Nimbix Dev Summit Heterogeneous Architectures Bitfusion Nimbix Dev Summit Heterogeneous Architectures
Bitfusion Nimbix Dev Summit Heterogeneous Architectures
 
From DevOps to NoOps
From DevOps to NoOpsFrom DevOps to NoOps
From DevOps to NoOps
 
Network Automation Journey, A systems engineer NetOps perspective
Network Automation Journey, A systems engineer NetOps perspectiveNetwork Automation Journey, A systems engineer NetOps perspective
Network Automation Journey, A systems engineer NetOps perspective
 
Micro Front-End & Microservices - Plansoft
Micro Front-End & Microservices - PlansoftMicro Front-End & Microservices - Plansoft
Micro Front-End & Microservices - Plansoft
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsHybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
 
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsHybrid and Multi-Cloud Strategies for Kubernetes with GitOps
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOps
 
Zoo keeper in the wild
Zoo keeper in the wildZoo keeper in the wild
Zoo keeper in the wild
 
Free GitOps Workshop
Free GitOps WorkshopFree GitOps Workshop
Free GitOps Workshop
 
Choosing the right parallel compute architecture
Choosing the right parallel compute architecture Choosing the right parallel compute architecture
Choosing the right parallel compute architecture
 
WTF is a Microservice - Rafael Schloming, Datawire
WTF is a Microservice - Rafael Schloming, DatawireWTF is a Microservice - Rafael Schloming, Datawire
WTF is a Microservice - Rafael Schloming, Datawire
 
AOUG_11Nov2016_Challenges_with_EBS12_2
AOUG_11Nov2016_Challenges_with_EBS12_2AOUG_11Nov2016_Challenges_with_EBS12_2
AOUG_11Nov2016_Challenges_with_EBS12_2
 
IBM Blockchain Platform - Architectural Good Practices v1.0
IBM Blockchain Platform - Architectural Good Practices v1.0IBM Blockchain Platform - Architectural Good Practices v1.0
IBM Blockchain Platform - Architectural Good Practices v1.0
 
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
Kubernetes, Toolbox to fail or succeed for beginners - Demi Ben-Ari, VP R&D @...
 
DESIGN West 2013 Presentation: Accelerating Android Development and Delivery
DESIGN West 2013 Presentation: Accelerating Android Development and DeliveryDESIGN West 2013 Presentation: Accelerating Android Development and Delivery
DESIGN West 2013 Presentation: Accelerating Android Development and Delivery
 
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit EuropeAutomation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
 
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The UglyDevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
 
OnAndroidConf 2013: Accelerating the Android Platform Build
OnAndroidConf 2013: Accelerating the Android Platform BuildOnAndroidConf 2013: Accelerating the Android Platform Build
OnAndroidConf 2013: Accelerating the Android Platform Build
 

Mais de NETFest

Mais de NETFest (20)

.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET
.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET
.NET Fest 2019. Николай Балакин. Микрооптимизации в мире .NET
 
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE....NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...
.NET Fest 2019. Сергей Калинец. Efficient Microservice Communication with .NE...
 
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET
.NET Fest 2019. Оля Гавриш. .NET Core 3.0 и будущее .NET
 
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов
.NET Fest 2019. Оля Гавриш. Машинное обучение для .NET программистов
 
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem....NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...
.NET Fest 2019. Roberto Freato. Provisioning Azure PaaS fluently with Managem...
 
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design
.NET Fest 2019. Halil Ibrahim Kalkan. Implementing Domain Driven Design
 
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex
.NET Fest 2019. Сергій Бута. Feature Toggles: Dynamic Configuration at Wirex
 
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A....NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...
.NET Fest 2019. Michael Staib. Hot Chocolate: GraphQL Schema Stitching with A...
 
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture
.NET Fest 2019. Андрей Литвинов. Async lifetime tests with xUnit and AutoFixture
 
.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests
.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests
.NET Fest 2019. Анатолий Колесник. Love, Death & F# Tests
 
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос....NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...
.NET Fest 2019. Алексей Голуб. Монадные парсер-комбинаторы в C# (простой спос...
 
.NET Fest 2019. Roberto Freato. Azure App Service deep dive
.NET Fest 2019. Roberto Freato. Azure App Service deep dive.NET Fest 2019. Roberto Freato. Azure App Service deep dive
.NET Fest 2019. Roberto Freato. Azure App Service deep dive
 
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com...
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com....NET Fest 2019. Александр Демчук. How to measure relationships within the Com...
.NET Fest 2019. Александр Демчук. How to measure relationships within the Com...
 
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real....NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...
.NET Fest 2019. Anna Melashkina та Philipp Bauknecht. Dragons in a Mixed Real...
 
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem
.NET Fest 2019. Alex Thissen. Architecting .NET solutions in a Docker ecosystem
 
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ....NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...
.NET Fest 2019. Stas Lebedenko. Practical serverless use cases in Azure with ...
 
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali....NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...
.NET Fest 2019. Сергей Медведев. How serverless makes Integration TDD a reali...
 
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET
.NET Fest 2019. Сергей Корж. Natural Language Processing in .NET
 
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur....NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
.NET Fest 2019. Eran Stiller. Create Your Own Serverless PKI with .NET & Azur...
 
.NET Fest 2019. Eran Stiller. 6 Lessons I Learned on My Journey from Monolith...
.NET Fest 2019. Eran Stiller. 6 Lessons I Learned on My Journey from Monolith....NET Fest 2019. Eran Stiller. 6 Lessons I Learned on My Journey from Monolith...
.NET Fest 2019. Eran Stiller. 6 Lessons I Learned on My Journey from Monolith...
 

Último

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in DelhiRussian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
kauryashika82
 
1029 - Danh muc Sach Giao Khoa 10 . pdf
1029 -  Danh muc Sach Giao Khoa 10 . pdf1029 -  Danh muc Sach Giao Khoa 10 . pdf
1029 - Danh muc Sach Giao Khoa 10 . pdf
QucHHunhnh
 

Último (20)

Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in DelhiRussian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
Russian Escort Service in Delhi 11k Hotel Foreigner Russian Call Girls in Delhi
 
Application orientated numerical on hev.ppt
Application orientated numerical on hev.pptApplication orientated numerical on hev.ppt
Application orientated numerical on hev.ppt
 
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptxBasic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
 
Grant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy ConsultingGrant Readiness 101 TechSoup and Remy Consulting
Grant Readiness 101 TechSoup and Remy Consulting
 
Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
Kodo Millet  PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...Kodo Millet  PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
Kodo Millet PPT made by Ghanshyam bairwa college of Agriculture kumher bhara...
 
Mixin Classes in Odoo 17 How to Extend Models Using Mixin Classes
Mixin Classes in Odoo 17  How to Extend Models Using Mixin ClassesMixin Classes in Odoo 17  How to Extend Models Using Mixin Classes
Mixin Classes in Odoo 17 How to Extend Models Using Mixin Classes
 
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdfUGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
UGC NET Paper 1 Mathematical Reasoning & Aptitude.pdf
 
How to Create and Manage Wizard in Odoo 17
How to Create and Manage Wizard in Odoo 17How to Create and Manage Wizard in Odoo 17
How to Create and Manage Wizard in Odoo 17
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The Basics
 
PROCESS RECORDING FORMAT.docx
PROCESS      RECORDING        FORMAT.docxPROCESS      RECORDING        FORMAT.docx
PROCESS RECORDING FORMAT.docx
 
psychiatric nursing HISTORY COLLECTION .docx
psychiatric  nursing HISTORY  COLLECTION  .docxpsychiatric  nursing HISTORY  COLLECTION  .docx
psychiatric nursing HISTORY COLLECTION .docx
 
Magic bus Group work1and 2 (Team 3).pptx
Magic bus Group work1and 2 (Team 3).pptxMagic bus Group work1and 2 (Team 3).pptx
Magic bus Group work1and 2 (Team 3).pptx
 
Key note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfKey note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdf
 
1029 - Danh muc Sach Giao Khoa 10 . pdf
1029 -  Danh muc Sach Giao Khoa 10 . pdf1029 -  Danh muc Sach Giao Khoa 10 . pdf
1029 - Danh muc Sach Giao Khoa 10 . pdf
 
Understanding Accommodations and Modifications
Understanding  Accommodations and ModificationsUnderstanding  Accommodations and Modifications
Understanding Accommodations and Modifications
 
Micro-Scholarship, What it is, How can it help me.pdf
Micro-Scholarship, What it is, How can it help me.pdfMicro-Scholarship, What it is, How can it help me.pdf
Micro-Scholarship, What it is, How can it help me.pdf
 
Mehran University Newsletter Vol-X, Issue-I, 2024
Mehran University Newsletter Vol-X, Issue-I, 2024Mehran University Newsletter Vol-X, Issue-I, 2024
Mehran University Newsletter Vol-X, Issue-I, 2024
 
Unit-IV; Professional Sales Representative (PSR).pptx
Unit-IV; Professional Sales Representative (PSR).pptxUnit-IV; Professional Sales Representative (PSR).pptx
Unit-IV; Professional Sales Representative (PSR).pptx
 
ICT role in 21st century education and it's challenges.
ICT role in 21st century education and it's challenges.ICT role in 21st century education and it's challenges.
ICT role in 21st century education and it's challenges.
 
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
 

.NET Fest 2019. Леонид Молотиевский. DotNet Core in production

  • 1. Тема доклада Тема доклада Тема доклада KYIV 2019 .Net Core in production By Leonid Molotiievskyi .NET CONFERENCE #1 IN UKRAINE
  • 2. 2 About me • Hands-on software architect and technological consultant • Good at splitting a monolith to microservices • Built a huge enterprise financial solution from scratch • Technical guy who believes that right people decisions are more important than technological ones • Speaker and mentor
  • 3. 3 Spoilers about what we are going to talk Agenda Context overview Environment that we used to live with Scaling How did we scale our services?
  • 4. 4 Hell for the DevOps teamDo we solve the right problem? Useful advices The things that can help you to resolve the problem Lessons learned How can we benefit in future? Q&A Questions and answers
  • 6. 6 Context overview • Financial domain • 25+ microservices • Team 70+ people • 20+ environments • Three versions in support one in development
  • 8. Scaling How did we scale our services?
  • 10. 10 Solution? - And a set of dummy queues left after descale/redeploy appear
  • 11. 11 Gateway: infinite redirect - Where do we store keys for cookies?
  • 13. Hell for the DevOps team What technology decisions helped us to survive
  • 14. 14 Each morning… • Dev/Staging/Prod cluster is down • RabbitMq/Mongo/Consul/Prometheus is not operational • The fire-fighter team is on the duty
  • 16. 16 Queues are growing… - 1 • “TTL time is too small” or?
  • 17. 17 Queues are growing… - 2 • A queue has a set of consumers • Service A consumes the message • Service A starts processing the message • Heath check of consumer fails due to high load of service A/network issue/OOM killed/etc. • Duplicated message appear in the queue
  • 18. 18 OOM Killed issue • .Net Core 2.2 doesn’t respect docker limits: https://github.com/aspnet/AspNetCore/issues/3409 https://github.com/dotnet/coreclr/issues/18971 • ” Server GC was designed with the assumption that the process using Server GC is the dominant process on the machine. By default it uses as many heaps as there are # of processors on the machine.”
  • 19. 19 Let’s fix issue by upgrade to .Net Core 3.0? https://github.com/mongodb/mongo-csharp-driver/pull/372/files
  • 20. 20 Socket file descriptor leak in HttpClient
  • 21. 21 Docker: no space left on the device level=info msg="[8] System error: write /sys/fs/cgroup/docker/01f5670fbee1f6687f58f3a943b1e1bdaec26 30197fa4da1b19cc3db7e3d3883/cgroup.procs: no space left on device"
  • 24. Useful advices What can prevent nasty situations
  • 25. 25 What can help you to find them? Configured monitoring to track: • Memory consumption • CPU consumption • Number of threads on worker node • Number of open socket descriptors per node/pod • Connection refused errors • Correlation Ids in logs • Number of messages in queues • Number of consumers for queues
  • 26. 26 Use the standard health check middleware
  • 27. 27 Setup environment in the way… • Infrastructure services must have HA setup • Deploy at least two instances of each service • Setup monitoring and alerting • To be sure that “temporary data” disappear after redeployment • To not configure something manually
  • 28. Lessons learned What we get from it
  • 29. 29 Lessons learned • ”Do it as simple as possible” principle doesn’t work. “Do it in the smart way” - works • Think about application scaling from the begging • Know about open issues inside your target framework • Do not blame DevOps team, try to help them to find out what is the reason