Sesión impartida en DevOps days Madrid 2020 https://devopsdays.org/events/2020-madrid/welcome/

1. advanced deployment
scenarios
rings and canaries
DevOps

2. hello!
Sergio Navarro Pino
Tech Lead at Analytic Always
@snavarropino
2

3. Agenda
1. Canary releases
 How to implement canary releases?
 Considerations
 Routing users to the canary
 Release gates
2. Deployment rings
3. Dark launchings
3

4. 1.
Canary releases
4
Danilo Sato https://martinfowler.com/bliki/CanaryRelease.html

5. Motivation
5

6. Benefits
✘ Reduce the risk of introducing a new software versions
✘ Safe rollback strategy if issues are found
✘ The ability to do capacity testing of the new version in a production
environment. Does my new version require enhanced hardware
capabilities?
✘ Collateral effect: you get zero downtime deployment !!
6

7. WHY?
✘ We cannot easily reproduce production traffic patterns
✘ We have no proper testing environment due to
○ Complex integration with third parties
○ Elevated hardware requirements
✘ We don’t have enough validation before going live
✘ You want to increase your confidence when a new release is deployed
7

8. 2.
How?
8

9. Similar to blue/green deployment we start deploying the
new version to a subset of our infrastructure. no
request are routed there yet
9
1. deploy new version

10. We start sending some request to the new version
10
2. start routing users

11. If no issues are found related to new version, we
gradually increase the percentage of users routed To
the new version
11
3. Monitorize

12. Once all users have been routed to the new version, we
remove the old one
12
4. remove old version

13. Place your screenshot here
13
Let’s meet our heroes

14. Application monitoring
✘ A list of awesome APM (Application Performance Monitoring) tools &
products (commercial and OSS)
https://github.com/antonarhipov/awesome-apm
14

15. Place your screenshot here
15
Azure Application Insights

16. “Canary deployment also gives you a
rapid way to rollback - if anything goes
wrong you may route all users to the old
version
16

17. key points to be taken in account
✘ We are running two versions in parallel
✘ Be sure your software supports it
https://es.slideshare.net/sergio_pino/despliega-como-los-grandes-
zero-downtime-deployment
17

18. key points to be taken in account
✘ Deliver to users a consistent experience
✘ How we are going to monitorize?
✘ Do we always have a rollback path?
✘ Don’t replace Devops practices
18

19. How we route users?
✘ Random pattern
✘ Geographic pattern
✘ IP range pattern
✘ Based on user type (freemium vs premium). Feature toggles
✘ Based on some application logic. Feature toggles
19

20. Place your screenshot here
20
How to finish the deployment?

21. What about release gates?
21
Gates allow automatic collection of health signals
from external services, and then promote the
release when all the signals are successful at the
same time or stop the deployment on timeout
https://docs.microsoft.com/en-
us/azure/devops/pipelines/release/approvals/gates?view=azure-devops

22. ✘ 10% canary users
✘ 25% canary users
✘ 100% canary users
Define several stages, and a gate to transition
between them
22
Transition will only
happen if there are no
alerts related to Failed
request, response time and
availability

23. How gates are evaluated?
23

24. Place your screenshot here
24
Azure DevOps Pipelines
Release Gates

25. 4.
Deployment rings
25

26. 26
Deployment rings were first discussed in Jez Humble's
book. They support the production-first DevOps
mindset and limit impact on end users, while gradually
deploying and validating changes in production.
Impact (also called blast radius), is evaluated through
observation, testing, analysis of telemetry, and user
feedback.

27. 27

28. Is it so easy?
28

29. 5.
Dark launching
29

30. The facebook case
30
During the two weeks prior to launch we began what we call a
"dark launch" of all the functionality on the backend. Essentially a
subset of user queries are routed to help us test, by making
"silent" queries to the code that, on launch night, will have to
absorb the traffic. This exposes pain points and areas of our
infrastructure that needs attention prior to the actual launch.
Increasing the demand on one subsystem may generate more
logs than anticipated and overwhelm analysis processes, or
unexpected network bottlenecks may appear.
https://www.facebook.com/note.php?note_id=96390263919

31. How the code looks?
Execute new code
but show the user
the result of old
code
31

32. Recap
Two versions
in parallel
Testing in
production
Monitorize
your
changes
32

33. thanks!
Any questions?
You can find me at
@snavarropino
33

34. Credits
Special thanks to all the people who made and released
these awesome resources for free:
✘ Presentation template by SlidesCarnival
✘ Angular app by Ismael Ramos
https://github.com/Ismaestro/angular8-example-app
34