DevOps (start walking in the same direction) by ops

1. DEVOPS
START WALKING IN THE SAME DIRECTION

2. WHO I AM?
Demis Rizzotto
System Engineer,DevOps Linux
Engineer @Exxoss
Currently consultant @Lampiris

3. THE PROBLEM OF
DOWNWARD SPIRAL

4. DIAMETRICALLY OPPOSED GOALS AND INCENTIVES
DEV
▸ Respond to the rapidly
changing competitive
landscape
OPS
▸ Provide stable, reliable, and
secure service to the
customer
THE CORE, CHRONIC CONFLICT
WHEN ORGANIZATIONAL MEASUREMENTS AND INCENTIVES ACROSS DIFFERENT SILOS PREVENT THE
ACHIEVEMENT OF GLOBAL, ORGANIZATIONAL GOALS

5. FRUSTRATION
▸Unnecessary process steps
▸Unnecessary rework
▸Unnecessary features
▸Waiting for someone else to act on an action or task
▸Waiting for an approval
▸Waiting for an environment
▸Creating a ticket for a manual task
▸Reporting to management by manually updating
spreadsheets
▸Manual status reporting
▸…

6. RESULTS
▸ Fragile application are prone to failure
▸ Ever increasing backlog and technical debt
▸ Urgent security rework and remediation
▸ More urgent projects into the queue
▸ Big deployments very difficult to diagnose when fail

7. DEVOPS
More features
quicker
Stability

8. DEVOPS
DEV OPS
FACILITATE AND AUTOMATE DEPLOYMENT
HELP DEV TEAM BECOME AUTONOMUS
FACILITATE DIAGNOSTIC
GET DEVS INTERESTED IN « PROD STUFF »
GIVE ACCESS TO LOGS
GIVE ACCESS TO MONITORING
OPTIMIZE LOGS
DEFINE METRICS
QUALITY TESTS
MONITOR YOUR APPS

9. DEV IN DEVOPS

10. DEV IN DEVOPS
IMPROVEMENTS
▸ The ability to accurately track dependencies
▸ Services that are resilient and degrade gracefully
▸ The ability to archive data to manage the size of the production
data set
▸ The ability to easily search and understand log messages across
services
▸ The ability to trace requests from users through multiple services
▸ Simple, centralized runtime configuration using feature flags, …

11. DEV IN DEVOPS
▸ Make Dev responsible for their own services
▸ Integrate Dev into the incident management processes
▸ Have Dev cross-train Ops
▸ Embed Ops knowledge and capabilities into Dev
▸ Design for IT Operations
EMBED IT OPERATIONS INTO DEVELOPMENT

12. OPS IN DEVOPS

13. OPS IN DEVOPS
IMPROVEMENTS
▸ Self service
▸ Infrastructure as code
▸ Standardize infrastructure
▸ Make all infrastructure data visible
▸ Make application info data visible
▸ Modify the incident resolution process and blameless post-mortems
▸ Monitor the health of the deployment pipelines

14. OPS IN DEVOPS
▸ Single “repository of truth” containing both the code and
environments
▸ Create the one-step Dev, Test and Production environment build
process
▸ Extend the deployment pipeline processes into production
EXTEND DEVELOPMENT INTO IT OPERATIONS

15. WE ARE ALL DEVOPS
WE ARE ALL DEVOPS
▸ Stop think . Do
▸ Share responsability
▸ It’s Never Someone Else’s Problem
▸ Dedicated Designated Ops in every team

16. ▸ Different type of devops, but devops is possible for all type
and size of companies
▸ Iterate!
▸ Identify bottleneck and remediate
▸ Define small target and share results
DEVOPS IS FOR ALL
DEVOPS IS FOR ALL

17. DEVOPS IS FOR ALL
▸ AUTONOMUS TEAMS
▸ Trust but verify
▸ Local autonomy, but global consistency
▸ SELF SERVICE
▸ How ? API, web portal, infra as code , AWS accounts, …
▸ AUTOMATE
▸ If you are tired to do it manually. Automate it.
▸ REDUCE NON-VALUE ADDED
▸ If isn't your business it's maybe better don't do it your self
KEY OF SUCCESS : REDUCE FRUSTRATION

18. INFRASTRUCTURE

19. ON DEMAND ( IAAS )
EASIER TO REBUILD THAN TO REPAIR
INFRASTUCTURE

20. INFRASTRUCTURE
IMMUTABLE INFRASTRUCTURE
▸ Automate the setup and deployment for every part and
every layer of your infrastructure.
▸ Never change any part of your system once it is deployed.
If you need to change it, deploy a new system.
AVOID HISTORY

21. INFRASTRUCTURE
INFRASTRUCTURE AS CODE
Versionning and code review
Easier track changes
Bootstrap your infrastructure from scratch
Automate

22. CI/CD

23. TESTING

24. TESTING
CONTINUOUS TESTING
▸ Automated tests
▸ code quality
▸ Unit tests
▸ Stress tests
▸ Don’t forget to test infrastructure (load banacer, recovery
etc.)
▸ Extend you build pipeline

25. SECURITY

26. SECURITY
INFORMATION SECURITY AS EVERYONE’S JOB, EVERY DAY
▸ Updates
▸ Use secrets management
▸ Integrate security into our CI/CD pipeline
▸ Check code libraries and their recommended configurations
▸ Dependency scanning
▸ Static analysis
▸ Dynamic analysis (ex: OWASP Zed Attack Proxy + Selenium)

27. SECURITY
SECURITY METRICS MONITORING
▸ Successful and unsuccessful user logins
▸ User password resets
▸ User email address resets
▸ Database syntax error
▸ User credit card changes

28. DEPLOYMENTS

29. DEPLOYMENTS
DEPLOYMENTS ARE ROUTINE AND PREDICTABLE
No stress . It’s a routine.
On business day when everyone is
already in the office and without our
customers even noticing

30. DEPLOYMENTS
HOW
▸ Work in small batches and delivering quickly and
frequently
▸ Blue green deployment
▸ Feature toggle
▸ Canary testing
▸ Dark launch
▸ Automated deployments

31. MONITORING

32. ALERTING
▸ Mesure service health not only server health (Health pattern)
▸ Reducing Alert Fatigue
▸ Cut alerts that aren’t actionable
▸ Adjust thresholds
▸ Consolidate related alerts
▸ Give alerts relevant names & descriptions
▸ Make sure the right people are getting alerts
▸ Multiple notification rules and choose right channel
▸ Use incident management tools(Pagerduty, Victoryops,..)

33. MONITORING
IF IT MOVES, TRACK IT
ANOMALY DETECTIONTRENDS

34. APM
End user monitoring
Application Performance Management

35. DELIVERY PIPELINE METRICS - TEAM LEVEL DASHBOARD

36. LOGS

37. LOGS
▸ Set your log severity right
▸ Remember you will not be the only one reading these logs
▸ Track your communication with other systems
▸ Log exceptions correctly
▸ Use an ID to track your events
▸ Normalize yours logs
▸ Separate and Centralize your Log Data

38. LOGS

39. DOCUMENT, MEASURE,
COMMUNICATE AND
COLLABORATE

40. DOCUMENT
WRITING EFFECTIVE DOCUMENTATION
▸ Treat documentation like a requirement
▸ Put the Information in the Most Appropriate Place
▸ *Keep your documentation close to the code
▸ Keep documentation just simple enough, but not too
simple
▸ Avoid creating large volumes of documentation of questionable
detail which become obsolete shortly after they are written

41. MEASURE
MEASURE PROGRESS
▸ KPIs
▸ Deployment frequency
▸ Deployment speed
▸ Failure rate
▸ Time from code to prod
▸ Share goals
▸ Increase the visibility of work

42. COMMUNICATE
▸ Find better ways to communicante
▸ Promote day to day direct communication
▸ Coffee machine meetings
▸ Chat (Slack, HipChat, Team,…)
▸ Use video (Slack, Skype, Hangout,..)
▸ Quality over quantity
▸ Post Mortem meetings
▸ Bootcamps,…
IN DEVOPS CULTURE, COMMUNICATION AND COLLABORATION ARE KEY

43. COLLABORATE
TIPS FOR BETTER COLLABORATION
▸ Change the seating and rotate people through development
▸ Improve the environment for collaboration (Breakout areas,
whiteboards, nice meeting rooms, a decent coffee machine)
▸ Cross domain workshops for analyze your process
▸ Block time for process improvement work (lean)
▸ Run lunch and learn / brown bag sessions
▸ Put a persistent chat room in place

44. Demis Rizzotto
demis.r@me.com
@DemisRiz
THANKS YOU!
QUESTIONS?