This session is an overview on what DevOps is (to me) and how it impacts traditional organizations the most. DevOps is way more than just continuous delivery! From an Agile (synergetic) mindset, DevOps takes a step beyond and focusses on automation, collaboration and learning. Apart from that I also look forward to what oppurtunities lie ahead when implementing DevOps.
On March 2nd I presented this DevOps Unraveled session for abt 40 IT-managers at business university Nyenrode. This was part of the Masterclass Agile management
(Dutch website http://www.executiveeducation.nl/open-programmas/programmadetails/masterclass-agile-management/sectie/introductie.html ).
2. Agenda
1. Introduction
2. What is DevOps anyway?
3. Stories from the trenches
4. Theories behind DevOps
5. Where is DevOps bringing us?
6. Accelerators and inhibitors of DevOps
7. DevOps unraveled
2
3. 1. Introduction
About me…
1997 Tester
2001 My first Agile project
2004 Joined Ordina
2008 Thoughtleader Agile Testing & Author of ‘Testen2.0’
2012 Thoughtleader Agile
2013 DevOps PM
2014 Agile & DevOps coach and trainer
3
6. 1. Introduction
Goals
6
Uncovering what DevOps is
Theories behind DevOps
Why should an organization implement DevOps?
Define what the impact of DevOps on organizations is
Note:
DevOps is fairly new and under heavy construction
9. 1. Introduction
Marshall model: 4 paradigms
Ad hoc: no process or organization, focus on ad hoc situations, heavy on people
relations
− “Customer of the week”
Analytical: Focus on processes and tasks, functional silo’s, heavy on up front
structuring of the process
− CMMI, PRINCE2, ITIL, BiSL,6 Sigma, …
Synergetic: Teamwork, delivery, focus on results, heavy on people collaborating
towards valuable results
− Agile, Scrum, DevOps, Lean, …
− “None of us is as smart as all of us” ~Gerald M. Weinberg
Chaordic: Innovation mindset, no formal organization, embrace volatility
− Dee Hock (‘Chaordic’): ‘Simple rules and regulations give rise to complex, intelligent
behavior. Complex rules and regulations give rise to simple, stupid behavior”
− Google Friday
9
14. 2. What is DevOps anyway?
What do others say about DevOps?
15
Ordina:
Dev and Ops collaborating on joint business goals, striving for a maximum
of speed, quality and control in their efforts.
DevOps is “a cross-disciplinary community of practice dedicated to the
study of building, evolving and operating rapidly-changing resilient
systems at scale.” – Jez Humble
15. 2. What is DevOps anyway?
Quotes
Jez Humble:
− “In the era of continuous deployment, you need _tons_ of communication in order to
reduce the risk of frequent delivered releases.”
− “If it hurts, do it often.”
Gene Kim
− “You get what you design for. Chester, your peer in Development, is spending all his
cycles on features, instead of stability, security, scalability, manageability, operability,
continuity, and all those other beautiful ’ilities.”
Jeff Hodges
− "Knowing how the system’s behavior on day 20 is different from its behavior on day 15
is the difference between successful engineering and failed shamanism."
16
17. 2. What is DevOps anyway?
DevOps is Automation
19
18. 2. What is DevOps anyway?
DevOps concepts (just a few!)
20
Infrastructure as code
− Configuration management
− Puppet, Chef
Continuous Integration & Delivery
− Jenkins
Continous Deployment
− Docker, Vagrant
Monitoring
− Graphite
19. 2. What is DevOps anyway?
DevOps practices (just a few!)
A/B testing
Automated dashboards
Big visible dashboards
Blue-green deployment
Composable deployments
Consistent tooling
Cross-functional skills
Dark launching
Deployment pipeline
Develop for production
Feature toggles
Feedback from production
21
Integrated deployment planning
Integrate production stories
Packaged artifact
Polyskilled experts
Production support
Scripted deployments
Stop the line (Lean)
Test-driven everything
Version everything
…
20. 2. What is DevOps anyway?
Elements of DevOps
22
Collaboration
21. 2. What is DevOps anyway?
DevOps is collaboration
Cross functional teams
− Dev and Ops
− Dev Engineers & Ops Engineers
− Projects AND Changes AND Incidents in one team
Same metrics
− Learning as a team
− Valuable feedback loops
Joint tooling
− Process
− Deployments
− Monitoring
23
22. 2. What is DevOps anyway?
Elements of DevOps
24
23. 2. What is DevOps anyway?
DevOps is learning from production
Feedback from production
− Monitoring
− Business value
− New feature / change / incident
Impact on (current) Design and Architecture
− Not only Dev!
Continuous improvement
− System
− Process
25
24. 2. What is DevOps anyway?
Elements of DevOps
27
Jenkins
25. 2. What is DevOps anyway?
Impact of Continuous Delivery
28
28. 3. Stories from the trenches
ING, SchubergPhilis, Chamber of Commerce
29. 3. Stories from the trenches
Speakers
Evelijn van Leeuwen, ING
Harm Boertien & Arthur van Schendel, SchubergPhilis
Frank Verbruggen, Kamer van Koophandel (via Ordina)
32
31. 4. Theories behind DevOps
Thoughtleaders
Patrick DeBois
Gene Kim
Jez Humble
34
32. 4. Theories behind DevOps
DeBois - ideas
35
Founder of the DevOps movement
DevOpsDays 2009, Ghent (B)
Jedi.be
33. 4. Theories behind DevOps
DeBois – 4 areas
36
1. Extend delivery to production
− ‘Dev and Ops collaborate to improve anything on delivering the project to production‘
− deployment pipeline
2. Extend Ops feedback to project
− ‘All information from production is radiated back to the project’
− Non-functional behavior
− dark launching
3. Embed project knowledge into Ops
− ‘When the project takes co-ownership of everything that happens in productions’
− Production support
4. Embed Ops knowledge into project
− ‘When Ops are involved from the beginning of the project’
− Develop for production / Blue-green deployment
34. 4. Theories behind DevOps
C.A.L.M.S. (Damon Edwards, Jez Humble)
Culture: Own the change to drive collaboration and communication
Automation: Take manual steps out of your value chain
Lean: Use lean principles to enable higher cycle frequency
Measurement: Measure everything and use data to refine cycles
Sharing: Share experiences, successful or not, to enable others to learn
37
http://www.rackspace.com/blog/quantifying-devops-capability-its-important-to-keep-calms/
35. 4. Theories behind DevOps
C.A.L.M.S. - Culture
Own the change to drive collaboration and communication
Yeah, but let’s do Chef first…
Flow, flow, flow!
Trust
Excellence
Craftsmanship
Failure is an option
38
37. 4. Theories behind DevOps
C.A.L.M.S. - Automation
Take manual steps out of your value chain
Damon Edwards / Jez Humble
Radically improve cycle time (months hours)
Automating common (!) tasks
− Let’s automate IT
Continuous Delivery
− “Through automation of the build, deployment and testing process, and improved
collaboration between developers, testers, and operations, delivery teams can get
changes released in a matter of hours – sometimes even minutes – no matter what the
size of a project or the complexity of its code base.”
Database migrations
Managing environments
40
40. 4. Theories behind DevOps
C.A.L.M.S. - Lean
Use lean principles to enable higher cycle frequency
Production line
Eliminating waste to improve throughput
Metrics
Quality – time to restore service, % unplanned work
Cost – transaction cost of making a change
Delivery – Change lead time
Safety – make changes and experiments safe to fail
Morale – job satisfaction is highly correlated with business
outcomes
43
Recommended sources:
- The principles of Product Development FLOW –
Donald G. Reinertsen
41. 4. Theories behind DevOps
C.A.L.M.S. - Measurement
Measure everything and use data to refine cycles
Capture everything
− Monitoring
− Stats
− Metric store (meta-metrics)
Business metrics
− Revenue
− # orders
− # users
− …
45
http://www.slideshare.net/PuppetLabs/how-to-measure-
everything-a-million-metrics-per-second-with-minimal-
developer-overhead-puppetco
42. 4. Theories behind DevOps
C.A.L.M.S. - Measurement
Measure everything and use data to refine cycles
Ops metrics, such as
− Time To Resolve (TTR)
− Time To Detect (TTD)
− Time Between Failures (TBF)
Make it human again
− Visualize
− Personas
46
http://devops.com/blogs/devops-scorecard/
43. 4. Theories behind DevOps
C.A.L.M.S. - Sharing
Damon Edwards / John Willis / Jez Humble
Share experiences, successful or not, to enable others to
learn
Open source model for tools and knowledge
Learning from development production
Big visible charts / displays
Collaboration = Joint goals
47
44. 4. Theories behind DevOps
Three Ways
1. The First Way : Systems Thinking emphasizes the performance of the entire
system
2. The Second Way is about creating and amplifying the right to left feedback
loops
3. The Third Way is about creating a culture that fosters both continual
experimentation, taking risks and learning from failure; and understanding
that repetition and practice is the prerequisite to mastery.
48
1.
2.
3.
45. 4. Theories behind DevOps
Three ways
49
1. Systems Thinking
− No longer compartimentation of processes
− One system
− How many steps are needed to deploy 1 single feature (VSM)?
2. Feedback loops
− Testing, See the whole, Graphs from production
− Understanding customers
− Fostering ownership
3. Culture of continual experimentation and learning from failure
− Safe to fail? Culture!
− Learning!
− Hackdays
1.
2.
3.
http://www.slideshare.net/jmcgarr/
implementing-devops
46. 4. Theories behind DevOps
Theory of Constraints
A focusing process to identify the constraint and restructure
the rest of the organization around it.
Production lines
A company is a system
Gathering, analyzing, solving, and implementing
(management) problems
There is always a biggest constraint
− Continuous Improvement!
50
47. 4. Theories behind DevOps
Theory of Constraints
Bottleneck:
− Any resource with a capacity equal to or less than the demand placed upon it
Constraint:
− Anything that limits the system’s performance, relative to the system goal
ToC Tools:
− Visual diagrams
− Cause-effect thinking
− Connect intuition to logic
− Conflict Resolution Diagram
− Etc.
51
48. 4. Theories behind DevOps
Theory of Constraints & DevOps
Why aren’t we deploying more often?
Why do deployments take so long?
Why is releasing our software such a hassle?
…
Book: The Phoenix Project
52
53. 5. Where DevOps is bringing us
Enterprise DevOps NL
‘DevOps Dialogues’
ASL BiSL, itSMF, Agile Consortium
End user organizations: Defense, Achmea, A.S.R.,
Rabobank, Belastingdienst, ING, TenneT, CJIB en
Philips
Vendors: Ordina, CGI, Schuberg Philis, Sogeti, Quint
Wellington Redwood, The Lifecycle Company, Xebia en
Valori
DevOps, wat heb je eraan?
- Cultuur en DevOps
- DevOps in the enterprise
57
54. 5. Where DevOps is bringing us
The organization
58
Organization
Strategy
& Policy
Finance
Quality Process
Management
People
57. 5. Where DevOps is bringing us
Projects will become more rare
Selforganized domain focused teams with a lifecycle focus
− Launches
− Redesigns
− Changes
− Incidents
Scalable resourcing
− Scaling up / scaling down
− Adapting the processes
Less of a demand for PM’s
61
58. 5. Where DevOps is bringing us
Destination
4 areas are reached
− Dev designs, develops & delivers for deployability
− Supportability: monitoring & logging
− Feedback throughout the lifecycle (incl. Experiments)
− Full swing test automation & harnesses
Holistic Definition of Done
Dev+Ops infrastructure and deployment automation
More agility, more fun
Measures, such as:
− Cycle time for critical fixes decreases
− Deployment/production/platform-related issues decreases
− Decrease in unplanned work for Ops
63
61. 6. Accelerators and inhibitors
Accelerators
Cloud
Rhineland values
− Focus on professionals
− Weggeman
Remote working (‘HNW’)
Common goals!
Culture of Trust
Agile architecture
Startup mentality, like Apple ®
67
62. 6. Accelerators and inhibitors
Inhibitors
‘Classic’ thinking
− Processes, detailed plans, big decisions up front etc.
Functional silo’s
− Don’t create new ones!
Remote working (…)
Assurance
− From the classic paradigm
Certification
− From the classic paradigm
68
67. 73
3. Continuous learning provided
Feedback loops throughout the organization
Teams that own business processes
Cross departments
Auditors: please audit THIS!
7. Promises of DevOps unraveled
Results
Not in try-out:
What are current companies doing around DevOps?
Shamanism:
Sjamanisme is het manipuleren van bovennatuurlijke machten door een sjamaan die contact kan maken met geesten en de geestenwereld, veelal om op die manier door geesten verooorzaakte ziektes te genezen.
Jenkins
Github
Corollary = gevolgtrekking
Geen Continuous Delivery (DHL)
Geen Lots of Automation (dashboard)
Geen golden egg / silver bullet
http://www.ibm.com/developerworks/library/a-devops9/
CI: dmv feature branches
Docker
open platform for developers and sysadmins to build, ship, and run distributed applications
build any app in any language using any toolchain. “Dockerized” apps are completely portable and can run anywhere
https://www.youtube.com/user/jedibvba
5 years of DevOps: https://www.youtube.com/watch?v=uRMV6tT_mu0
These are the pillars of a successful DevOps implementation. Ignoring just one can compromise overall effectiveness.
They also serve as an excellent benchmark to assess your current capability and highlight the areas you need to improve.
https://www.youtube.com/watch?v=kbvMmNrz8Kk
John Willis presenting on DevOps Culture at SV DevOps Meetup (4/18/13)
Pathological = ziekelijk
Generative = generatief = het vermogen tot groei en voortplanting in zich hebbend, voortbrengend
Shirked = ontweken (van ontwijken)
Compartmented = gecompartimenteerd
Enquiry = onderzoek
Create constancy of purpose toward improvement of product and service, with the aim to become competitive, to stay in business and to provide jobs.
Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.
Cease dependence on inspection to achieve quality. Build quality into the product in the first place.
End the practice of awarding business on the basis of a price tag. Instead, minimize total cost. Move towards a single supplier for any one item, on a long-term relationship of loyalty and trust.
Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease costs.
Institute training on the job.
Institute leadership. The aim of supervision should be to help people and machines and gadgets do a better job. Supervision of management is in need of overhaul, as well as supervision of production workers.
Drive out fear, so that everyone may work effectively for the company.
Break down barriers between departments. People in research, design, sales, and production must work as a team, in order to foresee problems of production and usage that may be encountered with the product or service.
Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity.
Remove barriers that rob the hourly worker of his right to pride of workmanship.
Remove barriers that rob people in management and in engineering of their right to pride of workmanship.
Institute a vigorous program of education and self-improvement.
Put everybody in the company to work to accomplish the transformation. The transformation is everybody's job.
Metric store, such as Graphite
Recommended: at least one graphite server per data center
http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change
Persona’s: afbeelding, ieder zijn ding (each one doing their own thing)
Metric store, such as Graphite
Recommended: at least one graphite server per data center
http://www.slideshare.net/jallspaw/ops-metametrics-the-currency-you-pay-for-change
TTD = Time To Detect
TTR = Time To Resolve
TBF = Time Between Failures
Deployment frequency: How often were we deploying code and getting new code in the hands of our customers? This metric should trend up or remain stable from week to week. Example: Twice a week, 50 times a day
Change volume: For each deployment how many user stories and new lines of code were we shipping? Example: 3 new features per day, Average 500 lines of new code per week. Another parameter to consider in addition to volume is complexity of change.
Lead Time (from Dev to Deploy): How long does it take on an average to get the code from development complete through a cycle of A/B testing to 100% deploy and upgrade on production? Lead time should reduce as the team gets a better hold of the lifecycle.
Percentage of failed deployments: What percentage of deployments failed causing an outage or a negative user reaction? This metric should decrease over time. Example: 9% deployments failed this month as opposed to 15% last month. This metric should be reviewed in combination with the change volume. If the change volume is low or remained the same but the percent of failed deployments increased, then there maybe a disfunction somewhere.
Mean time to recovery: When we did fail, how long did it take us to recover? This is a true indicator of how good we are getting with handling change and this should ideally reduce over time. You can expect some spikes in this number due to complex issues not encountered before. Example: On an average it took the team 15 minutes to resolve each last week, 14 minutes this week.
Customer Ticket Volume: Number of alerts generated by customers to indicate issues in the service. This is a basic indicator of customer satisfaction. Example: 54 tickets were generated this week as opposed to 38 while the user volume remained steady is not a good thing
% Change in User Volume: Number of new users signing up, interacting with my service and generating traffic. As new users sign up is my infrastructure able to handle the demand? Example: This week the number of customers spiked by 30% due to an external event causing volume of requests to go up
Availability: What is the overall uptime for my service and did I violate any SLAs? Example: 99.9% uptime consistently for the last 3 months even with change in user volume
Performance (Response Time): Is my service performing within my predetermined thresholds? This metric should remain stable irrespective of % change in user volume or any new deployment. Example: Sub 5 second response time from all geographies and devices
The First Way emphasizes the performance of the entire system, as opposed to the performance of a specific silo of work or department — this as can be as large a division (e.g., Development or IT Operations) or as small as an individual contributor (e.g., a developer, system administrator).
The focus is on all business value streams that are enabled by IT. In other words, it begins when requirements are identified (e.g., by the business or IT), are built in Development, and then transitioned into IT Operations, where the value is then delivered to the customer as a form of a service.
The outcomes of putting the First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system (as per Deming).
The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.
The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
NOT A DEVOPS THEORY, but an underlying and important theory
Identify
Exploit – utilize maximally
Subordinate all other processes to the constraint
Elevate the system’s constraint
Start with Identification - but be aware of Inertia!
Weggeman Bohica: https://www.youtube.com/watch?v=hpEX6gGYkwo
Weggeman Planning & Control: https://www.youtube.com/watch?v=32pqEbgNKbQ
Trust:
Trust begins to emerge when we have a sense that another person or organization is driven by things other than their own self-gain. – Simon Sinek
There is nothing wrong with AGILE (synergetic!) Assurance or Certification programmes!
What’s a little known fact about you?
Jenkins
Github
Composable deploy =
Jenkins
Github
http://www.lean4it.com/2013/05/a-devops-causal-loop-diagram.html
Change Frequency has a positive impact on Change Capability. The more Changes, the better you are at handling Change. Note that "size" of change is not relevant to this relationship - it is a separate variable.
Change Frequency has a negative relationship to Change Size. The more often you change, the smaller the changes will be, everything else being equal.
Change Capability has a negative relationship to Change Risk. That is, if Change Capability improves, Change Risk decreases, and vice versa.
Change Size increases Change Risk, and vice versa.
Finally, Change Risk has a negative effect on Stability.