WinOps - DevOps on Windows - is a community started in London, UK to share the lessons learnt from those organisations who are successfully doing DevOps in a Windows world. In this session we'll share some lessons learnt from DevOps implementations in large Enterprise organisations who are using Microsoft technologies, but we'll also share how we can learn lessons from the open-source community. We'd also like to encourage attendees to "spread the word" of WinOps and create new WinOps meetups in their own tech communities.
4. @TheOpsMgr @DevOpsGroup
Agenda
• A cautionary tale (or two)
• DevOps 101 (Express Edition!)
• DevOps on Windows – WinOps - Mini-Case studies
• Starting your own WinOps Community
8. TSB said in July the compensation and repair bill
associated with the IT problems had reached
£176m, but the final cost is likely to be higher
because at that point the bank had resolved only
a third of the 135,403 complaints it received.
13. John P. Kotter, “Accelerate!”, HBR
The hierarchical structures and organizational
processes we have used for decades to run and
improve our enterprises are no longer up to the
task of winning in this faster-moving world
14. Gene Kim, Author of the Phoenix Project
“High performing organisations such as Amazon, Google,
Twitter, Etsy and Netflix have adopted a set of techniques we
now call DevOps and they are routinely deploying hundreds
or even thousands of production changes a day, while
preserving world-class reliability, stability and security. By
instituting a set of cultural norms, processes and practices,
these high performers are achieving breath-taking
performance.”
15. @TheOpsMgr @DevOpsGroup
“The architecture, technical practices, and cultural norms
that enable us to…
Increase our ability to deliver applications and services…
Quickly and safely, which enables rapid experimentation
and innovation, and the fastest deliver of value to our
customers…
While ensuring world-class security, reliability and
stability…
… so that we can win in the marketplace”
Gene Kim, DevOps Enterprise Summit, October 2018
24. Low Performers Key Success Factor High Performers
Command & Control Management Style Autonomous
Conservative Attitude to Change Experimental
Silo Organisation Structure Collaborative
Project-focussed Delivery Focus Product-centric
Waterfall Delivery Model Iterative (Agile)
Large (Huge) Batch size Smallest possible
Monolithic Systems Architecture Loosely coupled
Proprietary Technology Open (Source)
On-premise Infrastructure Public Cloud
Manual Processes Automated
High Performance IT is a different way of working
25. @TheOpsMgr @DevOpsGroup
What does this mean for your business? (2017)
High Performance IT powers High Performance Business
46x
Deploy
frequency
Lead time for
changes
440x 96x
Mean time to
recover
1
5
Change
failure rate
2x
Likelihood of exceeding profitability, market share and
productivity goals.
High performers compared with low performers, 2017 State of DevOps report
26. @TheOpsMgr @DevOpsGroup
What does this mean for your business? (2018)
High Performance IT powers High Performance Business
46x
Frequent
code
deployments
Faster lead
time for
changes
2555x 2604x
Faster to
recover from
incidents
7x
Lower change
failure rate
2x
Likelihood of exceeding profitability, market share and
productivity goals.
Elite performers compared with low performers, 2018 State of DevOps report
27.
28. @TheOpsMgr @DevOpsGroup
ASOS Case Study
Challenges
• Rapidly growing organisation ~25
scrum teams
• In–house, centralised ALM
framework now a bottleneck
• Maxed out ~ 300 releases / yr
Solution
• Azure adoption
• Re-architect codebase / decouple
monolith
• Adopt industry standard DevOps
tooling e.g. TeamCity, Octopus
Deploy
Result
• Now scaled to 75 scrum teams
• 3000+ release a year
• Grown to £1.8Bn revenue https://www.youtube.com/watch?v=oWe-
u4HDDac&feature=youtu.be
29. @TheOpsMgr @DevOpsGroup
Challenges
• Too slow to release code
• Datacentres capacity issues
• Siloed org supporting 14 different
ticketing platforms
Solution
• Product Aligned Org
• Shared Platforms inc open source
• Containerise, Instrument, Automate
Result
• Improved collaboration across
teams
• Shared platforms reduce waste
• Improved time to market
https://www.youtube.com/watch?v=PL0IEhpxNWQ&feature=youtu.be
30. @TheOpsMgr @DevOpsGroup
Facebook Case Study
Challenges
• Lack of visibility into current & desired
state
• SCCM knowledge siloed
• Reliance on GUI / config not in source
control (unlike Linux environments)
Solution
• Build a toolchain using Chocolatey,
Powershell, Chef, Phacility and other
modern tooling to replace SCCM
• Everything in source control
• Peer code reviews to share knowledge
Result
• Improved visibility and auditability
(it’s all in source control)
• Better collaboration
(everyone can edit a text file)
https://www.youtube.com/watch?v=oWe-
u4HDDac&feature=youtu.be
31. @TheOpsMgr @DevOpsGroup
CallCredit Case Study
Challenges
• Multiple manual deployments
• Lots of waiting in queues / handovers
• Long build times
• End Result – customers not getting
upgrades = No value delivered!
Solution
• 3 day hackathon with MSFT to get their
core platform into containers on Windows
as a proof-of-concept (via VSTS)
Result
• Validation that they can containerise &
automate
• Business buy-in to scale this out to
Production now they can see the benefits
• Plan to start re-factoring the monolith
https://www.youtube.com/watch?v=SNh7nNdLbqQ
32.
33. Just because you’re on Windows doesn’t mean you
can’t do DevOps (WinOps!)… so no excuses!
34. People and processes are the hardest challenge you’ll
face in building a Adaptive IT capability
38. Which is incredibly hard in siloed Enterprise
organisations, which is why taking a “systems
thinking” approach and mapping value streams is so
critical
39.
40. Find the biggest pain point, fix that, call what you did
to fix it DevOps and then rinse & repeat…
41. Concept
30 days 30-90 days 90-180+ days 180+ days
Identify organisational
capabilities and constraints
• Highlight drivers for
adoption
• Establish goals & objectives
• Create actionable next steps
1
Adopt
Delivery of new capabilities at scale
• Expand delivery & platform teams
• Workloads and data migration/
production cutover
• Implementation of DevOps Target
Operating Model
• Knowledge transfer and
enablement
3
Pilot
Demonstration of new capabilities
• Decision pack for strategic adoption
• Establish learning pathways
• Develop and build a Minimum Viable
DevOps Pipeline
• Migrate pilot workloads as
representative use cases
2
Practice
Ongoing improvement of
implemented solutions
• Continued knowledge transfer
and enablement.
• Continued coaching and
mentoring.
4
42. Everyone was focussed on solving IT problems that
were causing business problems… so that the success
had a measurable impact on the organisation
Anyone know who this guy is?
TSB – 550 branches across UK, 8500 staff, acquired in 2015 by Spanish Bank Banco Sabadell for £1.7Bn
20th TSB April 2018 started a “migration” from their former parent Lloyds Banking Group platform to the (new) Sabadell Proteo banking platform
Suffice to say… it didn’t go well…
Up to 1.9 customers were locked out of their accounts for days.
People didn’t get paid. Mortgages weren’t paid.
Direct debits and standing orders weren’t paid.
People could see other people’s accounts
People were defrauded of money…
Complete meltdown.
Things were was so bad TSB had to call in IBM to help.
So bad the CEO lost his job… <click>
So bad that when I gave this talk back at Ignite in September the problems still hadn’t been fixed <click>
The FCA (the UK equivalent of the FSCA / SARB in RSA) are contemplating mandating MTTR in the Financial Services regulations.
Partly as a result of this issue. But there have been many other cases of high profile failures
And these failures have a real cost <click>
GBP 176M = $3Tn ZAR
So potentially half a billion pounds (xxx USD) cost… which is 5x the £100M saving they were hoping to make from the migration
<change topic>
But it’s NOT only the CEO’s and CIO’s that have to worry about losing their jobs in this brave new world… the 4th industrial revolution as some people are calling it…
Who here would call themselves an IT Pro / Operations person / systems administrator?
So back in 2013 2 academics Frey and Osborne published a paper where they analysed the likelihood of job losses due to automation based on a number of key characteristics of the work.
What they came up with was… <click>
78% likelihood for “IT Operations technician”
So if any of you were wondering whether you should get into devops and learn some developer skills it’s worth pointing out that… <click>
Developers are in the “quite unlikely” bucket…
Speaking of DevOps – let’s whip into a DevOps 101 to understand what DevOps is and isn’t <click>
Before we dive into WHAT Devops is, and isn’t, it’s worth asking the question “Why do we need a new operating model for IT”?
There are many factors as play but I’d like to highlight 3 factors that have a major influence on the need for change
The first is Cloud Computing – cloud computing has fundamentally changed the underlying economics of computing. New entrants – digital native startups – no longer need to build their own data centres, buy servers, storage and networking in order to get started. They can use per minute billed, global scale IaaS, SaaS and PaaS services and scale their usage to meet customer demand
The second is the rise of the Smartphone – “we’ve got an app for that” is now a ubiquitous expectation across almost all consumer and B2B sectors. This has put enormous pressure on IT to ship new features, and new back-end integrations services - (API’s) - into their often aging legacy software estates
Thirdly is the changing demographics of “digital natives” and “millennials” who have grown up in a internet-enabled, smartphone-centric world. They expect “on demand” services, on their phones and laptops, with a seamless customer experience between the two, and ideally that seamless experience extending into the physical world, for example for bricks&mortar retailers.
All of this combines into a wave of Digital Disruption & Innovation that needs new, faster and better ways of deliverying IT services. The old ways just won’t work any more…
John P Kotter – Harvard Emeritus professor and leading management theorist for 30 years – but it best when he said [quote]
In turn, this lead IT practitioners and researchers to look at what the leading tech (and often digital native) IT companies were doing – the Googles, Amazons and Netflixes of the world.
And what they found was… [click]
These organisations where DIFFERENT – as Gene Kim, (co-Author of “the phoenix project which I urge you all to read…) said they have different “cultural norms, processes and practices”
Which leads to his 2018 definition of DevOps… <click>
Click
Click
Click
Click
Click
So when we looked patterns practices and cultural norms we found they broke down into 5 key areas – which leads to what’s called the CALMS model of DevOps <click>
What they found was that high performing IT organisations were different in 5 key areas…
A FOUNDATION of Culture – they embraced change, were more autonomous and more empowered
4 Pillars of core practices to support DevOps Transformation and high-performance IT…
Automation – they were leveraging the latest automation tools like Puppet, Chef and Ansible
Lean IT – they were embracing Lean IT principles borrowed from manufacturing (particularly automotive and the Toyota Production System e.g. Kanban) but more importantly focussing on small batch sizes and the flow of work through the system
Measurement – they were using advanced measurement techniques to provide constant feedback on their customer needs, and the impact of their IT initiatives. Less opinion, more data science!
Sharing – a focus on breaking down silos between departments and sharing knowledge and best practice, often using ideas drawn from open source software development.
Let’s quickly drill down into 2 of those in a bit more detail – breaking down the silos and the flow of work… [click]
most traditional IT organisation are organised by TECHNICAL ROLE into a series of silos – each with a different waiting queue, with a different SLA, each focussing on optimising it’s own part of the end to end value stream.
Think of it like a car engine – if your carburettor or fuel injection system only focussed on “its role” and tried to “maximise its utilisation” it would pour the maximum amount of fuel and air it can up to the limit of its designed capacity into the cylinder, regardless of the current throttle input or engine revs. End result – the engine stalls and dies. The system breaks.
DevOps is systems thinking – let’s look at the OUTCOME we’re trying to achieve, and design a model optimise for the fast flow of work, normally with smaller batch sizes, and organised around multi-disciplinary teams.
SO what does a product-aligned, multi-disciplinary DevOps team look like? [click]
The idea of a DevOps product delivery team is that they own a defined product from initial design and development right through into production deployment AND operations – Dev & Ops together in a vertically integrated team.
The product owner is empowered to prioritise the backlog of work, based on the organisational goals, customer needs and the fast feedback of data from customers and product (remember the M for measurement in CALMS).
The goal here is to balance out the “user stories” (new features) with the “Operability stories” (things that make it easier to manage, reduce outages or cloud hosting costs etc).
All of this underpinned by a focus on automating as much as possible, to free up time for higher value work.
Where you start across the DevOps pipeline is very dependent on the organisation
Where is your constraint?
In an agile way, delivering cumulative outcomes that reduce the cost of delay and get value into the hands of our customers faster
But all of this requires behavioural change from ALL the stakeholders…
<click>
Cloud Enabled
DevOps Optimised
Integrated Systems API’s
Agile & Lean Process
Skilled Workforce
The performance difference between high and low performers across 4 key metrics that measure Speed & Stability is stark.
And this flows through into higher organisational performance.
So they did this survey in 2019… anyone want to guess the numbers? <click>
In 2018 they introduced the Elite performance category…
How much do you think Paul Pester would have paid to have 7x the change of keeping his job…
Or instead of 159 days the recovery time being 159/2604 = 1.5hrs…
Ok, so onto part 3 – lets look at some of this in the real world based on case studies presented over the last 4 years of WinOps conferences in London <click>
Justs worth mentioning that these case studies all come with YouTube links to the case study presentations so you can all watch them at your leisure, and the presentation deck will be available after the session via the Ignite portal and I will tweet out a link as well. <click>
Due to acquisition worldwide they were supporting 14 different ticket platforms…
https://www.youtube.com/watch?v=PL0IEhpxNWQ&feature=youtu.be
https://www.slideshare.net/WinOpsConf/connon-macrae-evolution-of-ticketmasters-journey-to-devops
Any objection based on not having the technology on Windows is no longer valid – all the major tools vendors and cloud vendors support Windows workloads and technologies.
But if you thought technology was the problem… then YOU are probably the problem
Transformations, change initiatives and Projects fail… more than they succeed.
So how can you make your transformation scalable so that even if it’s a “fail” at organisation level you’ve still had a lasting impact at the individual or team level…
This is why there is a lot of emphasis on “ways of working” so we start this transformation at the lowest, most granular level.
Remember the model of behaviour change? We have to take people along this journey
Understanding that different people will change at different rates.
And some might not change at all <click>
What we often see is what we call the Icecream sandwich model of organisational change
C suite want it because they don’t want to end up like Paul Pester…
Engineering want it because they want to learn all those shiny new skills (cos we like shiny!)
Middle Management… are terrified of letting go of command & control because they don’t know how to LEAD so they are frozen in fear…
DevOps is all about systems thinking and focusing on the constraints that affect the ENTIRE system and then focusing your pilots and changes at the point of constraint.
And things like value-stream mapping and event storming are great here.
It’s amazing the power of getting a diverse group of people into a room with whiteboards, post-its and sharpies and actually get them to map out their dependencies….
It’s AMAZING the power of getting a diverse group of people into a room with whiteboards, post-its and sharpies and actually get them to map out their dependencies….
A lot of what we [DevOpsGroup] do often feels more like marriage guidance counselling for dysfunctional organisations where the relationship has broken down and it’s heading for divorce (or in IT terms “outsourcing”) unless we can bridge these silos.
Start small – all the case studies we discussed started with small pilots, hack days, 1 team and so forth…
Hot Tips - Find the biggest pain point that people have been whinging and moaning about for ever, fix that, call what you did to fix it DevOps and then rinse & repeat…
We have a four stage model – Concept, Pilot, Adopt, Practice
And the final key lesson from these case studies is that even though they were IT led projects, and sometimes they were fixing “IT problems”, they were all solving the BUSINESS problem that was being impeded by the IT constraint – shipping features faster, improving stability and scalability etc.
<Click>
This is the secret to selling the transformation to the wider business – “we want to build these new capabilities that are going to help you win customers, build better products, get products to market faster” – not “I want to move all these servers to Azure using Azure Migrate and Azure Site Recovery”
[Changing topic]
So changing topics a bit now to focus on the WinOps community as a whole and as a call to action to what you can do back in your own communities to drive the DevOps on Windows message forward… <click>
Microsoft is mobilising great resources to help along with the great new Microsoft Learn platform too.