Microsoft has embarked on its DevOps journey more than 8 years ago, when it started modeling a new set of engineering tools and practices called One Engineering System (1ES), fully based on their own platforms – Azure DevOps (ex Visual Studio Team Services - VSTS) and Azure. Since then, multiple product development teams have gone through a massive transition in terms of how they organize their work and deal with infrastructure, tools and processes. This session tells the story of the transformation of 70+ teams with more than 800 people that develop Azure DevOps, both Service and Server (ex Team Foundation Server – TFS). These teams have been previously producing the monolithic 3-tier on premises TFS environment every couple of years. Nowadays, they moved to the cloud cadence and deliver daily updates to Azure DevOps Services, their global scale, always available, cloud SaaS service made up of dozens of micro-services. At the same time, they deliver the same capabilities on premises every couple of months. Join us to learn how Microsoft’s teams transitioned from an inert waterfall to a highly performing agile organization, whereby drastically changing the way how they develop, test, deploy and monitor their products. We will explain how they manage large agile distributed product development, how they improved their development process by switching to GIT, how they changed their approach to testing and quality control and managed to take full control over their technical debt.

1. HOTEL EUROPE, October 19th, 2019
DevOps Transformation
In Microsoft
Case Study
ANA ROJE IVANČIĆ, OGNJEN BAJIĆ
M I C R O S O F T M V P S F O R D E V E L O P E R T E C H N O L O G I E S
P R O F E S S I O N A L S C R U M T R A I N E R S ( P S T ) F O R S C R U M . O R G
A G I L I S T I T , C R O A T I A

2. HOTEL EUROPE, October 19th, 2019

3. HOTEL EUROPE, October 19th, 2019
In Software Development for over 20 years, worked as Dev, PM, Test, RM, SM, PO…
Working with TFS/VSTS/VSO/Azure DevOps since 2005
Helping teams improve DevOps practices since 2005
Microsoft MVPs for Developer Technologies (ALM/DevOps)
Professional Scrum Trainers (PST) for Scrum.org since 2018
Professional Scrum Foundations (PSF) and Professional Scrum Developer (PSD)
Speakers: Ana Roje Ivančić and Ognjen Bajić

4. HOTEL EUROPE, October 19th, 2019
THE JOURNEY TO
DEVOPS

5. HOTEL EUROPE, October 19th, 2019
Azure DevOps Team
850 people
60+ feature teams
4 primary locations
3,500850

6. HOTEL EUROPE, October 19th, 2019
Azure DevOps
https://azure.com/devops
Deliver value to your users
faster using proven agile tools
to plan, track, and discuss work
across your teams.
Azure Boards
Get unlimited, cloud-hosted
private Git repos and
collaborate to build better code
with pull requests and
advanced file management.
Azure Repos
Build, test, and deploy with CI/CD
that works with any language,
platform, and cloud. Connect to
GitHub or any other Git provider
and deploy continuously.
Azure Pipelines
Test and ship with confidence
using manual and exploratory
testing tools.
Azure Test Plans
Create, host, and share packages
with your team, and add artifacts
to your CI/CD pipelines with a
single click.
Azure Artifacts

7. HOTEL EUROPE, October 19th, 2019
Azure DevOps is the toolchain of choice
for MS Engineering
DevOps in Microsoft
Data: Internal Microsoft engineering system activity, November 2018
https://aka.ms/DevOpsAtMicrosoftè
96,000
Engineers using
Azure DevOps
2m
Git commits per month
163,000Deployments per day
6.3m
Builds per month
500m
Test executions per day
500k
Work items updated
per day
5m
Work items viewed
per day
Satya Nadella

8. HOTEL EUROPE, October 19th, 2019
PLANNING AND
TEAMS

9. HOTEL EUROPE, October 19th, 2019
April 2010 Sprint 1
October 2019 Sprint 159
How Do They Work?
3 weeks

10. HOTEL EUROPE, October 19th, 2019
Team Structure

11. HOTEL EUROPE, October 19th, 2019

12. HOTEL EUROPE, October 19th, 2019

13. HOTEL EUROPE, October 19th, 2019
Feature Teams 10-12 people
Cross disciplined
Self managing
Clear charter and goals
Own features in production
Own deployment of features
Fully responsible for quality
Physical team rooms
Intact for 12-18 months

14. HOTEL EUROPE, October 19th, 2019
Sticky Note Exercise - Self Forming Teams
Employee choice,
not manager driven
Typically <20% change,
but 100%
get to make a choice
Cross-pollinate talent
and micro-culture

15. HOTEL EUROPE, October 19th, 2019
Planning – Rolling Planning and Aligned Autonomy
Strategy
12 months
12
Quarter
4 sprints
4
Sprint
3 weeks
1
Semester
6 months
6
Teams are responsible for the detail:
HOW are we building it?
Are we building it with QUALITY?
Leadership is responsible for the big picture:
WHAT are we building?
WHY are we building?

16. HOTEL EUROPE, October 19th, 2019
Product OKRs
Service OKRs Service OKRs Service OKRs Service OKRs Service OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Team
OKRs
Objectives and Key Results (OKRs)
Azure Boards Azure ReposAzure Pipelines Azure Test Plans Azure Artifacts

17. HOTEL EUROPE, October 19th, 2019
1. per service
◦ Value delivered for each Objective with short video
◦ Value planned for the next sprint
2.
◦ Each team reviews with higher leadership their plans, progress, impediments
How Teams Stay In Sync?
Q3Q2Q1 Q4
3 weeks

18. HOTEL EUROPE, October 19th, 2019
DAY IN THE LIFE
OF AN ENGINEER

19. HOTEL EUROPE, October 19th, 2019
Source in a single git repo
All code flows to master
Engineers don’t merge code
they didn’t write
Frequent small check-ins
Everyone Works Out of Master
Master
Week 3Week 2Week 1
Sprint Previous Sprint Next
175 commits/day
into Master
Release: Current Sprint x
Release: Sprint Previous x

20. HOTEL EUROPE, October 19th, 2019
Automated Tests Running In Pull Requests

21. HOTEL EUROPE, October 19th, 2019
„Bug Cap” to Keep Quality Under Control
If your bug count exceeds your bug cap, stop working on new
features until you’re back under the cap!
4 40x =10
Everyone follows a simple rule they call the “Bug Cap”:
# enginneers
in the team
# number of
bugs per
engineer
Bug Cap per
team

22. HOTEL EUROPE, October 19th, 2019
A strategy adopted by their
teams to provide focus, and
assist with an interrupt culture
The team self-organizes each
sprint into two distinct sub-
teams: Features and Shield
Shielding Distractions: L-Team
F-Team
Works on commited
features (new work)
L-Team
Deals with all live-site
issues and interruptions

23. HOTEL EUROPE, October 19th, 2019
THERE IS NO PLACE
LIKE PRODUCTION

24. HOTEL EUROPE, October 19th, 2019
Different rings serve different customer groups
Rings provide control over customer exposure to features & bugs
◦ No down time
◦ Fully automated
Safe Deployment - 6 Production Deployment Rings

25. HOTEL EUROPE, October 19th, 2019
Production Telemetry
7 terabytes of data
collected daily
They monitor feature usage,
performance, exceptions,
logs
Goal: Discover and address
problems before end users
even become aware of them
Telemetry is decisive for
experimentation and
hypothesis verification

26. HOTEL EUROPE, October 19th, 2019
Live site status is always the top priority
Transparent root cause analysis for every LSI
LSI fixes go into backlog (2 sprint rule)
Actionable alerts
On-call Designated Responsible Individual (DRI)
Customer Focused Availability model (SLA)
Health reports per service and per team
Live Site Culture

27. HOTEL EUROPE, October 19th, 2019
DevOps Transformation
The Azure DevOps teams story so far
4-6 month milestones
Horizontal teams
Personal offices
Long planning cycles
PM, Dev, Test
Feature branches
20+ person teams
Secret roadmap
Bug debt accumulated
Deep organizational hierarchy
Success is a measure of install numbers
Features shipped once a year
3-week sprints
Vertical teams
Team rooms
Continual Planning & Learning
PM & Engineering
Everyone in master
8-12 person teams
Publicly shared roadmap
Debt paid as incurred
Flattened organization hierarchy
User satisfaction determines success
Features shipped every sprint
Before After

28. HOTEL EUROPE, October 19th, 2019
Build the culture you want, and you’ll get the behavior you’re after
You cannot cheat shipping
Collecting telemetry to verify value delivered
DevOps Transformation In Microsoft - Takeaways
arojeivancic@agilist.hr obajic@agilist.hr

29. HOTEL EUROPE, October 19th, 2019

30. HOTEL EUROPE, October 19th, 2019
THANK YOU!