1. DevopsDays Tokyo 2012
1
Friday, May 25, 12

2. Acknowledgments
This presentation builds on the work of many. DevOps
(and cloud operating model) thought leaders, including
Damon Edwards, John Allspaw, Jez Humble, Adam
Jacob, Luke Kaines, Jesse Robbins, Chris Read, Andrew
Clay Shafer Gene Kim, Patrick Debois and John Vincent.
Special thanks to Cameron Haight of Gartner for
collaborating and codifying many of the Devops
community ideas.
2
Friday, May 25, 12

3. What is Devops?
3
Friday, May 25, 12

4. Success Stories
Instagram
• In 2 Years
30 Million users
2 Backend engineers (devops)
1 Billion dollar acquisition by Facebook
Dropbox
• Last 3 years
+50 Million new users
$240 million per year in revenue
Etsy
• Last 3 Years
Hit 1 Billion pageviews
Hosting 875,000 sellers and 15 million users worldwide
$525 million in sales in 2011
4
Friday, May 25, 12

5. Raising the Bar for IT Infrastructure
Netflix
• 100% cloud
• Developer focused PaaS
• Revenue Growth: 23% (three average)
• Profit Growth: 48% (three average)
Amazon
• Production deployment every 11.6 seconds (weekday)
• 1,079 deployments in one hour (record)
• ~0.001% of deployments actually cause an outage
5
Friday, May 25, 12

6. What do they have in common?
Kaizen
• Continuous improvement
Muda
• Eliminate Waste
Mura
• Unevenness - (reduce inventory)
Muri
• Overburdened (standardization)
6
Friday, May 25, 12

7. Now in English
Optimize for minimal operation burden
Instrument simplicity for the business
Instrument everything
Optimize for Happiness
7
Friday, May 25, 12

8. Common Technical Characteristics
Cloud or Cloud like infrastructure
Open Source and Open Culture
They practice a form of Devops/Agile
8
Friday, May 25, 12

9. So What is Devops?
Improving collaboration and communication between
development and operations teams.
An adoption of methodologies (and some technologies)
used in software development by operations teams.
A renaissance of infrastructure technologies that is
strongly influenced by cloud computing.
Cameron Haight - Devops Workshop
9
Friday, May 25, 12

10. What do we say?
CLAMS
• Culture
• Lean
• Automation
• Measurement
• Sharing
subscribe in
iTunes
10
Friday, May 25, 12

11. Simply Put
Improving the business
value of the work done
Text
within IT.
Cameron Haight - Devops Workshop
11
Friday, May 25, 12

12. Characterization of Devops
Wall of Confusion
Dev Ops
Ah-ha! Ka-ching!
12
Friday, May 25, 12

13. Characterization of Devops
Shorten
m ove
Dev Re Ops
Ah-ha! Ka-ching!
Wa l l of C onfus ion
13
Friday, May 25, 12

14. The Convergence of Devops
Devops History
• Agile Infrastructure Thread
• Velocity Thread
• Lean Startup Thread
The Convergence of Devops
http://bit.ly/itrev2
14
Friday, May 25, 12

15. Agile Infrastructure Thread
2008
2009
Friday, May 25, 12

16. Velocity Thread
2006 - 2008
2009
2010
Friday, May 25, 12

17. Lean Startup Thread
Customer Development Model
Friday, May 25, 12

18. The Seven Sins of Waste (Applied to Devops)
Transportation
• Reduce boundary hand-offs/approvals
Inventory
• Leverage cloud for JIT (just-in-time) resource
Motion
• Automate and implement self services
Waiting
• Manage flow (Visualize, WIP limits) / Kanban
Over-processing
• Implement MVP / min. viable product
Overproduction
• Optimize for the whole / Theory of Constraints
Defects
• Bring the pain forward / 5 whys 18
Friday, May 25, 12

19. Culture Principles
No rock star mentality
• Shared contributions
Healthy attitudes towards failure
• Failures are leaning opportunities
The problem is the enemy
• No blame games
No victims
• Shared blame
Develop shared metrics
• Focus on end goal
Alignment of Purpose
• Shared goals/ Slay the Dragon
19
Friday, May 25, 12

20. In Development
Done means released
• Code deploy not code complete
Infrastructure as code
• Configuration is code and needs control
Version control everything
• Everything is an artifact (scripts,xml, source)
Frequent releases
• Self service / Continuous delivery
Test end to end
• Test driven code and infrastructure
Instrument operations
• Feature flags / Canary releases / Immune systems
20
Friday, May 25, 12

21. In Operations
Automate where possible
• Desired state consistency
Instrument pervasively
• Collect data to detect trends early
If anything fails stop the line
• Reduce technical debt early
If it’s hard do it more often
• Practice make perfect
Enable graceful degradation
• Some availability is better than none
Reprovision not repair
• It’s easier to recover to a known state
MTTR vs MTBF
21
Friday, May 25, 12

22. The Organization
Hack Days
• Creating collaborative projects
Free Time
• Allowing special free project time
Embedded Engineers
• Dev in ops or ops in dev
Chat Rooms
• Skype, Hipcat, Watercooler
Fun Working Environments
• Games, reading rooms, bars
22
Friday, May 25, 12

23. Technical Debt
23
Friday, May 25, 12

24. Technical Debt - Vicious Cycle
Technical Vicious Toxic
Terminal
Debt Cycle Operations
24
Friday, May 25, 12

25. Tall of Two Startups
25
Friday, May 25, 12

26. Meat to Math Ratio
26
Friday, May 25, 12

27. A Cloud Gone Wrong
27
Friday, May 25, 12

28. Clouds Gone Wrong
Friday, May 25, 12

29. Friday, May 25, 12

30. “As-is”
operations server
templates
TAR
RPM restarts
TAR
TAR
code,
EC2
content pupp
deploys et RS
app RS deployment
devs deployment
S3
yum
repo
Provision-time builds
reconfigs
TAR EXE
TAR AUTO
server
TOOLS
code templates
TAR
RPM
middleware
devs platform CONTROL PROVISION RELEASE
system
eng
Friday, May 25, 12

31. Friday, May 25, 12

32. Fully Automated Infrastructure
Everything starts
here SVN
hud agent cfg
yum pup
son repo pet
packages
active users run packages
directory deck
commands
nag splunk
ios
resource model
sys cfg
right instance create
scale
events
new node Node a log data
CONTROL PROVISION RELEASE
Friday, May 25, 12

33. Acknowldegements
This presentation builds on the work of many. DevOps
(and cloud operating model) thought leaders, including
John Allspaw, Jez Humble, Adam Jacob, Luke Kaines,
Jesse Robbins, Chris Read, Andrew Clay Shafer Gene
Kim, Patrick Debois and John Vincent.
Special thanks to Cameron Haight of Gartner for
codifying many of my and other’s ideas in a grown up
way.
33
Friday, May 25, 12

34. The Prescriptive DevOps
• Capture and codify how to start and
finish successful DevOps
transformations
– Create isomorphic mapping between
plant floors and IT shops
– Co-authoring with Patrick DeBois,
Mike Orzen, John Willis
– Describe in detail how to replicate
the transformations describe in
“When IT Fails: The Novel”
• Goals
– How does Development, IT
Operations and Infosec become
dependable partners
– How do they work together to solve
business problems (and Infosec, too)
@RealGeneKim, genek@realgenekim.me
Friday, May 25, 12

35. Resources
DevOps
Cookbook
35
Friday, May 25, 12