1. From
Waterfall
to
Weekly
Releases:
A
Case
Study
in
using
Evo
&
Kanban
(2004-­‐05)
Tathagat
Varma
hHp://thoughtleadership.in

2. Our Product
• Network Management domain
• Windows-based specialized hardware
(“Appliances”)
• Installed in data centers for traffic monitoring,
analysis and network troubleshooting
– but not generally on production network
• Typical users are technical folks – CIO, Network
Manager, Network Engineers
• Selling cycles typically align with quarterly or
annual budget cycles
• Many sales require customer specials

3. Old Process, circa 2003
• Customer Bugs prioritized based on multiple business parameters,
including (partial list) -
– Severity
– Impact on Revenue, Volume, Competitive, etc.
– Case age
• PMO would prepare Maintenance Release Plan of Record (MR POR)
and get buy-in for various types of MRs -
– Service Packs – bunch up ~50-60 bugs typically every quarter
– Hot Fixes – 1-2 high-urgency bugs that can’t wait until next SP
– Patches – workaround for customer-specific issues
• SPs would have
– Above The Line (ATL) requirements – must fix
– Below The Line (BTL) requirements – fix if time permits

4. Problems with Old Process
• Dev team had no bandwidth to take on maintenance releases
• Huge pile of customer escalations without “home”
• Compounded by high incoming field rate
• Low closure rate (largely due to no dedicated resources)
• Large wait for customers to get bug fixes
• Tech Support often tasked team directly and broke the process
• Hot fixes not always available to all customers
• Sometimes, a new bug fix might break a hot fix
• If a hot fix failed in the field, rollbacks would be very difficult
• Difficult to estimate time to resolve a bug and give an ETA
• High-priority bugs could arrive at any time
• Customer specials could arrive anytime with top priority
• High internal rejection rate of bug fixes by Tech Support

5. New Process, 2004-05
Dedicated
Customer
Sustaining
Team
New
“Cumulative
Hot Fix”
process
Improved
collaboration
with all
stakeholders

6. EvoluLonary
Project
Management
“Evo”
• E1:
Decompose
by
performance
results
and
stakeholders
• E2:
Do
high-­‐risk
steps
early,
learn
how
‘unknowns’
really
perform
• E3:
Focus
on
improving
your
most
valuable
performance
objecLves
first
• E4:
Base
your
early
evoluLon
on
exisLng
frameworks
and
stakeholders
• E5:
Design
to
cost
dynamically
• E6:
Design
to
performance
dynamically
• E7:
Invest
in
an
open-­‐ended
architecture
early
on
• E8:
MoLvate
your
team
by
rewarding
results
• E9:
PrioriLze
changes
by
value,
not
place
in
queue
• E10:
Learn
fast,
change
fast,
adapt
to
reality
fast.

7. Process improvement…the
beginnings…

8. Cumulative Hot Fix Process

9. Weekly Build Process

10. Product C
Product D
Protocols
Decide Drivers
Backend
Product B
GUI
Product A
Our Kanban Process in action…
PMO
CST
Manager
QA Team
Dev
Team
=
15
WIP
=
15
WIP
=
2
WIP
=
1wk
WIP
=
1wk
Tech
Support
Queue
=
0
Queue
=
0

11. So, what is happening?
• Though not an originally stated vision or goal, the “Work in
Progress” (WIP) is being limited to # of team members
• At any time, one developer is assigned only one piece of work,
thereby achieving “One-Piece Flow”
• New work is only assigned when current work is
completed (or cancelled/stalled), and a team member is
available
• No wait state or switching costs at an individual level
• Smaller lead time for bugs (in contrast to lead time for SP)
• The process is allowing ‘weekly deployment’ of each of the
hot fixes
• Finally, the flexibility gained is not a zero-sum game – there is
no penalty on performance in rest of the process

12. Did this move the needle?
• Bugs addressed each quarter
• Quality of bug fixes
• “Homes” for bugs
• Total bugs open
• Open days open
• People motivation

13. Shift from SPs to Cumulative Hot Fixes
while maintaining High Quality
Maintenance Releases (Service Packs, Patches, Hot Fixes)
Q3 2003 Through Q4 2006 (Fiscal Year)
0
2 3 4 4 4 4
2 2 2 1 2 3
1
5 2 1
5
3
1 0
0 1 0
0
0
1
0
2
8
11
16
18
20
62
28 27
26
26 24
28
21
100
92
87
96
88
92
85
80
97 96
93
100
94
91
7
12
15
25 25 25
30 30
28
27
26
32
22
66
0
10
20
30
40
50
60
70
Q3 03 Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05 Q3 05 Q4 05 Q1 05 Q2 06 Q3 06 Q4 06
0
10
20
30
40
50
60
70
80
90
100
Service Packs Patches Hot Fixes % Successful
Percentage of released
Maintenance Releases
(Service Packs,
Patches, Hots Fixes)
that addressed
customer reported
Total Maintenance
Releases (Service
Packs, Patches, Hot
Fixes) for this
quarter.
Service
Packs
Patches
Hot Fixes

14. Increase in bugs with “homes”
Customer Direct and Indirect % w/Homes - Historical
Week Ending February 03, 2006
55
46 45 47
44 45 45 44
55
52 54 56
35
45
39 38
41
51
46
31
40 40
59
63
60 61 61
68 69
74 72
69 69
55
73
61
53
5655
48
54 55
52 51 51 51 50 49
54 55
47 47
11
15
18 16
47
53 55
43 42 43
75
82
76 76 75
82 83 82 81
71 72
65
87
82
75
80
55
47
51 51 50 48 48 48 48 48
53 54
51 51
21
27 27 25
44
52 51
43 41 42
67
72
68 69 68
75 76 78 76
70 71
60
80
71
64
68
44
47
0
10
20
30
40
50
60
70
80
90
100
5/6/2005
5/13/2005
5/20/2005
5/27/2005
6/3/2005
6/10/2005
6/17/2005
6/24/2005
7/1/2005
7/8/2005
7/15/2005
7/22/2005
7/29/2005
8/5/2005
8/12/2005
8/19/2005
8/26/2005
9/2/2005
9/9/2005
9/16/2005
9/23/2005
9/30/2005
10/7/2005
10/14/2005
10/21/2005
10/28/2005
11/4/2005
11/11/2005
11/18/2005
11/25/2005
12/2/2005
12/9/2005
12/16/2005
12/23/2005
12/30/2005
1/6/2006
1/13/2006
1/20/2006
1/27/2006
2/3/2006
Direct Indirect Goal (80% w/Homes) Combined
Percent with Releases Identified: 56% (Driect), 80% Indirect, 68% (Combined)

15. Total Bugs Open
Severity Across Products (Historical)
Week Ending March 31, 2006
222
191
243
213
224
230
236
201199
210211
196197195
161
147146146146
132
122
131
121
127
111
127125
148150
65
605857
53
57
626466676768
71
747575
6866
494746
5049495052
60
63
69707069
7473
68
6160
5555
4749 5149
44
3538
323332343536
0
10
20
30
40
50
60
70
80
90
100
110
120
130
31-D
ec-0301-O
ct-0429-O
ct-0419-N
ov-0410-D
ec-0431-D
ec-0421-Jan-0511-Feb-0504-M
ar-0525-M
ar-0515-Apr-0506-M
ay-0527-M
ay-0517-Jun-0508-Jul-0529-Jul-0519-Aug-0509-Sep-0530-Sep-0521-O
ct-0511-N
ov-0502-D
ec-0523-D
ec-0513-Jan-0603-Feb-0624-Feb-0617-M
ar-06
0
20
40
60
80
100
120
140
160
180
200
220
240
260
Total Severity 1 Severity 2 Severity 3 Severity 4 Severity 5

16. Average Days Open
Severity: Average Days By Severity (Historical)
Week Ending March 31, 2006
203
155
136
244242
246246244
248
234235
226223
232
244
260
268
275275
299
232
201
210207
179
165
172
156
162
180
168
163
156
152
147
142
132135
139
143
148
140
144147146144
155
135
145
152
148
157156156
149
133
144
140
144147
154155157
163166
177179
183182
164162165
154
146
141
149
123
117
134134
123
0
25
50
75
100
125
150
175
200
225
250
275
300
325
350
375
400
425
450
475
500
31-D
ec-0301-O
ct-0429-O
ct-0419-N
ov-0410-D
ec-0431-D
ec-0421-Jan-0511-Feb-0504-M
ar-0525-M
ar-0515-Apr-0506-M
ay-0527-M
ay-0517-Jun-0508-Jul-0529-Jul-0519-Aug-0509-Sep-0530-Sep-0521-O
ct-0511-N
ov-0502-D
ec-0523-D
ec-0513-Jan-0603-Feb-0624-Feb-0617-M
ar-06
0
25
50
75
100
125
150
175
200
225
250
275
300
Avg Days Open Severity 1 Severity 2 Severity 3 Severity 4 Severity 5

17. People motivation
12 12 12 12 12
10 10 10
4 4 4 4 4
6 6 6
16 16 16 16 16 16 16 16 16 16 16 16
0
2
4
6
8
10
12
14
16
18
Resource Staffing
Sustaining Mainline Approved Headcount
• Started with 16 people dev team
• We had zero attrition in the team
• Once the backlog started coming down, engineers were ramped
off the team to do new features
• Eventually dismantled the team and rolled-up engineers into dev
teams when backlog came down to single digits

18. What is Kanban?
• Kanban (literally signboard or billboard) is a scheduling system
for lean and just-in-time (JIT) production. According to its
creator, Taiichi Ohno, kanban is one means through which
JIT is achieved.
• Kanban is not an inventory control system; it is a scheduling
system that helps determine what to produce, when to
produce it, and how much to produce.
• The need to maintain a high rate of improvement led Toyota to
devise the kanban system. Kanban became an effective tool
to support the running of the production system as a whole.
• In addition, it proved to be an excellent way for promoting
improvements because reducing the number of kanban in
circulation highlighted problem areas.
hHps://en.wikipedia.org/wiki/Kanban

19. A Kanban System at Toyota
dealership
hHps://twitpic.com/het3u

20. How does it work?
hHp://www.toyota-­‐global.com/company/vision_philosophy/toyota_producLon_system/just-­‐in-­‐Lme.html

21. How Kanban helps achieve “Just-
in-Time”
• For
example,
to
efficiently
produce
a
large
number
of
automobiles,
which
can
consist
of
around
30,000
parts,
it
is
necessary
to
create
a
detailed
producLon
plan
that
includes
parts
procurement.
Supplying
"what
is
needed,
when
it
is
needed,
and
in
the
amount
needed"
according
to
this
producLon
plan
can
eliminate
waste,
inconsistencies,
and
unreasonable
requirements,
resulLng
in
improved
producLvity.
hHp://www.toyota-­‐global.com/company/vision_philosophy/toyota_producLon_system/just-­‐in-­‐Lme.html

22. Kanban for Software
• Visualize the Workflow: Represent the work items and
the workflow on a card wall or electronic board
• Limit Work-in-Progress (WIP): Set agreed upon limits
on how many work items are in progress at a time
• Measure and Manage Flow: Track work items to see if
they are proceeding at a steady, even pace
• Make Process Policies Explicit: Agree upon and post
policies about how work will be handled
• Use Models to Evaluate Improvement Opportunities:
Adapt the process using ideas from Systems Thinking,
Deming, etc.
Kanban: Successful Evolutionary Change for your Technology Business – David Anderson

23. Why Kanban in Software
Engineering?
Don’t
build
features
that
nobody
needs
right
now
Don’t
write
more
specs
than
you
can
code
Don’t
write
more
code
than
you
can
test
Don’t
test
more
code
than
you
can
deploy
hHps://leanandkanban.files.wordpress.com/2009/04/kanban-­‐for-­‐sohware-­‐engineering-­‐apr-­‐242.pdf

24. What did We learn?
• Process improvement should be driven
by business needs – and NOT
because some process looks sexy!
• Don’t let a process limit your potential –
think beyond gurus!
• Don’t let absence of a process limit your
potential – do whatever it takes to serve
customer better!

25. Resources
• Was that Kanban? - http://finance.groups.yahoo.com/group/kanbandev/message/4131 and
http://finance.dir.groups.yahoo.com/group/kanbandev/message/4166
• http://refcardz.dzone.com/refcardz/getting-started-kanban
• Ship early and ship twice as often,
https://www.facebook.com/notes/facebook-engineering/ship-early-and-ship-twice-as-often/
10150985860363920
• How we built Flickr,
http://www.plasticbag.org/archives/2005/06/cal_henderson_on_how_we_built_flickr/
• Continuous Deployment at IMVU: Doing the impossible fifty times a day,
http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-
times-a-day/
• A New Version of Google Chrome now due every six weeks,
http://techcrunch.com/2010/07/22/google-chrome-versions/
• In praise of continuous deployment: The WordPress.com story,
http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/
• CONWIP, https://en.wikipedia.org/wiki/CONWIP
• Kanban applied to Software Development: from Agile to Lean,
http://www.infoq.com/articles/hiranabe-lean-agile-kanban

26. Photo by orcmid - Creative Commons Attribution License https://www.flickr.com/photos/91555706@N00
Created with Haiku Deck

27. Photo by Neil Melville-Kenney - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/27774670@N07
Created with Haiku Deck

28. Photo by Brett Whaley - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/21920030@N02
Created with Haiku Deck

29. Photo by raneko - Creative Commons Attribution License https://www.flickr.com/photos/24926669@N07
Created with Haiku Deck

30. Photo by Photosightfaces - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/30595068@N06
Created with Haiku Deck

31. Photo by geezaweezer - Creative Commons Attribution License https://www.flickr.com/photos/33909206@N04
Created with Haiku Deck

32. Photo by familymwr - Creative Commons Attribution License https://www.flickr.com/photos/36196762@N04
Created with Haiku Deck

33. Photo by Mónica, M - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/52329444@N00
Created with Haiku Deck

34. Photo by joiseyshowaa - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/30201239@N00
Created with Haiku Deck

35. Photo by `James Wheeler - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/24128704@N08
Created with Haiku Deck

36. Photo by William Christiansen - Creative Commons Attribution License https://www.flickr.com/photos/56222080@N04
Created with Haiku Deck

37. Photo by merrycrafts - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/10314331@N08
Created with Haiku Deck

38. Photo by susanvg - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/28362388@N00
Created with Haiku Deck

39. Photo by monkeyc.net - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/73584213@N00
Created with Haiku Deck

40. Photo by Romain [ apictureourselves.org ] - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/72898690@N00
Created with Haiku Deck