Software risk impact is more predictable than you might think. This session discusses similarities of uncertainty in various industries and relates this back to how we can measure and analyze impediments and risk for agile software teams.

1. Risk, Options and Cost of Delay
Troy Magennis
LKNA 2014 San Francisco. May 2014

3. risk events
1 2 3
Performance AND
Vendor Delay
Performance OR
Vendor Delay
Nothing Goes
Wrong
Time
Probability

4. Definition: Risk
The impact of uncertainty
on an outcome

5. Technical Risk
Financial
Risk
Market Risk
• Real Options
• Right Staff / liquidity
• Dev Practices
• Dependencies
• Constraints
• Lean Startup
• Agile Processes
• Competitive
Awareness
• Having
funding/cash
• Having a
strategy
• Economic
prioritization
• Real Options
“Aleatory Risk”
Cannot be reduce by more info

6. Delay
(Technical Risk)
Low
Adoption
(Market Risk)
Low
Cashflow
(Financial Risk)
Less
Resources
(Financial Risk)
Risk
Positive
Feedback
Loop

7. Key Point
Occurrence of a risk Increases
exposure to other risks
Break the chain early
AKA: Early and meaningful
contact with enemy – RISK
(source: quote from Reinertsen, but sources from US marines?)

10. Correlation != Causation
We can see average flight delay
matches the shape of “Late
Aircraft,” but don’t yet know why…

11. Key Point
Serialized dependencies cascade
delays, but are not the root cause –
Why was the aircraft late?
The later you are, the later you get.

12. Four people arrange a
restaurant booking after work
Q. What is the chance they
arrive on-time to be seated?

13. Commercial in confidence
Person 1 Person 2 Person 3 Person 4
1in16EVERYONEisON-TIME
15TIMESmorelikelyatleastonpersonislate

14. 1
2
3
4
5
6
7
Team Dependency Diagram

15. 1 in 2n
or
1 in 27
or
1 in 128

16. 7 dependencies
1 chance in 128

17. 6 dependencies
1 chance in 64

18. 5 dependencies
1 chance in 32

21. Key Point
Risk of being impacted
decreases by half for every risk
vector/factor removed
But, not all risks have the same
likelihood (or impact)…

22. Frequency
Recency
Impact

23. If you haven’t seen an event after
testing for it n times, you can be
95% sure that its probability of
happening is less than
3/n
References: Wikipedia: Statistical Rule of Three and Thanks to John Cook: Estimating the chances of something that hasn’t happened yet,
http://www.johndcook.com/blog/2010/03/30/statistical-rule-of-three/
The Math: (1-p)n = 0.05 for p. Taking logs of both sides, n ln (1-p) = ln(0.05) ≈ -3.
Since log(1-p) is approximately -p for small values of p, we have p ≈ 3/n.

24. Statistical Rule of Three
• Example: Proofreading a
book, you find no
grammatical errors in n pages
• Error decreases as a
proportion to the number of
independent test cases
examined
• It hard to be
independent!
n percentage
20 15% (3/20)
100 3% (3/100)
200 1.5% (3/200)
500 0.6% (3/500)
1000 0.3% (3/1000)
0.00000
0.10000
0.20000
0.30000
0.40000
0.50000
0.60000
0.70000
0.80000
1
21
41
61
81
101
121
141
161
181
201
221
241
261
281
301
321
341
361
381
401
421
441
461
481
p

25. ‘s Absence of Evidence isn’t
Evidence of Absence
But, it does demonstrate the
occurrence is rare with
growing certainty
Depends on consequence….
Ps. The most common
Black Swan is project
on-time delivery!

26. CONSEQUENCE
MATTERS

27. Capture
Actual
Impacts
Calculate
“Impact”
Order from
highest to
lowest
Discuss, Root
cause Top 10
Prioritize
Sum of Days
impacted for 3
last months
Sum of Days
impacted for 3
last months
Category
Start
End

28. “Value”
Cost of Delay
Product 1
Product 2
Product 3
Complete
Order?
3
2
1
“Time”
Remaining
Time/Effort to solve
Economic Prioritization – same time, different value

29. Product 1
Product 2
Product 3
1
2
3
Economic Prioritization – same value different time
“Value”
Cost of Delay
Complete
Order?
“Time”
Remaining
Time/Effort to solve

30. W.S.R.F. =
Prioritization Heuristic
to optimize reward
“Do Highest First”
Impact of risk
Time to resolve/mitigate
Weighted Shortest Risk First
Sum of delay time
of same risk causes
over the last 3 (?)
months
Effort estimate of
the resolution time
of risk root cause

32. All Sheep in Scotland Are Black
• A psychologist, a biologist, a mathematician, and a physicist were riding
a train through the Scottish countryside. Looking out the window, they
all noticed a lone black sheep on a hill.
• The psychologist intoned, “Well, what do you know. I didn’t realize the
sheep in Scotland were black.”
• The biologist corrected him, saying, “You don’t know that all the sheep in
Scotland are black – just some of them.”
• Piping in, the mathematician retorted, “Tut, tut, tut, to be correct you
must say, ‘At least one’ sheep in Scotland is black.”
• The physicist had the last word, though, stating, “Gentlemen, all we know
with certainty based on our observations is that at least one sheep in
Scotland is black on at least one side, at least part of the time.”
• Moral: There are hard and soft sciences, and extrapolation is not always
justified.
http://creationsafaris.com/humor.htm

33. Total
Story
Lead
Time
30
days
Story / Feature Inception
5 Days
Waiting in Backlog
25 days
System Regression Testing & Staging
5 Days
Waiting for Release Window
5 Days
“Active Development”
30 days
Pre
Work
30
days
Post
Work
10
days
9 days (70 total)
approx 13%

34. THE SHAPE OF CYCLE TIME
What distribution fits cycle time data and why…

35. If we understand how cycle time is
statistically distributed, then an
initial guess of maximum allows an
inference to be made
Alternatives -
• Borrow a similar project’s data
• Borrow industry data
• Fake it until you make it… (AKA guess range)

36. Why Weibull
• Now for some Math – I know, I’m excited too!
• Simple Model
• All units of work between 1 and 3 days
• A unit of work can be a task, story, feature, project
• Base Scope of 50 units of work – Always Normal
• 5 Delays / Risks, each with
– 25% Likelihood of occurring
– 10 units of work (same as 20% scope increase each)

37. Normal, or it will
be after a few
thousand more
simulations

38. Base + 1 Delay

39. Base + 2 Delays

40. Base + 3 Delays

41. Base + 4 Delays

42. Base + 5 Delays

44. Exponential Distribution (Weibull shape = 1)
The person who gets the work can complete the work
Teams with no external dependencies
Teams doing repetitive work E.g. DevOps, Database teams,

45. Weibull Distribution (shape = 1.5)
Typical dev team ranges between 1.2 and 1.8

46. Rayleigh Distribution (Weibull shape = 2)
Teams with MANY external dependencies
Teams that have many delays and re-work. E.g. Test teams

47. What Distribution To Use...
• No Data at All, or Less than < 11 Samples (why 11?)
– Uniform Range with Boundaries Guessed (safest)
– Weibull Range with Boundaries Guessed (likely)
• 11 to 30 Samples
– Uniform Range with Boundaries at 5th and 95th CI
– Weibull Range with Boundaries at 5th and 95th CI
• More than 30 Samples
– Use historical data as bootstrap reference
– Curve Fitting software

48. Probability Density Function
Histogram Weibull
x
1201101009080706050403020100
f(x)
0.28
0.24
0.2
0.16
0.12
0.08
0.04
0
Scale – How Wide in
Range. Related to the
Upper Bound. *Rough*
Guess: (High – Low) / 4
Shape – How Fat the
distribution. 1.5 is a
good starting point.
Location – The
Lower Bound