APM event hosted by Midlands Branch on 24 May 2023. Speaker: Andy Nolan Estimating is the process of determining the level of cost, effort, resources and schedule you need to successfully implement your project. An accurate budget and schedule has been shown to improve project success - estimating is not only required, it's essential for a successful business. This event was held on 24 May 2023. Estimating appears in many forms in a project's life from developing the initial budget and schedule, to estimating the duration of tasks in your plan, through to estimating risks and uncertainties. The Rolls-Royce Heritage centre were available before the event for those wanting to explore the history of Rolls-Royce Aero engine development, this included a large collection exhibit engines from early day piston to modern large turbojet engines. Attendees had the opportunity to discover and learn about Rolls-Royce products via the Heritage Centre, Network with fellow PM professionals, and grow knowledge of how important estimation is within the project Management function. https://www.apm.org.uk/news/the-art-of-estimating/

1. Welcome from the APM Midlands Branch
Derby, May 2023

2. The Art of Estimating
24 May 2023
Derby
apm.org.uk/event

3. Agenda
© 2023 Association for Project Management 3
17.30 Registration, Networking
& Heritage Centre
18.30 Welcome & Introductions
18.35 The Art of estimating
- Andy Nolan
- Sophie Osborne
20.20 Q&A
20.30 Thanks & Close

4. Housekeeping
© 2022 Association for Project Management 4
HSE
- Fire Points
- Smoking
- Toilets
- Phones to Silent
Feedback

5. The Art of Estimating
Issue 10.00
The Art of Estimating
Andy Nolan, Sophie Osborne

6. APM / ACostE
Estimating Guide
Available from
the APM
Bookshop
MG

7. Section 1
Introduction
SO

8. Benefits
Business
Cases
Estimating Risk
Schedule
Cost
Resource
When is estimating used?
✓Project Launch
✓Then throughout the project life
Where is estimating used?
✓Benefits Management
✓Risk Management
✓Schedule Management
✓Cost Management
✓Resource Management
SO

9. Business and project
requirements
Project closeout
checklist
Step 3
Develop Estimate
Step 1
Establish Estimating
Capability
Tools
Templates
Checklists
Guides
Training
Historic data
Historic estimates
Lessons Learnt
Step 5
Monitor & Control
Track estimate KPI
& assumptions
Update estimate
Raise change
requests as
needed
Step 6
Project Close
Review actual
against estimate
and identify root
cause for
deviations
Archive data and
estimate for future
projects
Estimate baselined &
Agreed budget,
resource and schedule
Change
Requests
Data, final
estimate &
Lessons Learnt
Step 4
Verify & Validate
Estimate
Step 2
Plan the estimate
Estimates, Actuals
& Lessons Learnt
Project Review
checklist
AN

10. Section 2
Why Estimates Fail
AN

11. Roads Defence
Bridges
and
Tunnels
Energy Rail Dams IT
Cost
overrun
20% 28% 34% 36% 45% 90% 107%
Schedule
Overrun
38% 49% 23% 38% 45% 44% 37%
A study of projects from
around the world by the Major
Project Association. Shows
the cost and schedule
overrun for various business
domains.
SO

12. Question?
Why do estimates fail
for you?
SO

13. Requirements, scope, and assumptions
Risk, Uncertainties and 3-point estimates
Data and Calibrated Parametric tools
Estimators Competency
Estimate Documentation
Estimate Verification & Validation
Time and resource to develop the estimate
Addressing cost/ Schedule challenges
The estimate purpose was understood
Estimating Techniques Used
Factors that Contribute to Estimate Success/Failure
A regression fit
through the root
causes for estimate
success / failure.
Chart based on the
analysis of 483
completed projects.
AN

14. Root cause
analysis
Over 120 root causes
assessed and
summarised into the
following 4 groups Culture &
Behaviours
50%
Risk &
Uncertainties
22%
Governance
& Assurance
16%
Capability
12%
AN

15. 15
Can you see
blue circles
or spirals?
SO

16. Do the 2
squares
marked
square-A &
Square-B
look the
same shade
of grey?
SO

17. Optimism
According to
research 80% of
people are
optimistic.
Usual about
their own
abilities.
AN

18. Question?
How might you
reduce the effects of
biases?
AN

19. Section 3
Techniques
AN

20. Techniques
There are a wide
variety of estimating
techniques. Their use
depends on (1) how
much data you have
(2) how much time
you have and (3) What
level of accuracy you
need.
3: Decision
Conferencing
(planning
Poker)
11: Monte-
Carlo
10: Bottom Up
4: Comparative
5: Buckets
(pick lists)
6: Beans
(Agile)
When you have
no data
When you have
little data
When you have
much data
Boosters
9: Parametric
7: Forecasting
8: Shopping
Lists
12: Multiple
Estimates
2: Wisdom of
the Crowd
1: Judgement
How much data you Have
How
much
time
you
Have
AN

21. ROM +/-40% +/-20% +/-10% +/-5%
% of total project
effort to produce
the estimate
0.01% -
0.05%
0.05% -
0.15%
0.15%-
0.25%
0.25%-
0.45%
0.45% -
2.0%
How long does it take to
develop an estimate?
Estimating effort is
determined by the size of
your project and the
accuracy you need.
The effort is needed to
generate both the estimate
inputs and outputs
0.5%
2.0%
1.5%
1.0%
RFI
RFP
RFQ
Contract
Estimate maturity
SO

22. Section 4
Judgement Based
Techniques
SO

23. Judgement
Judgment is one of the most popular Techniques we use, but how reliable is it? In the
hands of experts, Judgment is quick and reasonably accurate but prone to an individual's
biases and opinions. We use it when we have no data or tools. Or because we do not have
the time to develop a robust estimate.
Where
Early stage or low risk estimates
When
No data or tool
or time for a
robust estimate
Need
Domain
experience
Good For
Quick but low
maturity
estimates
Boosters
Experience,
Bottom-up and
Monte-Carlo
SO

24. How many
people live in
the UK?
MG
AN

25. How tall is the
tallest pyramid
of Giza
(Egypt)?
MG
AN

26. How many $M’s of
sweets are sold
each year in the
USA in the 2 weeks
leading up to
Halloween?
MG
AN

27. 0%
100%
200%
300%
400%
500%
600%
0 20 40 60 80 100
Mean
%
Error
Estimator Confidence
Mean %Error vs Confidence
Be Confident
This chart is a study of 3893 guesses showing a
relationship between estimator Confidence and
their guess accuracy. Confidence is scored
between 0 (no idea) and 100 (certain). So, only
use Judgement when you are Confident.
AN

28. -78%
-75% -74%
-70%
-66% -65% -66%
-69%
-62% -62%
-55%
22%
25% 26%
30%
34% 35% 34%
31%
38% 38%
45%
-80%
-60%
-40%
-20%
0%
20%
40%
0 - 10 10 - 20 20 - 30 30 - 40 40 - 50 50 - 60 60 - 70 70 - 80 80 - 90 90 - 100 100
%
Of
Population
Confidence
Population who over /under estimated vs confidence
Under estimate Over Estimate Median % Error
% of population
that under
estimated
% of population
that over-
estimated
overall 70% of people under-
estimated and how much they under-
estimated can be determined from
their confidence.
SO

29. Wisdom of the Crowd
Where
Low – middle maturity estimates
Need
Many people
with experience
Good For
Minimising the
effects of
judgement error
and biases
Boosters
More people,
more experience
or use on Bottom-
Up estimates
The Wisdom of the Crowd Technique relies on having many experts. Each person makes a
guess anonymously, and you take the middle (median) guess. In general, the more people in
the “crowd”, the higher the maturity of your estimate. It has been shown to reduce the
effects of biases and minimises the errors made by the judgement of a single person.
When
There is no data
or tool or the
time to develop a
robust estimate
SO

30. History
Sir Francis Galton (1822 –1911) was an English
Victorian statistician. At a 1906 country fair in
Plymouth, 800 people participated in a contest to
estimate the weight of an Ox. The mean guess was
accurate within 1% of the right answer. Sir Francis
Galton discovered that there is wisdom in the
crowd. But there are several conditions:
1. There must be some knowledge in the crowd. It
won't work if everyone is making "wild"
guesses.
2. There must be diversity of opinion. The
technique is not looking for consensus.
3. Everyone must be free to express their opinion
anonymously.
AN

31. Why does it work?
Many heads are better than one, but why?
It is about errors cancelling out. Assuming
there is no general tendency for the team
to over or underestimate, then given a
crowd of people, some will overestimate,
some under, but they might be about right
“on average”.
Based on 3760 guesses, we randomly
formed groups of different sizes. For each
group, we calculated the Median % Error
and their average confidence. The chart
shows the results of 20 million simulations.
-100%
-50%
0%
50%
100%
150%
200%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Median
%Error
Median % Error vs Crowd Size
High
confidence
group
Low
confidence
group
The number of
people in the
“Crowd”
AN

32. You can improve your estimate score by adding
more people, or by increasing the teams
confidence e.g. adding experts
SO

33. Decision Conferencing
When
There is no data
or tool or the
time to develop a
robust estimate
Good For
Team learning
and revealing
estimate
assumptions
Decision Conferencing, also known as Planning Poker and Wideband Delphi, is like the
Wisdom of the Crowd, in that you need many experts. However, in Decision Conferencing,
the team will debate the guesses, and if new information is revealed, the team may have
another round of guessing.
Boosters
More people,
more expereince
or use on Bottom-
Up estimates
Need
Many people
with experience
Where
Low – middle maturity estimates
SO

34. History
Decision conferencing goes by several names,
Planning Poker and Wideband Delphi.
The Wideband Delphi Estimating method is a
consensus-based technique for estimating. It
derives from the Delphi method which was
developed in the 1950-1960s at the RAND
Corporation as a forecasting tool.
Planning Poker was first defined and named by
James Grenning in 2002 and later popularized by
Mike Cohn in the book Agile Estimating and
Planning
Mike Cohn
AN

35. Process
As with the Wisdom of the Crowd, everyone makes
a guess, then reveal their results. The people with
the lowest and highest answers explain their
rationale. The group debate the rationale. The
group then have another round of guessing. They
may need to go around this loop several times
until there is some level of convergence.
Person 1
Guess=12
Person 2
Guess=5
Person 3
Guess=30
Person 4
Guess=15
Person 5
Guess=8
Round 1
Person 1
Guess=12
Person 2
Guess=11
Person 3
Guess=25
Person 4
Guess=15
Person 5
Guess=10
Round 2
Person 1
Guess=12
Person 2
Guess=11
Person 3
Guess=15
Person 4
Guess=15
Person5
Guess=12
Round 3
Everyone makes a guess
anonymously
Results are revealed at
the same time
Min and Max estimates
are debated to reveal
assumptions
Is there
consensus?
Stop
Have you
done 3
rounds?
Yes Yes
No
No
AN

36. Section 5
Monte-Carlo
AN

37. The History of Monte-Carlo
Enrico Fermi (physicist), Stanislaw Ulam (mathematician) and Jon
Von Neumann (Computer specialist) were working on the
Manhattan project in 1943.
Ulan was sick and was playing the card game Solitaire
(Patience). He was having trouble winning so he decided to use
mathematics to predict the probability of winning. It turned out
to be an “intractable problem” that could not be solved with
discrete mathematics.
He proposed the idea they could simulate the game on the new
ENIAC computer. If they ran it a hundred times, they could
simply count how often they won. The idea was never tested out
on Solitaire, but was recognised as a solution to the complex
maths they needed at the time.
Stanislaw Ulam
SO

38. Jon Von
Neumann
He headed the development
of the first computers in the
1940’s. ENIAC (Electronic
Numerical Integrator and
Computer) was amongst the
earliest electronic general-
purpose computers. It had a
speed of one thousand times
faster than that of electro-
mechanical machines.
Klára Dán
von Neumann
Klára is considered to be the first
computer programmer in the world.
One of the first programmes written
was a Monte-Carlo simulation. The
800 instruction programme took a
month to run in part because the
ENIAC was “down” 23 hours out of
every 24.. In total only three
simulations were run.
AN

39. Private | © 2022 Rolls-Royce | Not
Subject to Export Control
With Schedule Risk Analysis, we
allocate “Roulette Wheels” to
activities in our plan that have
uncertainties or risks
For each simulation, Monte-Carlo
takes the random value generated
from each Roulette Wheel, then add
them up. This will be done
thousands of times.
Allocate Roulette
Wheels
Frequency
Estimated End Date
A “Frequency” plot
of simulated times.
AN

40. Fact
Assumption
Risk
Uncertainty
I need to take a flight.
I assume I will fly from London.
There is a 10% risk I have to stay in
short term parking costing £100
Flights can be £500 to £1000
SO

41. Fact
Assumption
Risk
Uncertainty
I am going to the Maldives.
Assume my “mother-in-law-to-be”
coming with us!
There is a 10% chance we will take
an excursion costing £100.
Spa treatments will cost between
£50 and £100 a day.
SO

42. Risk
Uncertainty
Assumptions
Monte Carlo
Reserve
Confidence
3-Point
Estimate
Sensitivity
Analysis
Monte-Carlo
You build up a model
from Facts, Assumptions,
Risks and Uncertainties.
Then simulate your
project many thousands
of times. From the output
we can generate 3-point
estimates and derive a
meaningful level of
reserve.
Facts
SO

43. Frequency
Estimate
Max based on 90%
confidence
Min based on 10%
confidence
Most Likely (Mid
Point) based on
50% confidence
50%
10%
90%
Percentile Percentile is in the range 0% - 100%.
Often used as a statement of the
probability of success e.g. P(80).
Precision: the width of the +/- values i.e. +/- 40%
Precision +/-
Confidence Interval:
represents the probability
that the Actual value lies
between the Min and Max
Confidence
Interval
Accuracy: an estimate is accurate IF the
actual outcome lie inside the Min-Max range
Actual
Reserve: covers residual risks and
uncertainties.
Reserve
AN

44. Facts
Assumptions
Your
estimate
Facts
Assumptions
Facts
Assumptions
Facts
Assumptions
Facts
Assumptions
Facts
Project
Close
Maintain the estimate through life
Maintaining your
Monte-Carlo Model?
Once we have an estimate, we
should maintain it through the life
of our project. Over time Risks,
Uncertainties and Assumptions will
be replaced by Facts. Update your
estimate and re-run the Monte-
Carlo model on the residual Risk
and Uncertainty.
Risk
Uncertainty
Risk
Uncertainty
Risk
Uncertainty
Risk
Uncertainty
Min Estimate
Mid Estimate
Max Estimate
The Cone of Uncertainty
AN

45. Section 6
We only know what we
know
AN

46. The Assumptions about a cup of tea
What could be
simpler then making
a cup of tea? So if
you were to make
some tea, what
assumptions will you
be making?
SO

47. 1. Assume people will want tea
2. Assume tea is for 2
3. Assume I have a kettle
4. Assume the kettle works
5. Assume I have clean water
6. Assume the electricity is on
7. Assume I have water
8. Assume I have cups
9. Assume the cups are clean
10. Assume I have a tea-pot
11. Assume I have a tea strainer
12. Assume the pot is warmed first
13. Assume people want “normal” tea
14. Assume I have some “Normal” tea
15. Assume how long to “stew” the tea
16. Assume people will want milk (not a substitute)
17. Assume people are not lactose intolerant
18. Assume no-one wants sugar or a sugar
substitute
19. Assume I “stew” the tea for 3 minutes before
removing the tea
20. Assume I add the milk second not first
21. Assume I have a spoon
22. I assume I have biscuits/cake
23. You have enough time to drink it
24. Assume I am capable
25. Assume a clean spoon
26. Assume time to nake it
27. Assume I have paid the water bill
28. Can get access to kitchen
29. Can por water and don’t have a bad arm
30. I don’t burn myself
31. Assume I am not distracted
32. Assume cup does not leak
33. A Butler is available
34. Assume the dogs will want to go out
SO

48. Question
What are some of
the risks and
uncertainties
when estimating a
car journey?
AN

49. 1. Poor Traffic
2. Poor Weather
3. Car reliability
4. Getting lost
5. Road works
6. Accidents
7. Breakdowns
8. Inexperienced driver
9. Poor driving style
10. Special events
11. Diversions
12. Altitude
13. Poor road conditions
14. Traffic Lights
15. Police
16. Speed cameras
17. Night driving
18. Top speed of the car
19. Run out of fuel
20. Emergency toilet stop
21. Escaped pet in car
22. Escaped animals on the road
23. No change for toll
24. Bridge closure
25. Train crossing
26. No parking
27. Water crossing – wait for ferry
28. Run out of screen wash
29. Travel sickness
30. Need to go back for something
31. Licenced car
32. Valid insurance
33. Car not stolen
34. Obstruction e.g. debris
35. Riots
36. Major catastophy
37. Protestors
38. Lightning strike
39. Wheel fall off
40. Flat battery
41. Car charging time
42. Leave the house on time
43. Craze drivers on the road
44. Have a car
45. Flooding
46. Front door wont open
47. Defrost car
AN

50. Question?
How could you
improve the success
of identifying
assumptions, risks etc
AN

51. Section 7
Reserve for Unknowns
AN

52. We estimate when there
are consequences.
And, the greater the
consequences, the
“harder” we estimate
Why do we estimate?
SO

53. Known Unknown
Known
Known-Knowns
Things we know that we
know. Based on Facts
and Assumptions.
Known-Unknowns
Things we know we do
not know. Used to
determine
Reserve Type 1: based
on known Risks and
Uncertainties
Unknown
Unknown-Knowns
The things we don’t
know we know..
Reserve Type 2 BUT
may be treated through
change control
Unknown-Unknowns
The things we don’t
know we don’t know.
Reserve Type 2 BUT
may be treated through
change control
What is Reserve?
We need Reserve for the
risks and uncertainties we
know about and Reserve for
those we don’t!
Reserve Type-1 is for the risks
and uncertainties you know
about
Reserve Type-2 is for the
risks and uncertainties you
don’t know about.
SO

54. Monte-Carlo only considers the risks and
uncertainties you know about, not the ones you don’t
Frequency
Estimate
Max based on 90%
confidence
Min based on 10%
confidence
Most Likely (Mid
Point) based on
50% confidence
50%
10%
90%
Percentile
Reserve
AN

55. Minutes
Time
Minutes
Time
Max
Mid
Min
Minutes
Time
Max
Mid
Min
Imagine tracking the time it takes
you to drive home each day. Each
journey will contain different risks,
uncertainties and surprises.
The more data points you have,
the more unknowns will be
included in your future estimates.
Your new estimate can then be
based on the Min, Mid and Max
historical experiences (or 3-
standard-Ddeviations or P(10),
P(50) and P(90))
Technique 1: Historic Data
SO

56. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentile
Frequency
%Error = (Estimate / Actual) - 1
Historic Data Percentile
The problem with Technique 1 is
that you need to have data on
similar past projects. In this
Technique, we are using the
estimate process as the common
link between projects. This
technique bases your future
reserve on past estimate error
1. Track your historic estimate
performance, create a
frequency plot as shown and
then calculate a rolling total vs
variance (see line).
2. Generate your estimated
Most-Likely (Cost, Schedule or
Resource)
3. Derive your Reserve from the
chosen percentile
Technique 2: Historic Estimates
AN

57. Technique 3: Models Improving on Technique 2, create a
parametric model to predict
uncertainty based on key uncertainty
drivers. The model will need to be
calibrated to real project in order to
predict all types of Unknowns.
You can build models that consider
uncertainty parameters such as
• Estimate Maturity
• Project Priority
• Project Size
• Novelty (TRL)
• Complexity
• Product Type (IT, R&T, R&D etc)
AN

58. Section 8
Conclusions
AN

59. Conclusions
✓ Use Data/History
✓ Don’t Work Alone
✓ Use Cross Checks
✓ Add Reserve for Unknowns
SO

60. Any questions?
MG

61. © 2022 Association for Project Management 61
Thank you