APM presentation given by David Simms in Coventry on 23rd June 2016.

1. Why Can’t People Estimate:
Estimation Bias and Mitigation
With SEER Introduction
David Simms
Galorath International
© 2016 Copyright Galorath Incorporated

2. Key Points for today . . .
Estimates can be
better, reducing
bias & strategic
mis-estimation…
Parametrics can
help.
© 2016 Copyright Galorath Incorporated 2
Tempering
with an
“outside view”
can mitigate
some bias
Without care
estimates are
usually biased
(even with
experts)

3. Some of the most important
business decisions about a
project are made at the time of
minimum knowledge and
maximum uncertainty.

4. ESTIMATION & PLANNING:
An Complete Estimate Defined
• An estimate is the most knowledgeable statement you
can make at a particular point in time regarding:
• Effort / Cost
• Schedule
• Staffing
• Risk
• Reliability
• Estimates more precise with progress
• A WELL FORMED ESTIMATE IS A
DISTRIBUTION
4

5. Human Nature:
Humans Are Optimists
Harvard Business Review explains this Nobel
Prize Winning Phenomenon:
• Humans seem hardwired to be optimists
• Routinely exaggerate benefits and discount costs
Delusions of Success: How Optimism Undermines
Executives' Decisions (Source: HBR Articles | Dan
Lovallo, Daniel Kahneman | Jul 01, 2003)
5
Solution - Temper with “outside view”:
Past Measurement Results, traditional forecasting, risk
analysis and statistical parametrics can help
Don’t remove optimism, but balance optimism and
realism

6. Why Estimate?

7. Estimation Methods 1 of 2
Model
Category
Description Advantages Limitations
Guessing Off the cuff estimates
Quick
Can obtain any answer
desired
No Basis or substantiation
No Process
Usually Wrong
Analogy
Compare project with past
similar projects.
Estimates are based on
actual experience.
Truly similar projects must exist
Expert
Judgment
Consult with one or more
experts.
Little or no historical data
is needed; good for new or
unique projects.
Experts tend to be biased;
knowledge level is sometimes
questionable; may not be
consistent.
Top Down
Estimation
A hierarchical decomposition
of the system into
progressively smaller
components is used to
estimate the size of a
software component.
Provides an estimate
linked to requirements and
allows common libraries to
size lower level
components.
Need valid requirements.
Difficult to track architecture;
engineering bias may lead to
underestimation.

8. Estimation Methods 2 of 2
Model Category Description Advantages Limitations
Bottoms Up
Estimation
Divide the problem into
the lowest items.
Estimate each item…
sum the parts.
Complete WBS
can be verified.
The whole is generally bigger than the
sum of the parts.
Costs occur in items that are not
considered in the WBS.
Design To Cost
Uses expert judgment to
determine how much
functionality can be
provided for given
budget.
Easy to get under
stakeholder
number.
Little or no engineering basis.
Simple CER’s
Equation with one or
more unknowns that
provides cost / schedule
estimate.
Some basis in
data.
Simple relationships may not tell the
whole story.
Historical data may not tell the whole
story.
Comprehensive
Parametric Models
Perform overall estimate
using design
parameters and
mathematical
algorithms.
Models are usually
fast and easy to
use, and useful
early in a program;
they are also
objective and
repeatable.
Models can be inaccurate if not
properly calibrated and validated;
historical data may not be relevant to
new programs; optimism in parameters
may lead to underestimation.
8

9. Improving Estimation Is Important
• Estimation is critical for defining success for all
types of projects
• Many organizations treat estimation as a second
rate activity and do not invest in the process
• Everyone estimates….but the process and the
quality or success rate varies greatly
• Estimation and measurement go hand in hand –
BOTH contribute to improvement
• Having a repeatable estimation process is critical to
both estimating AND to successful projects

10. Cognitive Bias: How Fair Are We
(Source BeingHuman.org)
•Human beings exhibit inherent errors in
thinking
•Perception has more to do with our
desires—with how we want to view
ourselves—than with reality." Behavioral
economist Dan Ariely
Cognitive bias:
Tendency to make
systematic decisions
based on PERCEPTIONS
rather than evidence
•Our brains using shortcuts (heuristics) that
sometimes provide irrational conclusions
Researchers theorize in
the past, biases helped
survival
•from deciding how to handle our money
•to relating to other people
•to how we form memories“Bias affects
everything:
© 2016 Copyright Galorath Incorporated 10

11. The Planning Fallacy (Kahneman &
Tversky, 1979)
•Manifesting bias rather than confusion
•Judgment errors made by experts and
laypeople alike
•Errors continue when estimators aware of
their nature
Judgment errors
are systematic
& predictable,
not random
•Underestimate costs, schedule, risks
•Overestimate benefits of the same actions
Optimistic due to
overconfidence
ignoring
uncertainty
•“inside view” focusing on components rather
than outcomes of similar completed actions
•FACT: Typically past more similar assumed
•even ventures may appear entirely different
Root cause: Each
new venture
viewed as unique
© 2016 Copyright Galorath Incorporated 11

12. Reference Class Forecasting
Attempt to force the
outside view and
eliminate optimism
and
misrepresentation
Choose relevant
“reference class”
completed
analogous projects
Compute probability
distribution
Compare range of
new projects to
completed projects
Provide an “outside
view” focus on
outcomes of
analogous projects
© 2014 Copyright Galorath Incorporated 12
Best predictor of performance is actual performance
of implemented comparable projects (Nobel Prize
Economics 2002
Predicts outcome of planned action based
on actual outcomes in a reference class:
similar actions to that being forecast.

13. Dealing With the “Problem of
Assumptions”
• Assumptions are essential Write them down but…
Wrong assumptions can drive an estimate to
uselessness
• Use an assumption verification process
© 2016 Copyright Galorath Incorporated 13
1. Identify assumptions
2. Rank order
assumptions based on
estimate impact
3. Identify high
ranking assumptions
that are risky
4. Clarify high ranking,
high risk assumptions
& quantify what
happens if those
assumptions change
5. Adjust range of
SEER inputs to
describe the
uncertainty in
assumption

14. We should always listen to the experts . . . .
Shouldn’t we?
© 2016 Copyright Galorath Incorporated

15. Trouble Starts By Bias or Strategic Mis-
Estimation Ignoring Iron Triangle
• Typical Trouble: Mandated features needed within
specific time by given resources
• At least one must vary otherwise quality suffers and
system may enter impossible zone!
Quality
Resources Schedule
Scope (features, functionality)
Sometimes strategic mis-estimation
is used to get projects started or to win
Some customers think price to win is strategic mis-
estimation (it is not)

16. Plans Based on Assumptions About
Average Conditions Usually Go Wrong
• Executive desire to
“Show me the
number” forces
averages
• When average is used
to represent an
uncertain quantity….
• it ends up distorting the
results
• Because it ignores the
impact of the inevitable
variations
• Flaw may be fixed by
tools such as Monte
Carlo Analysis
• Adds a distribution &
probability to “the
number”
16

17. Adding Reality to Estimates –
Example – 2 (Source SEI)
Step Best Expected Worst
1 27 30 75
2 45 50 125
3 72 80 200
4 45 50 125
5 81 90 225
6 23 25 63
7 32 35 88
8 41 45 113
9 63 70 175
10 23 25 63
500
What would you forecast
the schedule duration to be
now?

18. Monte Carlo Analysis To Quantify
and Mitigate Bias
90% confidence,
project under 817
days
Almost guaranteed
to miss the 500
days
50%
confidence,
project will be
under 731 days

19. Example: Project Cost Alone Is not
The Cost of IT Failure (Source: HBR)
• Case Study: Levi Strauss
• $5M ERP deployment contracted
• Risks seemed small
• Difficulty interfacing with customer’s systems
• Had to shut down production
• Unable to fill orders for 3 weeks
• $192.5M charge against earnings
on a $5M IT project failure
“IT projects touch so many aspects of organization
they pose a new singular risk”
http://hbr.org/2016/09/why-your-it-project-may-be-riskier-than-you-think/ar/1

20. Causes of Project Failure
Source: POST Report on UK Government IT
Projects
Lack of a clear link between the project and the organisation’s key
strategic priorities, including agreed measures of success.
1.
Lack of clear senior management and ministerial ownership and leadership2.
Lack of effective engagement with Stakeholders3.
Lack of skills and proven approach to project management and risk
management
4.
Lack of understanding of and contact with the supply industry at senior levels
within the organisation
5.
Evaluation of proposals driven by initial price rather than long-term
value for money (especially securing the delivery of business benefits)
6.
Too little attention to breaking development and implementation into
manageable steps
7.
Inadequate resources and skill to deliver the total delivery portfolio8.
Source: POST Report on UK
Government IT Projects

21. Fallacy of Silent Evidence
What about what we don’t know?
How confident would you feel if the Silent Evidence was visible?

22. What is Parametric Estimating?
• Parametric Approach:
• Employs cost estimating relationships
• Uses mathematical equations
• Allows estimation based on measurement of key
parameters that influence the relationship
• Based on technical or physical characteristics of the
product and production environment
• SEER’s Parameters Cover 5 Key Areas
• People
• Process
• Technology
• Platform (Environment)
• Size
22

23. Draw Out Range By Obtaining 3
Estimates
• Optimistic value (sopt)
• Most likely value (sm)
• Pessimistic value (spess)
• Expected value (EV)
© 2016 Copyright Galorath Incorporated 23
6
)4( pessmopt sss
EV



24. Sophisticated Schedule Modeling:
Essential For Viable Project Plans
For a given Size, Complexity and Technology
Work Expands
To Fill Time
(Work expands due to lack
of pressure)
Minimum Time
To Complete
(Effort reduces to
reduce schedule)
Optimal Effort for staff
(Lower Effort for
Longer Schedule)
Effort Increase
due to Longer Schedule
Calendar Time
EffortMonths
Reasonable Solution
Range
Software Solution

25. Range vs. Point Estimates
(Source US Army)
RangeofRisk&Uncertainty
Estimating Accuracy
Trumpet
Technical and Program Maturity
ROM-30%to+75%
Parametric-10%to+20%
Analogy-15%to+30%
Engineering
-5%to+15%
Actual
-3%to
+10%
Target Cost
Point estimate is most likely within range
estimate
with higher potential for cost increase
Range estimate provides a degree of risk and
uncertainty
+75%
-30%
BA
Materiel
Solution
Analysis
Systems Acquisition Sustainment
Technology
Development
Production &
Deployment
Operations &
Support
Engineering and
Manufacturing Development
C
Pre-Systems Acquisition

26. What can a parametric model tell
you?
26
What is
likely to
happen
Feel lucky?
Firm Fixed
Price?
Understand the risk before you commit!

27. Difference Between Bottoms Up
and Parametric
• “Bottom-up” or Manual Approach (Detailed Work
Measurement)
• Measures everything directly and exactly
• Little or no analysis of alternatives
• Predictive value is limited at best
• Parametrics
• Measures or estimates key parameter values and
ranges
• Understand (mathematically) their effect on time,
effort
• Once relationships are known, parameter values can
be changed and the effects evaluated
© 2016 Copyright Galorath Incorporated

28. Example: Parametric Estimate
Compared With History

29. SEER MAJOR Components
• SEER-SEM – software/application development,
maintenance, integration and testing
• SEER-H – system, hardware and electronics
development, production and support
• SEER-IT – IT infrastructure, services and operations
and systems engineering
• SEER-MFG – hardware manufacturing and assembly
• SEER-SYS – Systems Engineering
• SEER-Cost IQ Case based reasoning estimation
© 2016 Copyright Galorath Incorporated 29

30. Multiple Estimation Methods Used
• Knowledge based parametrics
• Built-in estimating relationships, ready to use
• Can be tuned and calibrated to improve accuracy
• Expert judgment – utilize SME inputs for a task
• Analogy – estimate based on past examples
• Top down – estimate total cost and allocate
• Bottom up – estimate each detailed task or process
step
© 2016 Copyright Galorath Incorporated 30

31. SEER Enterprise Costing Solution:
Parts thru Total Ownership Cost
SEER-MFG
SEER-SEM
SEER-IT
Detailed SEER-H
Estimate:
• Development
• Production
• Operation &
Support
• System Level
Costing
Detailed Estimate
Parametric Estimate
Software/IT Estimate
Actual Procurement
Cost Data
SEER-H
© 2016 Copyright Galorath Incorporated 31

32. SEER – Risk Driven Estimates
The Engine For Project Evaluation
Least, likely, and most
inputs provide a range of
cost and schedule outcomes
Confidence (probability)
can be set and displayed
for any estimated item
• SEER predicts
outcomes
• SEER uses inputs to
develop probability
distributions
• The result is a
probabilistic estimate
• SEER will predict a
likely range of
outcomes
• Monte Carlo provides
project-level
assessments of risk
© 2016 Copyright Galorath Incorporated 32
BiasMitigation:Risk

33. Knowledge Bases
• SEER-H uses knowledge bases to
derive and preset initial
parameter settings based on
industry experience
• Choose knowledge bases for:
• Application (Aircraft
Stabilizer, Analog Display,
Electro-Mechanical Control,
etc.)
• Platform (Air-Manned, Air-
Unmanned, Groung Mobile,
etc.)
• Acquisition Category (Build
to Print, Buy & Integrate,
Make, Modification, etc.)
• Standard (Commercial,
Military, Space, etc.)
• Operations & Support
(Remove & Replace, Repair,
Scheduled Maintenance, etc.)
• Users may add their own
knowledge bases if desired
BiasMitigation:CompletedActuals

34. © 2016 Copyright Galorath Incorporated 34
SEER for Software Example:
Top 10 Impacts & Sensitivity Analysis

35. Estimates Are Validated
Against Data (Yours and/or Other)
SEER-SEM
Estimate
Your Data
Regression
Trendline
Galorath
Benchmark
Trendline
Your History
Data

36. 36
Project Monitoring & Control
“When Performance is Measured Performance Improves”
• Adds Performance
Measurement (Earned Value)
methods to SEER-SEM
• Accepts progress &
expenditure inputs
• Provides cost, schedule, and
time variances
• Provides cost, schedule, and
time indices
• Performance-based cost &
schedule Estimate at
Completion
• Displays health and status
indicators
© 2016 Copyright Galorath Incorporated

37. SEER-SEM Total Cost of Ownership
Why Should We Care: Impacts ROI & development
decisions can impact maintenance costs.
37
Staff Vs Maintenance Rigor
0
500
1000
1500
2000
2500
3000
3500
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85
Time
staffhourspermonth
develop
Rigor vhi+
Rigor nom
Rigor vlo
BiasMitigation:ForgottenCosts

38. Searchable Cost Catalogs
• Catalogs used for actual cost metrics for common
tasks and parts
• Configurable to the customers requirements
• Refresh estimates based on the latest metrics
© 2016 Copyright Galorath Incorporated 38
BiasMitigation:Analogies

39. Ply Cost Estimator – CATIA V5BiasMitigation:viablecosttrades

40. Cost IQ - Obtaining Cost Directly
From Limited Requirements
A Case Based Reasoning system that can transform high
level requirements and specifications into a cost modeling
workup within a sophisticated cost estimating model.
This approach can provide even concept planners
with an advanced understanding of the cost of
potential designs.
© 2016 Copyright Galorath Incorporated 40
BiasMitigation:Referenceclasses

41. • Scenarios can be generated proactively or
incidentally
• Deliberately defining a standard estimation WBS pattern
• Turning an exemplar estimate into a WBS pattern
Scenarios
• Pre-configured WBS
patterns may be
stored as a “scenario”
• Scenarios may be
configured with
options for the user to
select which elements
to include
• The WBS structure
will be loaded with the
user’s tailoring
© 2016 Copyright Galorath Incorporated 41
BiasMitigation:CompletedActuals

42. SEER-DB
• Store all estimates in a secure repository
• Secure access to individuals or user groups
• Supports estimate revisions and locking
© 2016 Copyright Galorath Incorporated 43
BiasMitigation:ActualHistory

43. SEER – In Summary
• SEER Parametric cost
estimation software
provides estimates of
Effort, Cost, Schedule
For Information
Technology projects &
Program’s
• Gives a consistent
framework for
estimating and
planning projects
• Helps you understand
why projects cost
what they do
© 2016 Copyright Galorath Incorporated 44

44. Key Points for today . . .
Estimates can be
better, reducing
bias & strategic
mis-estimation…
Parametrics can
help.
© 2016 Copyright Galorath Incorporated 45
Tempering
with an
“outside view”
can mitigate
some bias
Without care
estimates are
usually biased
(even with
experts)

45. At Galorath – We Do Estimation
Galorath solutions: Tools and consulting
Galorath works with industry leaders to help adopt and
improve organizational estimation best practices
• Over 30 years in business
• Hundreds of customers in government & industry;
many Fortune 500
• Headquartered in El Segundo, CA; International
office in Andover, U.K. with staff in the U.S. and
Europe
• Professional services organization provides
consulting and training