1. Estimation & Planning Processes Decide
Project Success
NASA Project Management Challenge 2012
Dan Galorath, Founder & CEO
Copyright Galorath Incorporated 2011

2. Introduction
• Estimating is critical for all kinds of systems
• Yet many treat is as a second rate process
• Having a repeatable estimation process is critical to
both estimating AND to successful projects
• Estimation and measurement go hand in hand
© 2009 Copyright Galorath Incorporated 2

3. Delusions of Success: How Optimism
Undermines Executives' Decisions Richard Hartley, HBR)
(Source:
• Problem:
Humans seem hardwired to be
optimists
• Optimism from cognitive biases &
organizational pressures
• Exaggerate talents & degree of control
• Attribute negative consequences to external factors
• Anchoring (relying too heavily on one piece of
information) magnifies optimism
Solution: Temper with “outside view”
• Most pronounced for new initiatives
Supplement traditional forecasting w/ statistical
Parametrics
Don’t remove optimism, but balance optimism & realism

4. Example of Tempering: (Source How To
Measure Anything)
• German Mark V Tanks
• Allies estimated production by analyzing serial numbers
• Used the captured tanks as a random sample and predicted
probability of various production levels

5. An Estimate Defined
• An estimate is the most knowledgeable statement you
can make at a particular point in time regarding:
• Effort / Cost
• Schedule
• Staffing
• Risk
• Reliability
• Estimates more precise with progress
• A WELL FORMED ESTIMATE IS A
DISTRIBUTION
5

6. Estimation Methods 1 of 2
Model
Description Advantages Limitations
Category
No Basis or substantiation
Guessing Quick Can obtain any No Process
Off the cuff estimates
(Common) answer desired Usually Wrong
Generally optimistic
Truly similar projects must
Analogy Compare project with past Estimates are based on
exist.
(Common) similar projects. actual experience.
Less optimistic
Experts tend to be biased;
Expert Little or no historical knowledge level is sometimes
Consult with one or more
Judgment data is needed; good for questionable; may not be
experts.
(Common) new or unique projects. consistent.
Generally optimistic
A hierarchical Need valid requirements.
Provides an estimate
decomposition of the Difficult to track architecture;
Top Down linked to requirements
system into progressively engineering bias may lead to
Estimation and allows common
smaller components is used underestimation.
(Common) libraries to size lower
to estimate the size of a Generally optimistic (miss
level components.
software component. non-WBS items)
6

7. Estimation Methods 2 of 2
Model Category Description Advantages Limitations
Uses expert judgment
to determine how Easy to get under
Design To Cost Little or no engineering basis.
much functionality can stakeholder
(sometimes) Optimistic
be provided for given number
budget.
Simple relationships may not tell the
Equation with one or
whole story
Simple CER’s more unknowns that Some basis in
Historical data may not tell the whole
(common) provides cost / data
story
schedule estimate
Less optimistic
Models are
usually fast and Models can be inaccurate if not
easy to use, and properly calibrated and validated;
Perform overall
useful early in a historical data may not be relevant
Comprehensive estimate using design
program; they are to new programs; optimism in
Parametric Models parameters and
also objective parameters may lead to
(common) mathematical
and repeatable. underestimation.
algorithms.
Easy tradeoffs More or less optimistic depending
can provide on parameters
better plans
Why should we care: Each method has challenges and
we should be familiar with each
7

8. Common Challenges In Estimating
• High effort and expenses in producing estimates
Proposals - Acquisition Strategy inputs
• Informal request - POM budgets
• Significant investment in time to generate numbers
• Weak accuracy in numbers leading to cost overruns
• Customer frequent request for revisions cause even
most expenses
• Numbers susceptible to many forces without robust
cost generation systems
• Lack of understanding of cost risk
• Difficulty in calibrating “Engineering Judgment
© 2009 Copyright Galorath Incorporated 8

9. Poor Estimates Effect on Projects
• Inaccurate estimates can reduce project success:
• Poor implementations
• Critical processes don’t scale
• Emergency staffing
• Cost overruns caused by underestimating project needs
• Scope creep
• Forever changing project goals
• Frustration
• Customer dissatisfaction
• Cost overruns and missed schedules
• Project Failures
• Important project business decisions made early with
minimum knowledge & maximum uncertainty
Why should we care: Poor estimates and plans are
a root cause of program failure
9

10. Initial Cost Estimation Problems (Software
Program Managers Network)
• Many programs that have been evaluated tend to initially estimate using a
very optimistic method.
• Low bid to win contract
• Naïve level of effort estimation
• Human optimism (HBR & Rich Hartley USAF Undersec Acquisition)
• Often wrong since they are not based on a thorough analysis of requirements. The formula
Cost = Size x Complexity/Productivity
• Has three unknowns upfront: size, complexity, and productivity.
• Methods & estimating tools determine size given system requirements
• Organizational databases of productivity on comparable size and
complexity projects can be used
• Or other bottom-up estimation techniques
“In any event, all initial cost estimates should
be considered as potential high risk and
should be reviewed at each program review”
SPMN

11. Software Is A Critical Component of Military &
Aerospace Systems Failures Ariane 5
• Ariane 5 video
• Note: $500 Million payload
• Failure due to reused software from (Ariane 4)
• with incompatible assumptions for exception
condition that was not required
“the culture within the Ariane program…” only addressed
“random hardware failures… which can quite rationally be
handled by a backup system”
“the view had been taken that software should be considered
correct until it is shown to be at fault”!
Why should we care: Software cost & schedule
should be sufficient for successful missions

12. Software Is A Key Risk Item In
Weapons Systems
• Navy Mobile User Objective Satellite Communication
System delays to the Joint Tactical Radio
System, a set of software-defined radios causes
advanced MUOS capabilities to drastically underused…
GAO
• GAO identified 42 programs at risk for cost & schedule
1. military requirements changes
2. software development challenges
3. workforce issues
• National Institute of Standards and Technology (NIST)
• Software defects cost nearly $60 Billion Annually
• 80% of development costs involve identifying and
correcting defects
Software, not Hardware or technology readiness levels
were called out

13. Death March Projects Are Likely To Fail
Why should we care: If you have a project on
a death march failure is highly probably

14. Balancing Resources & Schedule Is
A Science
For a given Size, Complexity and Technology
Minimum Time Work Expands
To Fill Time
To Complete (Effort Increases
(Effort Increases
due to lack of pressure )
to Reduce Schedule)
Effort Increase
Minimum Time due to Longer
Effort Months
Schedule
Optimal Effort
(Lower Effort
for Longer
Schedule)
Calendar Time

15. Why Total Cost Grew for Two Space Programs
David Graham, NASA
Development Growth Causes
Requirements
Generation & Translation
8% Budget/Funding
11% 25%
Cost Estimation
Underestimation of Risk
11%
11% Schedule Slips (Govt &
Contractor)
9%
Price Increases
25%
Other
Quantitative Framework
5 “TheSuccess Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale
Systems in an Increasingly Complex Environment” - (Booz|Allen|Hamilton, 2003)

16. Use Earned Value To Quantify Progress
Versus Effort (Oversimplified)
• Main EVM concern is what has been accomplished in a
given time and budget, versus what was planned for the
same time and budget
• Project generally deemed healthy if what has been
accomplished is what was planned, or more
• Project deemed unhealthy if accomplishment lags
expectations
• Definition: Earned value = budgeted value for the work
accomplished (what you got for what it cost you)
Healthy Unhealthy
$ Budget $ Budget
EV
EV
Time = Now Time = Now
16

17. Deploying Before Complete Leads To Program
Disasters: You Better Understand Schedule
17

18. Understand Project Risks Include Them In Planning
Decisions (Example SEER-SEM Outputs)
Probability
Schedule Probability Probability
Effort Probability
Example Application 1 Example Application 1
99% 99%
90% 90%
80% 80%
70% 70%
60% 60%
50% 50%
40% 40%
30% 30%
20% 20%
10% 10%
1% 1%
0 4 8 12 16 20 0 1800 3600 5400 7200 9000
Time (calendar months) Effort (person-hours)
Defects Probability
Probability Example Application 1
99%
90%
80%
70%
60%
50%
40%
30%
20%
10%
1%
0 12 24 36 48 60
Defects (count)
18

19. Considering Maintenance During Planning Can
Yield More Successful Long Term Results
Staff Vs Maintenance Rigor
3500
staff hours per month
3000
2500 develop
2000 Rigor vhi+
1500 Rigor nom
1000
Rigor vlo
500
0
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85
Time
19

20. Data Doesn’t Have To Be Perfect To Be
Useful: But Is Has To Be Viable
• 80 Calories per serving
• 2.5 Servings per can
• 4 Ounces, Condensed, 8 Ounces With Water

21. You have an estimate …
Now what?
21

22. The Error of Causal Analysis
Creating a False Association
• Correlation does not imply causation
• Just because two data points may sit side by side
doesn’t mean they are the same or will have the same
outcome
• Casual analysis is a recognized error in medicine
Tumor Can Cause
Headache Perhaps ???
Headache doesn’t mean a
tumor
22

23. Use Historical Measurement to
evaluate your estimate!
It’s easy to dig deeper and deeper to justify an estimate!
23

24. Data Beware Apples and Oranges:
Phase : All activities may not be included.
System Concept Integration
missing missing
Phases
System Concepts System Req & Design
System Req Analysis Preliminary Design
Detailed Design Code / Unit Testing
Software Test System Integration / OT&E
24

25. Estimation Process
25

26. Basic Cost Estimating Process (Source CEBOK)
WBS • Work Breakdown Structure
(WBS) Development
Baseline
• Program/System Baseline
Data
Collection
Development
Data
Analysis
Methodology
Validation
Reports
26

27. US GAO process for Credible
Estimates
27

28. 10 Step System Estimation Process
2011
1. Establish
Estimate Scope 10. Track Project
Throughout
Development
2. Establish Technical 9. Document Estimates
Baseline, Ground and Lessons
Rules, Assumptions Learned
8. Generate a
Project Plan
4. Refine Technical
Baseline Into
Estimable Components 6. Validate Business
Case Costs &
Benefits (go / no
go)
4. Collect data /
estimation inputs 6. Quantify Risks
and Risk Analysis
5. Estimate Baseline Cost,
Schedule, Affordability Value
28

29. Estimation Organizational Maturity V1.7
Level Informal or no
Manual effort
estimating
0
estimating without a
process
Level Direct Task
Spreadsheets
Ad Hoc
1
Estimation Process
Formal Simple model
Level Sizing
(e.g.
Direct
Task
(Size *
Productivity)
or informal
Some
measureme Informal
2 Process
nt &
function Estimation SEER Use
analysis
points)
Level Formal
Robust
Parametric Estimate vs.
Formalized
Multiple
Rigorous
measurement
Parametric
planning &
Risk Repeatable
3
Sizing estimation actual capture Estimate Management process
& analysis Control
(SEER) Process
Level Formal sizing
Repeatable
Robust
parametric
Rigorous
measurement
Parametric
estimation Risk
Process
improvement
4
process estimating with tracking Management via lessons
& analysis
(SEER) & control learned
Level Formal sizing
Repeatable
Robust
parametric
Rigorous
measurement
Parametric
estimation Risk
Continuous
process
5
process estimating with tracking Management
& analysis improvement
(SEER) & control
Why should we care? Maturity is related to estimate
viability… With better estimation process, projects
more likely to be successful in execution

30. Estimation Should Be Key Part of Process
(Source Q Redman, APMP Just Say No)
Phase -1 0 1 2 3 4
DDE
ROM
ROM
Formal Bid
Gate 4
Scope & Accuracy
ROM
15-20 people
ROM 4 weeks
(Bid Stds+
History)
EARLY ESTIMATING Modified
3-5 people, 3 - 5 days Budgetary
Top Down, parametric model Estimate
based price estimating Draft RFP/Gate 3
Vs. 6-8 people, 3
Current state: 90 people, 6wks weeks
(Bid Stds + History
)
Market Opportunity Acquisition
Us
Creation/ Customer Procurement
e
Assessment/ Planning/ POM Draft RFP RFP
Decision Plans Initiation
“What If’s” and Plus Ups
30

31. GAO Publication: Characteristics of credible cost
estimates and a reliable process for creating them
• This chapter discusses a 1972 GAO report on cost estimating
• We reported that cost estimates were understated and causing unexpected cost growth
• Many of the factors causing this problem are still relevant today
• We also discuss a 12 step process for producing high quality cost estimates
31

32. GAO Publication: Why cost estimates are required
for government programs and challenges developing
credible results
• Introduces why cost estimates are required for government programs
• Developing annual budgets, supporting management decisions about
which program to fund, and evaluating resource requirements at key
decision points
• Discusses various challenges associated with developing credible
results
32

33. Ask These Questions When
Identifying Estimate Scope
• Challenged projects
• Would you still go forward if you knew
• Schedule would be significantly longer?
• Cost would be dramatically higher?
• Probably: but perhaps more insight could identify
mitigation
• Plan functionality differently
• Certainly you could notify stakeholders of real costs
• Ensure staffing is appropriate for the constraints
• Failed Projects
• Would you start a project you knew was unaffordable?
Or if schedule was completely unrealistic?
• If knowing up-front could you do something about it?
• Often better to kill project before it begins than waste
resources & let the organization down 33

34. Lesson Learned: Estimate Must
Quantify Risk & Uncertainty
Firm Fixed
Price?
Feel lucky?
What is
likely to
happen
Understand the risk before you commit!
34
34

35. Estimation Best Practices
© 2011 Copyright Galorath Incorporated

36. Cost Estimate Qualities
(Adapted from CEBOK)
• The characteristics of high quality cost
estimates are:
• Accurate (Viable Within Range)
• Comprehensive
• Replicable and Auditable
• Traceable
• Credible
• Timely
36
Unit I - Module 1

37. System Description (Parametrics Can
Estimate More, Earlier) Adapted from CEBOK
“If you can’t tell me what it is, I
can’t tell you what it costs.”
-Mike Jeffers
“If you can tell me the range of
what it might be I can tell you the
range of cost, schedule &
probability”
-Dan Galorath
37

38. Types of Cost Estimates (Adapted From
CEBOK)
• Life Cycle Cost Estimate (LCCE)
• Independent Cost Estimate (ICE) 1
• Budget Estimate
• Rough Order of Magnitude (ROM)
• Estimate At Completion (EAC)
• Independent Cost Assessment (ICA)
• Analysis of Alternatives (AoA)
• Economic Analysis (EA)
38
Unit I - Module 1

39. Aircraft Example: Estimating
Techniques Adapted from CEBOK)
• Where applicable, use primary methods
• Analogy
• Model X100 series jet engines have only been used on one other
plane, but weight is 100% higher on this model; estimate 2x other
model
• Simple Parametric Param etric Estim ate
• As shown, need to estimate 2 lb 12
brake rotors, should be roughly $4M 10
from regression curve 8
Cost
6
• Commercial Parametric 4
2
• Key characteristics range are 0
0 2 4 6 8 10
Param eter (ie. w eight)
• 30-50k lines of code and 600-800
• kilos engine & 7 PC boards
• Engineering Build-Up
• Know Air Conditioning (AC) system costs on plane because received
quotes from HVAC vendor for all duct work and AC blower off the
39
shelf

40. Manual Estimates: Human Reasons For
Error (Metrics Can Help)
• Manual Task estimates yield SIGNIFICANT error
• Desire for “credibility” motivates overestimate
behavior (80% probability?)
• So must spend all the time to be “reliable”
• Better approach: force 50% probability & have “buffer”
for overruns
• Technical pride sometimes causes underestimates
40

41. Lesson Learned: The Risk In Risk
Analysis
"It's tough to make predictions, especially about the future." -- Yogi
Berra.
41

42. Cost Readiness Levels at Low TRLs and/or
Less Than Firm Requirements
• If the project has critical items at
less than TRL 4…
• Like asking Edison in 1876 “How
much longer for the light bulb”
• “Hard to say”, he would have said
as he had you thrown out
• Note that this is not the same as
asking, in 1879, once he had found
a workable carbon filament, “How
TRL
9 TRL9: Actual system “flight proven” thorough successful mission operations
much will a production version of
TRL
8 TRL8: Actual system completed and “flight qualified” through test and demonstration
the light bulb cost to develop and TRL
7 TRL7: System prototype demonstration in a space environment
produce Tom?” TRL
6 TRL6: System/subsystem model or prototype demonstration in a relevant environment
TRL
• This would have been a TRL 4 5 TRL5: Component and/or breadboard validation in relevant environment
TRL
question 4 TRL4: Component and/or breadboard validation in laboratory environment
TRL
• Tom’s cost estimators could have 3
TRL
TRL3: Analytical and experimental critical function and/or characteristic proof-of-concept
begun to model this 2
TRL
TRL2: Technology concept and/or application formulated
TRL1: Basic principles observed and reported
•
1
So if 1<TRL<3 CRL < 4
• Likewise, if requirements are not
firm,  CRL < 4 42

43. Table 1: CRL Rating Prior to Availability of
Probabilistic Risk Analysis
Basic CRL9 5 5.5 6 6.5 7 7.5 8 8.5 9
“Complexity*” CRL Description
of the CRL8 4.5 5 5.5 6 6.5 7 7.5 8 8.5
to go work
at the time CRL7 End of project actual cost
of the
4 4.5 5 5.5 6 6.5 7 7.5 8 9
estimate
CRL6 3.5 4 4.5 5 5.5 6 6.5 7 7.5 Cost fit for very firm
8 engineering decisions and
very firm budget
commitments (+/-5%)
CRL5 3 3.5 4 4.5 5 5.5 6 6.5 7
Cost fit for firm
CRL4 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 engineering decisions and
firm budget commitments
(+/-15%)
CRL3 2 2.5 3 3.5 4 4.5 5 5.5 6
Cost fit for PDR
CRL2 6 engineering decisions and
1.5 2 2.5 3 3.5 4 4.5 5 5.5 PDR budget use
(+/-25%)
CRL1 1 1.5 2 2.5 3 3.5 4 4.5 5
Cost fit for preliminary
5 engineering decisions and
Extrem Very Very Very Very Very Very Very Very preliminary budget use (+/-
el 35%)
Minimal
Cost fit for very
4 preliminary engineering
decisions and very
*Complexity considerations include human rating, launch Adequacy of “Estimating Methods”, preliminary budget use (+/-
system requirements, planetary destination, operational vs
experimental requirements, materials complexity, use of
experience of the estimators, quality of 45%)
deployables, parts count, challenging thermal CARD, availability of analogous data and
requirements, high data rates, electronic parts cost tools, time allowed for estimate,
class, stabilization requirements, power generation
independence of estimate, number of 43
type, propellant choice, propulsion requirements and many
other factors. Programmatic complexity includes team cross checks performed
size, team experience, schedule and many other factors.

44. Table 2: CRL Rating After Availability of Probabilistic Risk Analysis
Use S Curve inter-quartile cost range to translate to a CRL rating
–Calculate ratio of 75th percentile cost to 25th percentile cost; then lookup ratio on chart to read CRL
Lookup
25th Percentile Median Cost 75th Percentile Cost Ratio of 75th Read
Cost Percentile Cost to 25th CRL
Percentile Cost Description
End of project actual cost
100 100 100 1.00 9
Cost fit for very firm
95 100 105 1.11 8 engineering decisions and very
firm budget commitments
(+/-5%)
Cost fit for firm engineering
85 100 115 1.35 7 decisions and firm budget
commitments (+/-15%)
Cost fit for PDR engineering
75 100 125 1.67 6 decisions and PDR budget use
(+/-25%)
Cost fit for preliminary
65 100 135 2.08 5 engineering decisions and
preliminary budget use (+/-35%)
Cost fit for very preliminary
55 100 145 2.64 4 engineering decisions and very
44
preliminary budget use (+/-45%)

45. Estimation Best Practices
• Decide Why You Want An Estimate
• Map Estimation Goals To Estimate Process Maturity &
Develop Plan To Achieve The Maturity
• Have A Documented, Repeatable Estimation Process
• Make The Estimating Process As Simple As Possible;
But No Simpler
• Be Proactive: The Process Is Important, The Tools Go
Along With The Process
• Get Buy-in From Program Managers
• Hold People Accountable: Center Of Excellence Can
Prepare Estimate But Program Managers Must Own
Them
• Tie The Estimate To The Plan
45

46. Estimation Best Practices 2
• Evaluate Total Ownership Cost; Not Just Development
• Estimate A Range And Pick A Point For The Plan
• Re-estimate The Program When It Changes
• Avoid Death Marches: Programs With Unachievable
Schedules Are Likely To Fail And Drain Morale
• Keep A History: Start An Enterprise Database NOW…
• Business Case: Evaluate ROI In Addition To Costs
• Convert Expert Spreadsheets Into A Common
Language
46

47. Estimation Best Practices 3
• Track Progress Vs. Estimate Throughout The Life Cycle
• Estimate Schedule As Well As Effort (Cost) For
Complete Picture
• Tie The Business Case Into The Estimating Process
• Attack Non-productive Rework As Part Of The Process
47

48. Estimation Best Practices 4
• Have clear definitions:
• What does “complete” mean
• What activities are included and excluded (E.g.
development only or total ownership; help desk
included or excluded, etc.)
• Which labor categories are included and excluded in the
estimate (e.g. are managers included? Help desk? Etc.)
• Don’t ignore IT infrastructure and IT services costs
• Tracking defect sources can go along with the
process
48

49. Project Management Challenges
Relate To Estimation Planning
• “No good deed will go unpunished.” Unreasonable
expectations on the next project are supported by
heroic behavior on the previous project
• The most important business decisions about a
software project are made at the time of minimum
knowledge and maximum uncertainty.
• Adding and/or changing means more time and/or
more effort
• When a project is in trouble ask for more time rather
than more people. Deferring functionality common
approach to asking for more time
• Increasing concurrency is usually not a solution (e.g.
designing several releases concurrently)
A1-49

50. 7 Characteristics of Dysfunctional
Software Projects (Source: Mike Evans, et al.)
• Based on 350 projects:
• Failure to Apply Essential Project Management
Practices
• Unwarranted Optimism and Unrealistic
Management Expectations
• Failure to Implement Effective Software
Processes
• Premature Victory Declarations
• Lack of Program Management Leadership
• Untimely Decision-Making
• Lack of Proactive Risk Management
50

51. Additional Information
• www.galorath.com
• Dan on estimating BLOG: www.galorath.com/wp
• Email: galorath@galorath.com
© 2009 Copyright Galorath Incorporated 51