Techncial Performance Measures are the core of measuring what "done" looks like.

1. EVM World 2017
CPM-200: Scope and Organization Management
Lesson C: Using Technical Performance Measures
to Inform Earned Value Performance
Glen B. Alleman
Niwot Ridge, LLC
+1 303 241 9633
glen.alleman@niwotridge.com

2. Learning Objectives
• TLO #1: The student will recognize what a Technical
Performance Measure (TPM) is in the context of the
Professional Education Program Case Study as well
as DOD and INCOSE Systems Engineering Guidance.
• TLO #2: The student will be able identify and select
TPMs using the example Work Breakdown Structure
from the Case Study.
• TLO #3: The student will learn how to prepare
descriptions of TPMs needed to assess the physical
“progress to plan” of the program.
• TLO #4: The student will able to judge the validity of a
TPM from a list of TPM’s from the notional program.
2

3. Components we’ll meet along the
way to a credible PMB
Objective Status and Essential Views to support the proactive management processes needed
to keep the program GREEN
Risk Management
SOW
SOO
ConOps
WBS
Techncial and Operational
Requirements
CWBS &
CWBS Dictionary
Integrated Master Plan
(IMP)
Integrated Master Schedule
(IMS)
Measures of
Effectiveness
Measures of
Performance
Measures of
Progress
Key Performance Parameters
Program Specific
Key Performance Parameters
Technical Performance
Measures
Earned Value Management
System
TPMs
Live Here
Performance Measurement Baseline
TLO #1
3

4. Can Earned Value Alone Get Us
To Our Destination?
• How do we increase visibility into program performance?
• How do we reduce cycle time to deliver the product?
• How do we foster accountability?
• How do we reduce risk?
• How do we start our journey to success?
Increasing the Probability of Success means …
Connecting The Dots Between EVM and TPM to Reach Our Destination
TLO #1
4

5. Increasing the Probability of Program
Success Means …
Risk
SOW
Cost
WBS
IMP/IMS
TPM
PMB
Building A Credible Performance Measurement Baseline
This is actually harder than it looks!
TLO #1
5

6. Identifying the TPMs starts with good
Systems Engineering
TLO #1
6

7. Some More Guidance
Systems engineering uses technical performance
measurements to balance cost, schedule, and performance
throughout the life cycle. Technical performance
measurements compare actual versus planned technical
development and design. They also report the degree to
which system requirements are met in terms of performance,
cost, schedule, and progress in implementing risk handling.
Performance metrics are traceable to user–defined
capabilities.
― Defense Acquisition Guide
(https://dag.dau.mil/Pages/Default.aspx)
In The End ― It’s Always About Systems Engineering
TLO #1
7

8. This Has All Been Said Before
… the basic tenets of the process are the need for
seamless management tools, that support an
integrated approach … and “proactive identification
and management of risk” for critical cost, schedule,
and technical performance parameters.
― Secretary of Defense, Perry memo, May 1995
Why Is This Hard To Understand?
§ We seem to be focused on EV reporting, not the use of
EV to manage the program.
§ We think getting the IPMR out the door is the end of
Program Planning and Control’s efforts, not the beginning.
TPM Handbook 1984
TLO #1
8

9. A Final Reminder …
TPMs are one source of Risk Management processes
§ Risk Management Guide to DOD Acquisition, Seventh Edition, December 2014
§ INCOSE Technical Measurement, INCOSE-TP-2003-020-01
TLO #1
9

10. We see TPMs starting with the
Integrated Baseline Review (IBR)
• Confirms that the Performance Measurement
Baseline (PMB) captures the entire techncial
scope of work, which is schedule to meet
program objectives, identifies risks, assigns
resources to meet requirements, and
implements management control processes.
TLO #1
Currently reported Earned Value data contains invaluable planning and
budgeting information … for program management, however, shortcomings of
the system are it emphasis on retrospection and lack of integration with
techncial achievement.
– Cmdr. Nicholas Pisano
10

11. Back To Our Technical Performance
Measures
Technical Performance Measures do
what they say,
Measure the Technical Performance
of the product or service produced by
the program.
TLO #2
11

12. To develop TPMs we first need to
define some other measures
• Measures of Effectiveness (MoE)
• Measures of Performance (MoP)
• Key Performance Parameters (KPP)
• Than we can define the Technical
Performance Measures (TPM)
12

13. Measure of Effectiveness (MoE)
Measures of Effectiveness …
• Are stated in units meaningful to the buyer,
• Focus on capabilities independent of any technical
implementation,
• Are connected to the mission success.
Operational measures of success that are closely related to
the achievements of the mission or operational objectives
evaluated in the operational environment, under a specific set
of conditions.
MoE’s Belong to the End User
“Technical Measurement,” INCOSE–TP–2003–020–01
13

14. Measure of Performance (MoP)
Measures of Performance are …
• Attributes that assure the system has the
capability and capacity to perform,
• Assessment of the system to assure it meets
design requirements to satisfy the MoE.
Measures that characterize physical or functional attributes
relating to the system operation, measured or estimated
under specific conditions.
“Technical Measurement,” INCOSE–TP–2003–020–01
MoP’s belong to the Program – Developed by the Systems
Engineer, Measured By CAMs, and Analyzed by PP&C
14

15. Key Performance Parameters (KPP)
Key Performance Parameters …
• Have a threshold or objective value,
• Characterize the major drivers of performance,
• Are considered Critical to Customer (CTC).
Measures that Represent the capabilities and characteristics
so significant that failure to meet them can be cause for
reevaluation, reassessing, or termination of the program
The acquirer defines the KPPs during the operational concept
development – KPPs say what DONE looks like
TLO #2
“Technical Measurement,” INCOSE–TP–2003–020–01
15

16. Technical Performance Measures
(TPM)
Technical Performance Measures …
• Assess design progress,
• Define compliance to performance requirements,
• Identify technical risk,
• Are limited to critical thresholds,
• Include projected performance.
Attributes that determine how well a system or system
element is satisfying or expected to satisfy a technical
requirement or goal
TLO #2
“Technical Measurement,” INCOSE–TP–2003–020–01
16

17. TPMs start with the WBS
TLO #2
1.0
Safer Counties / Safer
Cities
1.1
SC2 Prime Mission
Equipment
1.1.1. Camera Systems
1.1.1.1 Camera
1.1.1.2 Camera
Interface Hardware
1.1.1.3 Camera
Integration
1.1.2 Wireless Tower
System
1.1.2.1 Wireless Tower
1.1.2.2 Tower Interface
Tower
1.1.2.3 Tower
Integration, Test,
Assembly, and
Checkout
1.1.3 Monitoring
Stations
1.1.3.1 PC Stations
1.1.3.2 PC Interface
Hardware
1.1.3.3 Monitoring
Stations Integration,
Test, Assembly and
Checkout
1.2
System Engineering
/Program Management
1.3
System Test and
Evaluation
1.4
Data
1.5
Training
Let’s focus on the wireless towers
and the services they provide to
public safety, as well as convenience
for the citizens.
We’ll need MOPs, MOEs, KPPs,
before we define the TPMs
Then we’ll connect these TPM’s to
assessing the performance of the
work in the IMS.
17

18. § This is where
TPMs are
connected with
the MoE’s and
MoP’s
§ For each
deliverable from
the program, all
the “measures”
must be defined
in units
meaningful to the
decision makers.
§ Here’s some
“real” examples.
Connecting the MoE, MoP, KPP, and TPMs
1. Provide bandwidth for
5,000 simultaneous
voice and data users
2. Provide location
service for 911
3. Provide 99.5%
availability to users
Measures of
Effectiveness (MoE)
1. Net Ready
2. E911 Location Quality
3. Phone
Interoperability
4. Manpower
5. Availability
Key Performance
Parameters (KPP)
1. Net Ready
§ IPv4/6 compliance
§ 1Gb Ethernet
2. Location quality
§ Accuracy threshold
@60M
§ Integrity threshold
@40M
3. Phone interoperability
§ Processing capability
meets growth matrix
4. Manpower
§ MTBC >1000 hrs
§ MCM < 2 hrs
5. Availability
§ Clear threshold >99%
§ Noise threshold >90%
Measures of Performance
(MoP)
1. Net Ready
§ Standard message packets
2. Location Quality
§ Multipath allocation
budget
§ Multipath bias protection
3. Phone Interoperability
§ MOSA compliant
§ Civil compliant
4. Manpower
§ Operating elapsed time
meters
§ Standby elapsed time
indicators
5. Availability
§ Phase center variations
Technical Performance
Measures (TPM)
Mission Capabilities and Operational Need
Technical Insight – Risk adjusted performance to plan
TLO #2
18

19. Identifying Technical Performance
Measures
Technical Performance Measures must be techncial
and must describe the performance in units of measure
meaningful to the decision makers.
TPM’s are not counts of things delivered. TPMs are not
assessments of completed work through effort or cost.
TPMs are measures of Technical Performance
TLO #3
19

20. How can we possibly know we’re on schedule or on
budget without knowing if we’re technically compliant?
TLO #3
20

21. Measures of Technical Parameters†
Attribute Description
Achieved to Date
§ Measured technical progress or estimate of progress
in units of performance
Current Estimate
§ Value of a technical parameter that is predicted to be
achieved now or in the future
Milestone
§ Point in time when an evaluation of the technical
measure is accomplished
Planned Value § Predicted value of the technical parameter
Planned Performance
Profile
§ Profile representing the project time phased
demonstration of a technical parameter
Tolerance Band § Management alert limits
Threshold § Limiting acceptable value of a technical parameter
Variances
§ Demonstrated technical variance
§ Predicted technical variance
TLO #3
† INCOSE Systems Engineering Handbook 21

22. A Familiar Graphic for TPMs
Variance
Planned Value
Planned Profile
Current Estimate
Milestones
Threshold
Upper Control Limit
Lower Control Limit
Mean To Between Failure
Time = Program Maturity
Achieved to Date
TPM
TLO #3
22

23. What Do We Need To Know About This
Program Through The TPMs?
• What WBS elements represent the TPMs?
• What Work Packages produce these WBS
elements?
• Where do these Work Packages live in the IMS?
• What are the Earned Value baseline values for
these Work Packages?
• How are we going to measure all these variables?
• What does the curve look like for these
measurements?
TLO #3
23

24. Let’s Connect Some More Dots
TLO #3
Risk
SOW
Cost
WBS
IMP/IMS
TPM
PMB
❺ Deliverables
defined in the SOW,
traced to the WBS,
with narratives and
Measures of
Performance (MoP)
❹ Budget at the Work
Package level, rolled to
the Control Accounts
showing cost spreads for
all work in the IMS
❻ Measures of
Performance (MOP) for
each critical deliverable in
the WBS and identified in
each Work Package in the
IMS, used to assess
maturity in the IMP
❶ The Products and
Processes in a “well
structured” decomposition,
traceable to the deliverables
❷ IMS contains all
the Work Packages,
BCWS, Risk mitigation
plans, with traces to
the IMP measuring
increasing maturity
through Measures of
Effectiveness (MOE)
and KPPs
❸ Technical and Programmatic
Risks Connected through the
WBS, Risk Register, IMP and IMS
The PMB is the Document
of Record for the Program
Performance is Measured
through the PMB
24

25. Raison d'etre for Technical
Performance Measures
The real purpose of
Technical
Performance
Measures is to
reduce
Programmatic and
Technical RISK
Risk
SOW
Cost
WBS
IMP/IMS
TPM
PMB
TLO #4
25

26. Key Elements of A Good TPM
• Traceability – Requirements to WBS to
TPMs to EV control accounts.
• Impact – How much WBS work, & therefore
EV money, is covered by the TPM(s)? What
is effect?
• TPM Banding/Sensitivity – What banding
(R/Y/G) and sensitivity (EV impact) should be
used for each TPM?
• Technical Readiness Level – What’s the
state of the technology supporting the
requirement(s) for which TPM is a metric?
Implementing Technical Performance Measurement , Mike Ferraro , General Engineer, PEO/SYSCOM Conference
TLO #4
26

27. Determination of Physical % Complete must be
informed by both task completion and technical status
4Progress of a set of tasks
– Drawings Completed
– Lines of Code Written
– Work Products Produced
– Reviews Completed
• Progress of the technical status
– Critical TPM Achievement
– System Capabilities Met
– Quality of Work Products
– System Under Review Acceptable
Measuring progress
of what people do
Measuring progress of the
Result of what people do
Progress is measured by effectiveness of outcomes to the end user
Physical % Complete P%C
TLO #4
BCWPi
i=1
N
∑ = BCWSi × P%Ci
i=1
N
∑
27

28. Connecting the EV Variables
Integrating Cost, Schedulele, and Technical Performance
Assures Program Management has the needed performance information to deliver
on‒time, on‒budget, and on‒specification
Technical Performance Measures
Cost Schedule
Conventional Earned Value
+
=
§ Master Schedule is used to
derive Basis of Estimate
(BOE) not the other way
around.
§ Probabilistic cost
estimating uses past
performance and cost risk
modeling.
§ Labor, Materiel, and other
direct costs accounted for
in Work Packages.
§ Risk adjustments for all
elements of cost.
Cost Baseline
§ Earned Value is diluted by
missing technical
performance.
§ Earned Value is diluted by
postponed features.
§ Earned Value is diluted by
non compliant quality.
§ All these dilutions require
adjustments to the
Estimate at Complete
(EAC) and the To Complete
Performance Index (TCPI).
Technical Performance
§ Requirements are
decomposed into physical
deliverables.
§ Deliverables are produced
through Work Packages.
§ Work Packages are
assigned to accountable
manager.
§ Work Packages are
sequenced to form the
highest value stream with
the lowest technical and
programmatic risk.
Schedule Baseline
TLO #4
28

29. TPM Checklist
MoE MoP TPM
Traceable to needs, goals,
objectives, and risks.
Traceable to applicable
MOEs, KPPs, system level
performance requirements,
and risks.
Traceable to applicable MoPs,
system element performance,
requirements, objectives, risks,
and WBS elements.
Defined with associated
KPPs.
Focused on technical risks
and supports trades between
alternative solutions.
Further decomposed, budgeted,
and allocated to lower level
system elements in the WBS and
IMS.
Each MoE independent
from others.
Provided insight into system
performance.
Assigned an owner, the CAM and
Work Package Manager.
Each MoE independent of
technical any solution.
Decomposed, budgeted and
allocated to system elements.
Sources of measure identified and
processes for generating the
measures defined.
Address the required KPPs.
Assigned an “owner,” the
CAM and Technical Manager.
Integrated into the program’s IMS
as part of the exit criteria for the
Work Package.
TLO #4
29

30. Summary of TPM Benefits
• Measures of results based on technical
achievement
• Integrated technical performance with budget and
schedule – providing an integrated systems
approach to performance measurement
• Establishes the linkage between engineering and
cost/schedule considerations in program
management decision making
• Measures the programmatic impact of technical
variance in terms of risk
• Reduces subjectivity through the establishment of
technical performance baseline
TLO #4
30

31. Past Issues with
Earned Value Management and Scheduling
• Ability to take credit for all work accomplished
• How to defend what work was earned
• Identifying EV milestones in the schedule focused
attention on those tasks so that maximum earned
value could be taken
• IMS was too detailed to manage the program
• CAM’s used other means to manage and assess
progress related to their work; i.e. detailed schedules
• Detailed schedules did not reflect what was reported in
IMS
31
Solution(– Quantifiable(Backup(Data((QBD)

32. Quantifiable Backup Data (QBD)
• A QBD is a supplement to the IMS to help
manage discrete work at a lower level than is
maintained in the IMS
• The QBD measures the work objectively
– 748-C calls out on Page 1
– Objectively assess accomplishments at the work
performance level
• QBD’s are part of the PMB
• QBD’s do not replace the EVMS Earned
Value Techniques (EVT)
32

33. Quantifiable Backup Data (QBD)
• A detailed listing of tasks necessary to complete all scope in a work
package during the defined period of performance.
• It is an approach used to objectively measure performance
– Each task on the list is weighted – total weighting equals 100% of the
work package BAC (weighting is important, should not be equal
weighting on every task).
– Upon completion of QBD development the QBD is placed under
configuration control.
– The CAM assesses physical percent complete of each QBD task.
– The percent complete is calculated from the cumulative assessments.
• The purpose of the QBD is to:
– Ensure and demonstrate that all contract work is accounted for;
– Ensure the schedule and budget are realistic and achievable;
– Mitigate schedule and budget risks; and,
– Provide a basis for objectively assessing progress for discretely
measured work packages.
33

34. 34

35. What We’ll Do In Our Hands On
Work Shop
Based on a notional case study for a
county surveillance system, we will
strategize about the system MOEs
and MOPs and decide upon
appropriate TPMs to inform our
program performance and BCWP
35

36. Case Study Set Up
• The county is concerned about controlling
disturbances, mitigating crime, and having
forensic video and sound evidence for legal
purposes
• The county desires to procure a surveillance
system with cameras and sound devices at
strategic locations, send the data feeds via
wireless technology, monitor, store and
retrieve selected incidents
36

37. Case Study Set Up (Continued)
• The county wants near “real time” response
and desires video quality such that they can
distinguish facial features and read a license
plates within 20 feet from the camera
• It also wants the system to alert operators
when there is evidence of a disturbance
• It wants equipment that will allow them to
store five years worth of data feeds collected
24/7
37

38. Case Study Set Up (Concluded)
• The county has submitted a Request for Proposal
(RFP) to procure, install, test and train personnel. The
county has set aside of budget of $40M and wants to
system operational within 18 months. Because of the
nature of the procurement, the county has required
monthly IPMRs (See Handout 1 for details)
• You are the proposal manager for the contract. Your
immediate job is to
– Determine the system’s “Measure of Effectiveness”
– Select appropriate Technical Performance Measures to
use to inform or drive reported cost and schedule progress
– Propose how you will ensure the county that cost and
schedule progress is consistent with measured technical
progress
38

39. Handout # 1
Safer County/Safer City
Surveillance Background and
Technical Requirements
39

40. Exercise # 1
• What are possible Measures of
Effectiveness (MOE) of the System?
• What are possible Technical Performance
Measures (TPM) that could be used to
drive or inform the cost and schedule
progress?
40

41. All these measures
used to inform BCWP
Reminder: MOEs, MOPs, and TPMs
are Connected to the IMP/IMS
SOW
ConOps
WBS
CWBS
Technical
Performance
Measures
Messages of
Performance
Measures of
EffectivenessKPPs
Integrated
Master Plan
Integrated
Master
Schedule
Performance
Measurement
Baseline
Techncial and Operational
Requirements
41

42. Handout # 2
Safer County/Safer City
Surveillance System
School Solution
42

43. Exercise # 2
• Discuss school solution
43

44. Handout # 3
Safer County/Safer City Surveillance
System
Work Breakdown Structure and
Integrated Master Schedule
44

45. Exercise # 3
• Which WBS elements & IMS Work
Packages need to be informed by TPM
progress?
45

46. Presentation of Workshop Results
With our MOE’s, MOP’s, KPP’s, and
TPM’s let’s assess out needed
capabilities against the technical
solution
46

47. OFFICIAL VERSION
COLLEGE OF PERFORMANCE MANAGEMENT
PROGRAM OF INSTRUCTION
200C VERSION 3
18 Sep 2014
Approvals:
William Mathis, V.P of Education and Certification 18 Sep 2014
[Add appropriate approval individual name and title] Date
[Add appropriate approval individual name and title] Date
[Add appropriate approval individual name and title] Date
47

48. VERSION
NUMBER
DATE NUMBER OF
FIGURE, TABLE
OR PARAGRAPH
A*
M D
TITLE OR BRIEF DESCRIPTION CHANGE
REQUEST
NUMBER
2 29 Sep 2013 ALL A,M Added page numbers, moved pictures up
2 11 Oct 2013 A.M Format change
3 22 Feb 2014 A,M Format changes for EVM World 2014
4 18 Sep 2014 A,M Format change for IPMW 2014.
5 22 Feb 2015 A,M Changed example from UAV to Wireless Tower
6 26 Feb 2015 A,M Added hands on exercise
7 11 Mar 2015 A,M Added workshop materials
Record of Changes
48