1. National Aeronautics and Space Administration
Risk Management during Integrated Systems Research
Technology Development
Presented by:
Douglas Brown, Environmental Responsible Aviation (ERA) Risk Manager
Gaudy Bezos-O’Connor, ERA Deputy Project Manager
Steven Hirshorn, Integrated Systems Research Program Systems Engineering & Integration Manager
NASA Project Management (PM) Challenge 2012
February 22-23, 2012
www.nasa.gov

2. Abstract
Co-Authors: Justin Hornback (former ERA RM), Gaudy Bezos-O’Connor (ERA DPM),
Steve Hirshorn (ISRP SE&I Mgr), and Douglas Brown (ERA RM)
The Environmental Responsible Aviation (ERA) Project, through system-level analyses, will select promising
N+2 vehicle and propulsion concepts and technologies to develop based on their potential benefit toward
simultaneously reducing fuel burn, noise and emissions.
These concepts and technologies will then be matured and their performance will be evaluated at the system
and sub-system level in relevant environments.
Risk management is a set of activities aimed at achieving success by proactively risk-informing the selection
of decision alternatives and then managing the implementation risks associated with the selected alternative.
In technology development efforts the measure of success to apply risk management assessment criteria is
not easily defined. The ERA Risk Management Process developed a process to define and assess risks to
achieving project success across a portfolio of independent sub-project technology development, that, when
combined are required to maximize likelihood of achieving ERA project goals. ERA project goals are to select
promising N+2 vehicle and propulsion concepts and technologies based on their potential benefit toward
simultaneously reducing fuel burn, noise and emissions. The ERA risk management process added another
layer of risk assessment; identifying a contribution factor based on subject matter expert input for each
technology weighing each technology potential contribution to ERA goals. This was then applied to risks
identified to the ERA project. The additional weighting factor improves identification of the true impact of
project risks to achieve goals compared to traditional risk management processes. This presentation
describes the process used to develop the weighting factor and share ERA's experience during
implementation.
N+1, N+2 and N+3 reflect time periods for technology insertion into the aircraft fleet:
•N+1: 2015-2020, N+2: 2020- 2025, N+3: 2025 +
2

3. Presentation Focus:
The Risk Management Challenge
• What is the appropriate risk management construct for an Aeronautics Technology
Development Project responsible for maturing airframe and propulsion technologies
from TRL 3 to 5/6 through Integrated Systems Research?
• Factors:
– Cultural and Project Execution Paradigm Shift from Fundamental Research to
Technology Development
– Technology Development Projects are finite life, not enduring
– Geographically dispersed government team
– High Degree of Industry/OGA Partnerships/Collaborations with significant
costshare across Project Portfolio (FY10-12)
– Varied or limited experience with project risk management outside of
airworthiness risks for aeronautics flight research projects.
– The large number of independent tasks part of the portfolio of ERA
• No project integrated critical path
– Must balance portfolio content versus schedule margin and budget reserves
• Budget reserves address prototype test article challenges and unique capability
investments.
• Schedule margin address technical risks and facility challenges
3

4. Presentation Outline
• What is Aeronautics Integrated Systems Research
– How does it differ from Aeronautics Fundamental Research?
• Overview of ARMD, ISRP and ERA Project
• Aeronautics National Goals and ERA Project Technical Challenges
• ERA Project Goals, Approach and Deliverables
• ERA’s Risk Informed Decision Making Strategy
4

5. What is
Aeronautics Integrated Systems Research?
N+1, N+2 and N+3 reflect time periods for technology insertion into the aircraft fleet:
•N+1: 2015-2020
•N+2: 2020- 2025
•N+3: 2025 +
5

6. NASA Aeronautics Portfolio
Integrated
Systems
Research Program
Fundamental Aeronautics Program Airspace Systems Program
Conduct cutting-edge research that will Conduct research at an integrated Directly address the fundamental ATM
produce innovative concepts, tools, and system-level on promising concepts and research needs for NextGen by dev-
technologies to enable revolutionary technologies and explore/assess/demonstrate eloping revolutionary concepts,
changes for vehicles that fly in all the benefits in a relevant environment capabilities, and technologies that
speed regimes. will enable significant increases
in the capacity, efficiency and
flexibility of the NAS.
Aviation Safety Program
Conduct cutting-edge research that will produce innovative
concepts, tools, and technologies to improve the intrinsic safety
attributes of current and future aircraft.
Aeronautics Test Program
Preserve and promote the testing capabilities of one of the United States’
largest, most versatile and comprehensive set of flight and ground-based
research facilities.
6

7. Integrated Systems Research Program Overview
Program Goal:
Conduct research at an integrated system level on promising concepts
and technologies and demonstrate the benefits in a relevant
environment
Environmentally Responsible Aviation (ERA) Project
Explore and assess new vehicle concepts and enabling
technologies through system-level experimentation to
simultaneously reduce fuel burn, noise, and emissions
Unmanned Aircraft Systems (UAS) Integration in the National
Airspace System (NAS) Project
Contribute capabilities that reduce technical barriers related to the
safety and operational challenges associated with enabling routine
UAS access to the NAS
7

8. Traceability from National R&D Plan to
ERA Project Technical Challenges
National R&D Plan
Energy and Enhance Mobility National Security
Environment
ERA Project Goals: Simultaneous Achievement of
the NASA Subsonic Transport System-Level Metrics (N+2 Timeframe)
-75% LTO & -70% Cruise
-42dB below Stage 4 -50% Aircraft Fuel/
NOx Emissions
Community Noise Energy Consumption
below CAEP6
ERA Project Technical Challenges (FY10-15)
Advanced Airframe & Engine
Innovative Flow Advanced Advanced UHB
Combustors for Integration for
Control Concepts Composites for Engines for SFC &
LTO Oxides of Community Noise
for Drag Reduction Weight Reduction Noise Reduction
Ni reductions Reduction
8

9. ERA Project Goals, Approach and Deliverables
• Project System-Level Performance Metrics:
• Simultaneous achievement of the community noise, emissions and fuel burn
metrics defined in the NASA Subsonic Transport System Level Metrics in the
N+2 timeframe
• Project Approach:
• Combine rigorous systems analysis with large-scale, integrated systems
research demonstrations of promising airframe and propulsion technology
solutions to TRL 5/6 by 2015
• Increase the viable trade space of vehicle configurations that can
simultaneously meet the goals
• Project Definition:
• 6-year life;
• 2 Phases: Phase 1 (FY10-12); Phase 2 (FY 13-15)
• Project Deliverables: Key Performance Parameters (KPPs)
• Technology Readiness Level (TRL) Maturation Maps
• Product Transition Opportunities: Technology Transition Maps
• Vehicle-level system metrics to measure progress towards the Project Goals
• Technical data to validate/enhance system and physics-based assessment
tools 9

10. ISRP and ERA Risk Management Plan
Implement both Continuous Risk Management
(CRM) and Risk Informed Decision Making (RIDM)
– 8000.4A, Agency Risk Management Procedural Reqs.
– 7120.08, NASA R&T Program & Project Mgmt. Reqs.
• Research and Technology Risk Management
Propulsion
– Flight hardware focused
Airframe, Aeroacoustic, and
– Risk management must balance the need Propulstion Airframe
to conduct challenging technology AeroAcoustic Wind Tunnel
Testing
development that will realize significant
gains.
Lightweight Integrated
Structures Testing Flight Testing and Advanced
Vehicle Concepts
Advanced Combustor Testing
Advanced Propulsor Testing 10

11. ISRP and ERA Project Risk Management Strategy
• Risk management at the program level will tend to be strategic and focused on
ensuring the success of the projects, while risks that focus on the tactical
technical / cost / schedule execution risks will be largely managed at the project-
level with program insight.
– ISRP Projects will manage their technical performance, schedule, and cost risks
according to their Risk Management Plans.
– Significant project risks or risks requiring resources beyond those available to the
Project will be “Tracked” by or “Elevated” to the Program.
• A common frame of reference for Likelihood and Consequence (L&C) exists
between ISRP and its Projects and across Projects:
– Stems from the need to be able to reference both Program and Project risks in an
apples-to-apples comparison when communicating risks to the ISRP Program
Director and to the Mission Directorate.
– Enable traceability of risks from Project task level to ISRP Programmatic Risks
• In order to ensure both Program and Project processes remain in sync and
remain aware of risks at both levels, frequent good communications must be
maintained .
11

12. Assessment of Risk in a Technology
Development Project
• Technology risks follow a different risk pattern than other types of risks
• The level of maturity for a technology affects the risk profiles of that technology,
i.e. lower maturity levels come with higher risks
• The technology landscape is constantly changing with new technologies coming
online promising increased performance. Risk assessment must weigh the
promise of new performance against the confidence of what has been done
before
• These considerations do not require us to change likelihood and consequence
criteria but consider the lens through which we view these scales
12

13. Technological Considerations For Risk Assessment
Considerations
Rating ERA Likelihood Ratings
Technology Maturity Support Base
Value Technical
Some Research Completed/ Never Done No Other Program Developing Similar
Before Technology 5 Very High PTCS > 75%
New Design Based On Existing One Other Program Developing Similar
Technology Technology 4 High 50% < PTCS <= 75%
More Than One Program Developing
Major Redesign Of Existing Technology
Similar Technology 3 Moderate 20% < PTCS <= 50%
Minor Redesign A Few Parallel Programs 2 Low 5% < PTCS <= 20%
Existing Multiple Parallel Programs 1 Very Low PTCS <= 5%
Rating 5 4 3 2 1
ERA Risk Major impact to Moderate impact to Some impact to Minor impact to Negligible or no
Ratings achievement of achievement of achievement of achievement of impact to
Subsonic Transport Subsonic Transport Subsonic Transport Subsonic Transport achievement of
Consequences System Level Metrics, System Level System Level System Level Subsonic Transport
Technical Metrics, Technical Metrics, Technical Metrics, Technical System Level
Deliverables, and KPP Deliverables, and Deliverables, and Deliverables, and Metrics, Technical
Goals KPP Goals KPP Goals KPP Goals Deliverables, and
KPP Goals
Contingency No Acceptable Some Possible Single Acceptable A Few Known Several Acceptable
Solutions Alternatives Alternatives Alternative Alternatives Alternatives
Considerations
Reliability Factor Reliability May Not Fairly Confident Highly Confident Fairly Confident Highly Confident
Be Increased Reliability Will Reliability Will Reliability Will Reliability Will
Increase Somewhat Increase Somewhat Increase Increase
Significantly Significantly
13

14. ERA Project’s Risk Assessment Approach
• ERA has applied a tailored continuous risk management process that enable
risk- informed decision making
– For Phase 1 Portfolio (FY10-12)
– For Phase 2 Portfolio Development in FY12 for authorization to proceed through a
Key Decision Point R(KDP) Review Process
• Risk Management Process defined:
– Risk Factors for technical, cost and schedule
– Consequence and Likelihood Definitions and Scoring
– Parent-Child Risk Construct
– ERA Project Risk Reporting
– ERA Risk Factor Weighting
• Current Phase 1 Portfolio
• Assessment of Phase 2 Portfolio Opportunities
14

15. ERA Project Phase 1 Portfolio
Continuous Risk Management Process
• Risk Factors:
– Technical Risk
• Industry/OGA contributions
• Technical Complexity (Test Article, Experiment/Test)
• Technical Benefit /System Impact
– Cost Risk:
• Workplan cost estimate maturity/fidelity
• Technical Complexity
• Workplan resource availability
• Acquisition/Procurement cost
• Industry/OGA collaboration dependency
– Schedule Risk:
• Workplan maturity/WBS fidelity
• Technical Complexity
• Workforce resource availability
• Facility/Flight test asset availability
• Acquisition/Procurement schedule
• Industry/OGA collaboration dependency
15

16. Cost and
Schedule easy
ERA Risk Consequence Criteria to quantify
and assess.
ERA Risk Ratings Consequences
Rating
Decision/
Value Technical Cost Schedule Communication
Notification
Level 1 (APGs) any impact
Greater than 20% Integrated
Major impact to achievement of
increase over that Level 2 Milestone(s): Systems Aeronautics
Subsonic Transport System Level
5 Metrics, Technical Deliverables,
allocated budget (Sub- < 1 month impact Research Research Mission
Project, Element or Program (ISRP) Directorate
and KPP Goals Level 3,4 Milestone(s): ≤ 1
Task level) & Centers
month impact
Moderate impact to achievement of Between 15% and 20% Level 2 Milestone(s):
< 1 month impact ERA Project
Subsonic Transport System Level increase over allocated
4 Metrics, Technical Deliverables, budget (Sub-Project,
Management ISRP & Centers
Level 3,4 Milestone(s): ≤ 1 (PM)
and KPP Goals Element or Task level) month impact
Some impact to achievement of Between 10% and 15% Level 2 Milestone(s):
< 1 month impact
Subsonic Transport System Level increase over allocated
3 Metrics, Technical Deliverables, budget (Sub-Project,
ERA PM Centers
Level 3,4 Milestone(s): ≤ 1
and KPP Goals Element or Task level) month impact
Minor impact to achievement of Between 5% and 10% Level 2 Milestone(s): ERA Sub-
< 1 month impact ERA Project Manger
Subsonic Transport System Level increase over allocated Project
2 Metrics, Technical Deliverables, budget (Sub-Project, Managers
(PM)/ Deputy PM
Level 3,4 Milestone(s): ≤ 1 (DPM)
and KPP Goals Element or Task level) month impact (SPM)
Negligible or no impact to Between 0% and 5% Level 2 Milestone(s):
< 1 month impact
achievement of Subsonic Transport increase over allocated DPM, Element &
1 System Level Metrics, Technical budget (Sub-Project,
SPM
Task Leads
Level 3,4 Milestone(s): ≤ 1
Deliverables, and KPP Goals Element or Task level) month impact
Challenge to define Negligible, Minor,
16
Moderate, and Major impact

17. ERA Project Phase 1 Portfolio
Continuous Risk Management Process
• Consequence Elements:
– Technical Risk
• Achievement of Subsonic Transport System Level Metrics,
• Technical Deliverables,
• Key Performance Parameters (KPPs)
– Technical Challenges
– TRL Maturation
– Technology/Product Transition Roadmap
– Cost Risk :
• % increase over that allocated budget (Sub-Project, Element or Task level)
– Schedule Risk:
• Level 1 Milestones: ISRP Program Level
• Level 2 Milestones: Project Level
• Level 3,4 Milestones: Sub-Project and Task Level
17

18. ERA Project Phase 1 Portfolio
Continuous Risk Management Process
• Likelihood Criteria may be defined in either Generalist terms or Probabilistic
terms:
• Generalist: Improbable  Unlikely  May  Likely  Very Likely
• Probabilistic: % or occurrence, Probability distribution (i.e. 10-5), etc.
18

19. ERA Parent and Child Risks
• All identified project risks start as Child Risks. Risks that apply to project level
metrics are then elevated to the Parent level and are then managed by the Risk
Management Board.
– Child risks reported to Risk Management board, votes to make the risk a parent
based on the risks assessment at the project level.
– Allows risk owner at the sub-project, element or task level to assess the risk in terms
of their approved plan.
– Allows ERA project management to assess the risk at the project level where they
have a better understanding of the entire project
• Preventing filtering or tweaking of the risk at the sub-project/element/task level.
ID
WBS Parent Parent Risk Trend # Open Date Risk Title Risk Statement L C Affinity Group Owner
Given the high cost and schedule uncertainty Schedule (3)
1.00 PM Y ERA Schedule 68 14-Sep-11 ERA Schedule inherent to technology development projects, 4 3 Cost (3) Fay Collier
5
and Resources and Resources there is a possibility that planned funding and Tim Warner
schedules will not support meeting ERA goals.
Given that the large wind tunnel facilities at Schedule (3)
3.0.0, PT N ERA Schedule 6 15-Jun-11 Shared GRC share personnel and services, there is a 5 3 Cost (3) Ken
3.1.3, and Resources Personnel & possibility of schedule conflicts arising between Suder
g
3.3.4, Services at GRC ERA and other test programs (e.g., FAP: SUP,
Facilities FAP: HYP), resulting in potential ERA
propulsion test schedule slips.
Given the expectation for Continuing Technical (3)
1.00 PM N ERA Schedule 24 15-Jun-11 Continuing Resolutions during FY12, there is possibility 5 3 Cost (3) Fay Collier
and Resources g Resolutions that ERA technology development activities will Schedule (3)
be delayed, resulting in schedule slips and loss
of productivity
19

20. ERA Risk Reporting Rank Trend
Risk
ID #
Affinity Group
Approach
(M,W,A,R)
Risk Title
Denotes assessment and number of “Child” Discrete Roughness Elements
Cost (5)
risks.
Risk Matrix
1 h 11 Schedule (5)
Technical (5)
M
Laminar Flow Glove Experiment
(DRE) Programmatic Planning and
Control
Pultruded Rod Stitched Efficient
5
h
11,13 Cost (3) Unitized Structure (PRSEUS)
2 13 W
Schedule (5) Programmatic Estimates and
L Control
I 4 68, 69
K
E
3 g 65
Technical (5)
Schedule (5)
M Combustor Development and Test
L
3 .. .. ... 65,66,76,77 UHB Geared Turbo Fan Engine
g
I
4 66 Technical (5) M Development Noise
H
Characteristics
O
67
O 2 ..
.......
D 5 g 76 Schedule (5) R
ERA Key Decision Point (KDP)
Schedule
1 64
6 5 77 Schedule (5) R
Flow Control Experiment for AFC
Rudder
1 2 3 4 5
CONSEQUENCE 7 g 68
Schedule (3)
Cost (3)
A ERA Schedule and Resources
Note: The numbers on the risk matrix refer to the Risk ID numbers.
Criticality L x C Trend Approach 8 g 69 Technical (3) M ERA Technical Challenges
High Decreasing (Improving) M - Mitigate Insufficient Resources to Mature
Increasing (Worsening) W - Watch
Med A - Accept
9 i 67 Technical (3) R
Vehicles Concepts and Associated
Technologies to Simultaneously
Unchanged meet ERA Goals
R - Research
Low New Since Last Period
10 i 64 Schedule (5) W
Hybrid Wing Body (HWB)
Community Noise Assessment
Project “Parent” Risks
20

21. ERA Project Phase 1 Portfolio Weighting of Risk
Translating Child to Parent Risks
ERA Risk Assessment Matrix
ERA WBS Sub- Level
Matrix Title project Resource Milestone Milestone
Demonstrate low-weight, damage-tolerant stitched composite structural Complete Noise
concept on curved panel subjected to combined tension and internal Transmission
02.01.4 AT 1
pressure loads.(COLTS Large Scale Pressurized Fuselage Test Assessment of
Complete) (Proposed FY12 APG)Complete PRSEUS panels
KPP Technical Technical Subsonic Transport Technical Deliverable and
Milestone Deliverable KPP Goal APG Challenge Maturation System Level Metric Validation Method Cost Weight
Report development of
predictive noise
transmission models N/A N/A Yes N/A .75
for like structural
concepts
Very Low Low Medium High Very High
5 10 16 20 23 25
4 7 13 18 22 24
3 4 9 15 19 21
2 2 6 11 14 17 Very Low Low Medium High Very High
1 1 3 5 8 12 5 8 12 15 17 19
1 2 3 4 5 4 5 10 14 17 18
3 3 7 11 14 16
2 2 5 8 11 13
Weighted table, based 1 1 2 4 6 9
on WBS task weight. 1 2 3 4 5
*Values rounded
In this example, if this risk was assessed at a Likelihood of 4 and Consequence of 5 by the task lead. LxC: 4 x 5. The risk
would be scored at 24, red, at the ERA project level. By creating the weighting factor to reflect the task’s contribution to ERA
goals, the risk would be scored at LxC: 4x3 as the likelihood would not change, only the potential consequence. This allows the
task lead to asses the risk based on their understanding of their task and creates a tool for the ERA risk manager a quantifiable
21
and traceable method to accurate assess task risks and communicate at the project level.

22. ERA Project Phase 2 Portfolio Selection Criteria – Risk Posture
• Example of Risk scoring Low = 5, Medium = 3, High = 1
• Supporting rationale will be developed for each Low, Medium or High score
Technical Risk Score (1/3
Cost Risk Score (1/3 weighting) Schedule Risk Score (1/3 weighting)
weighting)
RISK Workf Industr Techn
Facility ITD Industry Facility ITD ITD Industry
orce y Procur- Procure- Workforce ical Cum
Availabil Comple Compon Availabi Complex Compl Compon
Avail Compo ement ment Availability Benefi Score
ity xity ent lity ity exity ent
ability nent t
(Wei
ghtin 10% 25% 10% 20% 35% 25% 10% 15% 20% 30% 25% 35% 40% 100%
g)
ITD
#1
Low Low Low Low Low High High High High High High High High 4.8
ITD
#2
Low Low Low Low Low Low Low Low Low Low Low Low Low 5.0
ITD Medi
#3 um
High High High High Low Low Low Low Low Low Low Low 2.8
ITD
#4
Low Low Low Low Low Low Low Low Low Low Low Low Low 4.6
ITD# Mediu Mediu Mediu
5
High High
m m m
Low Low Low Low Low Low Low Low 2.5
ITD# Mediu
6
High
m
Low Low Low Low Low Low Low Low Low Low Low 2.7
ITD# Mediu Mediu Mediu Medi Medi Mediu
7
Low Low Low Low Low
m
Medium Medium
m m um um m 5.0
ITD#
8
Low Low Low Low Low Low Low Low Low Low Low Low Low 4.4
ITD# Medi
9 um
Low Low Low Low Low Low Low Low Low Low Low Low 4.2
ITD# Mediu Mediu
10
Low Low
m
Medium
m
Low Low Low Low Low Low Low Low 3.9
22

23. Concluding Remarks
Opening Question:
• What is the appropriate risk management construct for an Aeronautics
Technology Development Project responsible for maturing airframe and
propulsion technologies from TRL 3 to 5/6 through Integrated Systems
Research?
Lessons Learned to date:
• Tailoring Risk Management Processes to Aeronautics Integrated Systems Research
Technology Development poses different challenges than spaceflight development
• The technology landscape is constantly changing with new technologies coming
online promising increased performance. Risk assessment must weigh the promise
of new performance against the confidence of what has been done before
• The realization of a risk is not failure, the knowledge gained identifies new foci for
integrated systems research to continue the technology’s maturation path to enable
technology transition into an aircraft system.
• These considerations do not require us to change the Project’s likelihood and
consequence risk criteria but consider the lens through which we view these scales
23

24. Questions?
• Douglas Brown, ERA Risk Manager
Douglas.brown@nasa.gov
757.864.3515
LaRC
• Gaudy Bezos-O’Connor, ERA Deputy Project Manger
Gaudy.m.bezos-oconnor@nasa.gov
757-864-5083
LaRC
• Steven Hirshorn, Integrated Systems Research Program Systems Engineering
& Integration Manager
Steven.r.hirshorn@nasa.gov
202.358.0775
HQ
24

25. 25