1. You Can’t Grade Your Own
Homework
Dr. James Ortiz, Deputy Director
Independent Program Assessment Office
Sixth Annual NASA
Program Management Challenge 2009
February 25, 2009
Page 1

2. Goal
The NASA independent review process helps ensure the
highest probability of mission success to Programs and
projects.
Page 2

3. Topics addressed
• What is an independent life-cycle review ?
• Why do we do independent reviews?
• Who is involved?
• What is the Standing Review Board (SRB) ?
• How are independent reviews done?
• Why can’t you grade your own homework?
• Summary
Page 3

4. What is an Independent Life - Cycle
Review ?
• A review of Programs and projects at each life-cycle milestone
performed by competent individuals who are not dependent or
affiliated with the Program or project and who do not have an
organizational or personal interest or stake in the results of the review
• The review objectively assesses:
– Adequacy and credibility of the technical approach (requirements,
architecture, design)
– Schedule
– Resources
– Cost
– Risk
– Management approach
– Compliance with Agency policy (NPR 7120.5, NPR 7123.1)
– Readiness to proceed to the next phase
• The results of the independent review are advisory to the
Program/project and to the decision authority
Page 4

5. Why Do We Do Independent Reviews?
1. Agency wants Program/projects to receive independent assurance
that they are on-track
2. NASA senior management wants:
– Independent validation at key decision points of the
Program/project’s readiness to proceed into the next phase of
its life-cycle
– Externally-imposed impediments to Program/project success
are being removed
3. Agency needs to provide external stakeholders assurance we can
deliver to our commitments
4. Significant additional benefit is that preparation for the review
milestone allows for a holistic examination by the Program/project
and the review team
Page 5

6. Who Is Involved?
• Per the Agency governance structure, the independent life-
cycle review process is implemented as a collaborative effort
between:
– Agency Senior Management
– Center Management
– Technical Authorities (TA)
– Program/Project Management
Decision Authority Technical Authority
Associate
Administrator
Center
NASA AA MDAA NASA CE PA&E
Director
Programs Approve Approve Approve Approve
Establish Category 1
SRB, Approve Approve Concur Approve Approve
Projects
Develop
ToR.
Approve Category 2
Approve Approve Approve*
Chairperso Projects
n, RM, and
Other
Board
Members Category 3
Approve Approve
Projects
Page 6

7. What is the SRB?
(NPR 7120.5 & SRB Handbook)
• Independent life-cycle reviews are conducted by a Standing
Review Board (SRB)
– The SRB has a single chairperson and a NASA review manager
– The SRB remains intact, with the goal of having the same core
membership, for the duration of the Program or project
– Board members must be independent of the programmatic and
technical authority chain of command for the Program/project
– Board members are free from organizational and personal conflict of
interests
– The main attributes of SRB members are currency, competence and
independence
• SRBs provide the Agency a non-advocate, objective, and
competent assessment of the Program/project as it advances
through its key decision points (KDP)
Page 7

8. How Is It Done?
(NPR 7120.5D)
Page 8

9. How Is It Done?
(IPAO process)
I N P U T S
P/p Documentation
Agency Review Agency/Directorate/Centers Cost & Schedule SRB Briefings to Minutes/Decisions/Actions
Schedule Documentation & Documentation Program/Project
Review Chair Requirements Program/Project
Nomination Briefings
PLANNING PREPARING REVIEWING REPORTING CLOSING-OUT
Verbal Pre-Briefings to P/p Review Closeout
Monitor Agency Initiate Contact and Determine Receive CADRe Inputs
Write one-page summary Customer
Baseline Program/ Review Budget SRB Kick-Off Meetings
Write the Report Feedback
Project Review Identify and obtain approval of Attend Reviews
Prepare the Summary Develop Lessons
Determine IPAO SRB Chair/RM/SRB members Attend Site Visit
Briefings Learned
Review Budget Determine funding mechanism Develop ICE/ICA/ISA
Present the Briefings: Process Review
Review Assignments Develop the TOR
- Program/ project and Improvement
Develop Schedule/Logistics
- CMC Administrative
of the Review
- MD PMC Close-out
Develop the Cost Plan
- APMC (if required) Team Recognition
Develop the Schedule Plan
P R O D U C T S
Individual Review Budget ICE/ICA/ISA One Page Summary Review Summary
RM/CA/SA Assignments ToR, Cost Plan, Schedule Plan SRB Findings SRB Report Contracts Close-out by
Chair and Team Nomination RFA RRD Report COTR
and Guidance Letters Recommendations SRB Briefings Review Records
Review Schedule
RRD Report
Contract Task Statement
Process improvements
R&R for SRB
Page 9

10. Recent improvements
• The following diagram of the SRB “engagement” timeline approved by
the Agency Program Management Council (APMC) resulted from
process improvement activities jointly performed between IPAO,
Mission Directorates, Programs, and the Office of the Chief Engineer
(OCE).
Path to Orion PDR
DAC SRB observer role
11/08 5/09 “one pager” by Project, includes Plan to KDP
S/S Reviews
Internal
6/09 8/09 Project PDR
8/09 RFAs Findings
SRB assessment of
entry criteria to SRB Site “One pager” + 1 day (joint activity)
Project/Program Review ‐ Quick status
PMR Rev 1 ‐ Only big issues, not concerns/observations
11/08 3/09 4/09 8/09 ‐ Initial ICE
SRB
l‐‐‐‐l PDR/NAR KDP
1 – 6 l ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐l
mos 5‐10 days 30 days
Integrated
‐ Tech
‐ Cost
“Pre‐work” ‐ Sched Briefings: P/p, CMC, DPMC, APMC
‐ Risk
Model Development
Joint with IPAO/Project Independent Life Cycle Review (ILCR)
ICA/ICE/ISA reconciliations
Page 10

11. Life-Cycle Review Success Criteria
1. Alignment with and contributing to Agency needs, goals, and
objectives, and the adequacy of requirements flow-down from those
2. Adequacy of technical approach as defined by NPR 7123.1 entrance
and success criteria
3. Adequacy of schedule
4. Adequacy of estimated costs (total and by fiscal year), including
Independent Cost Analyses (ICAs) and Independent Cost Estimates
(ICEs), against approved budget resources
5. Adequacy/availability of resources other than budget
6. Adequacy of risk management approach and risk
identification/mitigation
7. Adequacy of management approach
Page 11

12. Summary: Why You Can’t Grade Your
Own Homework?
• The independent life-cycle review supports the NASA Administrator’s
Charge: “You can not grade your own homework regardless of the
position you hold as part of the Program or project”
• The independent life-cycle review process is an integral part of the
Agency’s check and balances built into the NASA governance structure
and complements the programmatic and technical lines of command
and authority
• It is implemented by Standing Review Boards (SRBs) staffed with
members that are current, competent and independent
• Its processes are continuously assessed for improvement by IPAO and
its stakeholders
• The independent life-cycle review process helps ensure the highest
probability of success of the Agency’s Program and projects
Page 12

13. Back-up Slides
Page 13

14. Terminology
• Program
– A strategic investment by a Mission Directorate or Mission Support Office that has a
defined architecture and/or technical approach, requirements, funding level, and a
management structure that initiates and directs one or more projects. A Program
defines a strategic direction that the Agency has identified as critical.
• Project
– A specific investment identified in a Program Plan having defined requirements, a life-
cycle cost, a beginning, and an end. A project yields new or revised products that
directly address NASA’s strategic needs.
• Standing Review Board (SRB)
– The entity responsible for conducting independent reviews of the Program/project per
the life-cycle requirements. The SRB is advisory and is chartered to objectively assess
the material presented by the Program/project at a specific review.
• Decision Authority (DA)
– The Agency’s responsible individual who authorizes the transition of a program/project
to the next life-cycle phase.
• Key Decision Point (KDP)
– The event at which the Decision Authority determines the readiness of a
Program/project to progress to the next phase of the life cycle (or to the next KDP).
Page 14

15. Program Life Cycle (NPR 7120.5)
Page 15

16. Project Life Cycle (NPR 7120.5)
Page 16

17. Governing Documentation
NPD 1000.0
NPD 1000.3
NPD 1000.5
Engineering NPD Program Mgt. OSMA NPD Health and Mission Support
Office NPDs
NPD Medical NPD
NPR 7123.1 and Other NPR 7120.5 and OSMA NPRs NID 1240-41 and Support Org
Engineering NPRs other PM NPRs OCHMO NPRs NPRs
Health &
Engineering Program/Project SMA MSO Functional
Medical
Requirements Mgmt Requirements Requirements Requirements
Requirements
Mission Directorate Center Engineering &
Programmatic Management
Requirements Policies and
Practices
Program Plans
Project Plans
Page 17