The 7 Things I Know About Cyber Security After 25 Years | April 2024
Amer.tahani
1. Headquarters
Independent Program & Cost Evaluation
State-of-the-Art
in
Independent Review Execution
Tahani Amer and Kaiser Adeni
Review Managers
Independent Program Assessment Office
8th Annual 2011 NASA Program Management Challenge
February 10, 2011
2. Outline
Headquarters
Independent Program and Cost Evaluation
• Look Back at FY10 Reviews
• Review Process Improvements
• SRB Balance & Structure
• SRB Members Roles & Responsibilities
• SRB Coordination (between Mission Directorates,
Centers, and Programs/projects (P/p))
• SRB Products
• SRB – HB updates
• IPAO State-of-the-Art Activities
• Look Forward
2
3. Look Back at FY10 Reviews
Headquarters
Independent Program and Cost Evaluation
• 8 Programs and 20 project review activities executed in FY10:
Preliminary Design Review: MAVEN
Critical Design Reviews: GRAIL, RBSP, GPM, TDRS, LDCM, MMS,
OCO-II
System Integration Reviews: MSL, GRAIL and Juno
ORR/FRR/PLAR: WISE
Program Approval Review: RPS and LQP
Program Implementation Review: SCaN, ESMP, D/NF
Special Reviews: MSL, CxP, Aquarius, SOFIA
8 Non-IPAO reviews: 2- GOES-R, ASP, AvSP, ATP, FAP, ENAS, CAS
3
6. FY10 IPAO Completed Reviews
IPAO supported 8 non-IPAO review activities in FY10
Headquarters
Independent Program and Cost Evaluation
6
7. Review Process Improvements
Headquarters
Independent Program and Cost Evaluation
• SRB process improvements:
• Quick Look Reports (one-pagers);
• 30-day reporting requirement;
• Increased coordination for programmatic assessments;
• Readiness-to-proceed assessments;
• Alternate opinions for non-consensus boards;
• Key Decision Point (KDP) Decision Memo improvements;
• Deferral of program reviews approved at APMC;
• Institutionalized electronic signature of SRB approval letters;
• Streamlining of ToR content as defined in draft NPR
7120.5E;
7
8. Review Process Improvements
Headquarters
Independent Program and Cost Evaluation
• SRB process improvements (continued):
• Better coordination of SRB team nomination process; early
stakeholders involvement in the process to ensure proper
team balance of competency/currency/independence;
• Final reports are posted on APMC electronic repository
(https://nx.arc.nasa.gov/nx/dsweb/View/Collection-93608); notification at
APMC when reports are posted;
• Strengthening and improving the rigor and integration of the
technical and programmatic assessments;
• Working with SMD and the Centers to formulate principles on
conducting joint Program/project reviews when there is a
significant external partner involved. Efforts are evolving
w/ExoMars and JPSS.
8
9. SRB Balance & Structure
Headquarters
Independent Program and Cost Evaluation
• SRB members should be competent, current, and
independent from the management or advocacy chain of
the P/p, with membership balanced between the host
Center and other organizations to ensure the needs of
the convening authorities are met.
• Although balance of each SRB member is important,
ultimately the goal is to have the SRB, as a whole,
balanced.
• More inclusive set of discussion with Center and TA
• It is not a numerical formula, but it goals to meet
Agency’s goals
9
10. SRB Balance & Structure
Headquarters
Independent Program and Cost Evaluation
Competency
Relevant experience and expertise with the technical, specific
technologies and programmatic domain of the P/p under review.
Currency
Addresses recent/current experience and expertise in
programmatics or the technical domain(s) of the project or program
under review.
Understanding of current Agency governance, project management
and systems engineering policies, procedures and methods,
specifically NPR 7120.5, NPR 7123.1, NPD 1000.5, specific Center
practices and procedures and the SRB Handbook.
Independence
Not in the programmatic chain of command of the program or
project and have no conflicts of interest either personally,
institutionally or organizationally.
10
11. SRB Members Roles & Responsibilities
Headquarters
Independent Program and Cost Evaluation
• A member of an SRB is an “agent” of the convening authorities:
More emphasis on programmatic risk assessment and analysis;
More emphasis on the individual member independent report (IMIR);
Advise the SRB Chair and the Review Manager (RM) on areas that
require particular attention by the SRB per their area of expertise;
Support the SRB “Quick-Look Report” summaries and briefings to the
Convening Authorities (CA), and provide expert opinion to SRB
preliminary and final reports.
11
12. SRB Coordination
Headquarters
Independent Program and Cost Evaluation
• Coordination with the Mission Directorates:
Early and frequent coordination is conducted with the Mission Directorate Program
Executive (PE) to:
Establish SRB membership
Chair with support from RM establishes the discipline areas to be covered
PE/MDs provide a list of potential candidates. CAs approve or provide alternate
nominations
RM leads the vetting effort for Organizational and Personal Conflict of Interest
(OCI/PCI) in coordination with the LaRC Legal Office and Contracting Officer(s)
Finalize Terms of Reference (ToR) content
Content is reviewed and agreed-to prior to final approval by the CA
Identify any additional Mission Directorate review criteria
Establish timing of the Site Visit
Establish post-review out brief schedule
CMC/DPMC briefing
APMC briefing (Cat 1 and some Cat 2 Programs/projects)
APMC special topics for Cat 2 project life cycle reviews
Pause and Learn (PAL)
12
13. SRB Coordination
Headquarters
Independent Program and Cost Evaluation
• Coordination with the Center
Coordination with the Center is conducted to:
Coordinate and develop a list of SRB candidates from the Center based on the
discipline areas established by the Chair
Facilitate the review of the ToR content and ensure that the Center specific
requirements are being met
Golden Rules
JPL Rules
Facilitate Center CA approvals of ToRs, team nominations, etc.
Establish dates for post-review CMC briefing
Establish a Community of Practice (CoP) at each Center (in progress)
Knowledge sharing of best practices and expectations for independent assessment
Review manifest coordination and approval
Identify Center personnel to participate in SRBs
Assist with nomination of high potential candidates for RM assignment with IPAO
Request potential Center Review Managers as detailees to IPAO
Develop a CoP for the Agency after completion on the discussion with Centers
13
14. SRB Coordination
Headquarters
Independent Program and Cost Evaluation
• Coordination with Program/project:
Program/project pre-planning coordination begins ~120 days prior to Site Visit to:
Establish Site Visit review requirements
Discipline areas to be covered
Data drop timeline for cost and schedule products
Establish agreement on Terms of Reference (ToR) review criteria
Single ToR for LCR (0.5E)
Readiness to Proceed Review
Establish timing of the Site Visit
Establish post-review out brief schedule
Quick- Look Report (One Pagers)
P/p briefing
CMC briefing
DPMC briefing
APMC briefing (Cat 1 and some Cat 2 P/p)
14
15. Headquarters
SRB Products
Independent Program and Cost Evaluation
SRB products produced for each Site Visit:
Briefings
One-pager (Quick Look Report)
P/p Briefing
CMC Briefing
DPMC
APMC Briefing (if a Cat 1 Project)
ToRs, team nomination letters, alternate opinions
Vetting Package
OCI/PCI mitigation plans/annual vetting
SRB member resumes/bios
SRB approval letters
Reports
Final SRB summary report to include each SRB individual reports
to DA. https://nx.arc.nasa.gov/nx/dsweb/View/Collection-93608
IPAO Review Record
RRD for the Agency
15
16. Headquarters
SRB –HB Update
Independent Program and Cost Evaluation
Guidance to the P/p and SRB members
The SRB-HB is posted on the
http://nodis.hq.nasa.gov/policy_letters/NM_7120-81_.pdf
Updates to comply with NPR 7120.5E
16
17. IPAO State-of-the-Art Activities
Headquarters
Independent Program and Cost Evaluation
• JCL assessments and improved programmatic analysis:
Using Defense Contract Management Agency (DCMA) and Government
Accounting Office (GAO) criterion as standard evaluation criteria for
schedules
Implementation of standard analysis timeline allowing for time to work
with the projects to improve cost and schedule concerns
Analysis processes have been defined and documented in Standard
Operating Procedure Instructions (SOPI)
• Better risk assessments:
Using state-of-the-art tools for integrated risk analysis
Establishment of Schedule Analysis Working Group (SAWG) to develop a
CoP for programmatic risk analysis
• Forensics Study:
Analysis of the SRB findings
Identify trends and systematic issues
Provide recommendation to improve PM at NASA
17
18. IPAO State-of-the-Art Activities
Headquarters
Independent Program and Cost Evaluation
• Lessons Learned (LL) after each review
IPAO members share the LL with RMs and PAG Analysts
Database of LL
• ILCR - Customer/CA/SRB member surveys
Developed ILCR surveys & Approved by CA
Implementing the survey: RBSP
• Training: RMs & Chairs
• Outreach Effort: Articles, PM Track, Visits to Centers, PAL
• Developmental Program: Detail opportunity to support
Agency Project Management
18
19. Look Forward
Headquarters
Independent Program and Cost Evaluation
• Implementation of the NPR 7120.5E
• Update the SRB Hand Book
• Continue coordination with MDs, Centers, and P/p
• Communities of Practice
• SRB Balance
• Independent Reviews involving external partners
• Independent Reviews and Technology P/p
19
21. One Step PDR
Life Cycle Review Overview
Headquarters
Independent Program and Cost Evaluation
KDP-C
KDP-B
(2)
Quick Look Report
PDR Readiness
Assessment (2) PDR-LCR
(30- 90 days) (30 Days)
-Required prior to LCR Technical Baseline with C/S/R
-Report to DA for life cycle reviews and Integrated Assessment of
preceding KDP B&C and during any Technical and Programmatic
major replan or rebaseline (3) Baseline
Not To Scale
Programmatic Data Drops to
SRB (includes JCL Model) P/p Center MD
Deliveries start at 100 days
before site review
Brief Brief Brief
(2)
CheckPoint if needed. Periodic SRB Involvement as Appropriate FOOTNOTES:
1. A One Step Review may be used for any LCR.
2. Appendix I provides information on the readiness assessment,
quick-look reports and checkpoints associated with life cycle
reviews.
3. For all other life cycle reviews, report to Chief Engineer if
significant unresolvable disagreements. 21
Page 21
22. Two Step PDR
Life Cycle Review Overview
Headquarters
Independent Program and Cost Evaluation
KDP-C
KDP-B PDR LCR
(2)
Quick Look Report (2)
Quick Look Report
PDR Readiness
Independent Integrated
Assessment (2) PDR
PDR Assessment
(30-90 days) (1-6 months) (30 Days)
-Required prior to LCR Technical Baseline Integrated
-Report to DA for life cycle reviews with Cost, Assessment of
preceding KDP B&C and during any Schedule, and Risk Technical and
major replan or rebaseline (3) Information Programmatic
Baseline
Not To Scale
Programmatic Data Drops to Resolve Tech
SRB (includes JCL Model) Issues/Risks, Update P/p Center MD
Cost/Schedule Baseline
Brief Brief Brief
(2)
CheckPoint if needed. Periodic SRB Involvement as Appropriate
FOOTNOTES:
1. A Two Step Review may be used for any LCR
2. Appendix I provides information on the readiness assessment,
quick-look reports and checkpoints associated with life cycle
reviews
3. For all other life cycle reviews report to Chief Engineer if 22
significant unresolvable disagreements 22 Page