SlideShare uma empresa Scribd logo
1 de 35
Baixar para ler offline
Overview of GAO Quicklook

          PM Challenge

         February 2012


Katie Gallagher and Tracy Osborne
             OCFO/SID
Outline
• Background
      – Why is this important?
• GAO’s Approach
      – What projects are involved?
      – Why does GAO request technical data?
• What should I expect if my project is in
  Quicklook?
• What does NASA’s data say?
• Agency Messages

1/22/2012                                      2
Background
• The GAO’s Assessment of Large-Scale [NASA] Projects is
  Congressionally-mandated
• GAO sends its findings to Congress in an annual “QuickLook
  Book” (QLB) Report focusing on changes in project cost and
  schedule and providing GAO’s explanations for these changes
• GAO is frequently called upon to testify about the results.
• Engagement objective: “Identify and gauge the progress and
  potential risks associated with selected NASA acquisitions.”
  (i.e., projects)
• GAO uses data from QuickLook Book (QLB) in other audits, such
  as the ongoing High Risk audit, and vice versa.

 1/22/2012                                                       3
Why the Stakeholder Interest?
• NASA’s historical inability to control cost and schedule
  growth
      – JWST, CxP, MSL growth cited in several provisions establishing new
        reports.
      – A 2004 GAO finding that “NASA lacks discipline in cost estimation.”
      – Multiple Operating Plans per year to address cost growth or phasing.

• Based on past Performance NASA is required to
  report to OMB, Congress and GAO routinely on
  Project cost and schedule.
      – All appropriations since FY 2008 have included direction for GAO to
        “Identify and gauge the progress and potential risks associated with
        selected NASA acquisitions.”

1/22/2012                                                                      4
Typical Results from Quicklook Reports
• Most of NASA’s cost growth for the current
  portfolio is attributed to projects baselined
  prior to 2009
• NASA projects do not meet GAO-identified
  best practice standards for technology
  maturity and design stability
• Other challenges that affect NASA project
  outcomes include Funding, Launch Vehicle,
  Development Partners, Parts Quality,
  Contractor Management

1/22/2012                                         5
Why do they ask about TRL’s and drawings?

   GAO’S APPROACH


1/22/2012                                      6
GAO’s Report is in Two Parts
• Agency Observations
    – Examines adherence to NASA policies such
      as 7120.5E and 1000.5A
    – Examines potential systemic reasons for
      NASA’s cost growth
    – Assesses NASA’s efforts to improve cost and
      schedule growth, including JCL policy

• Project Assessments
    – Provides an update on status of each
      project included in the audit
    – Outlines challenges specific to each project
      that GAO believes may lead to cost growth

   GAO Quicklook assesses performance of NASA’s major projects
        and the Agency’s management of those projects.
  1/22/2012                                                  7
How does a project get picked?
• GAO evaluates all spaceflight projects in formulation if a contract is
  released that has development content and is greater than $25 M
     – Can include projects in Phase A and Pre-Phase A if they have such a
       contract
• GAO evaluates all spaceflight projects in development with a life cycle
  cost greater than $250 M
• GAO continues to track projects in Operations until NASA negotiates
  removal from Quicklook
     – Minimal data collection on Operations projects; main focus is cost and
       scientific results
• For projects in formulation, GAO expects NASA to provide cost and
  schedule ranges
• For projects in development, GAO expects NASA to provide baseline
  cost, schedule, design and technical information


1/22/2012                                                                       8
Projects Included in Audit
                                                      2008                              2009                              2010                              2011
                                                                                           Requirement for Inclusion
                                          * Estimated Lifecycle Cost        * Estimated Lifecycle Cost        * Estimated Lifecycle Cost        * Estimated Lifecycle Cost
                                          >$250M                            >$250M                            close to $250M (if it is called   close to $250M (if it is called
                          Lifecycle       * In development                  * In development                  out in the budget as a            out in the budget as a
                                          * In formulation if it also has   * In formulation if it also has   discreet project then it may      discreet project then it may
                        Phase as of
                                          released a development            released a development            be included)                      be included)
        Project          Entrance         contract valued at >$50M          contract valued at >$25M          * In development                  * In development
                        Conference                                          * If in operations, it is not     * In formulation if it also has   * In formulation if it also has
                        (April 2011)      (consistent with other            included                          released a development            released a development
                                          external reporting                                                  contract valued at >$50M          contract valued at >$50M
                                          requirements)                                                       * In operations, in the prime     * In operations, in the prime
                                                                                                              mission phase                     mission phase
 1   Aquarius            Development                     X                                 X                                 X                                 X
 2   Ares                 Formulation                    X                                 X                                 X
 3   Dawn                  Operations                    X                                                                   X                                 X
 4   Fermi/GLAST           Operations                    X                                                                   X                                 X
 5   GPM                 Development                     X                                 X                                 X                                 X
 6   Glory               Development                     X                                 X                                 X                                 X
 7   GRAIL               Development                                                       X                                 X                                 X
 8   Herschel              Operations                    X                                 X                                 X                                 X
 9   ICESat II            Formulation                                                                                        X                                 X
10   JWST                Development                     X                                 X                                 X                                 X
11   Juno                Development                                                       X                                 X                                 X
12   Kepler                Operations                    X                                 X                                 X                                 X
13   LADEE               Development                                                                                         X                                 X
14   LDCM                Development                     X                                 X                                 X                                 X
15   LRO                   Operations                    X                                 X                                 X                                 X
16   MAVEN               Development                                                                                         X                                 X
17   Mars 2016 (TGO)      Formulation                                                                                                                          X
18   MMS                 Development                                                       X                                 X                                 X
19   MPCV               Pre-Formulation                                                                                                                        X
20   MSL                 Development                     X                                 X                                 X                                 X
21   NPP                 Development                     X                                 X                                 X                                 X
22   OCO/OCO-2           Development                     X                                                                   X                                 X
23   Orion                Formulation                    X                                 X                                 X
24   Phoenix              Closed Out                     X
25   RBSP                Development                                                       X                                 X                                 X
26   SDO                   Operations                    X                                 X                                 X                                 X
27   SLS                Pre-Formulation                                                                                                                        X
28   SMAP                 Formulation                                                                                        X                                 X
29   SOFIA               Development                     X                                 X                                 X                                 X
30   Solar Probe Plus     Formulation                                                                                        X                                 X
31   TDRS                Development                                                                                         X                                 X
32   Wise                  Operations                    X                                 X                                 X                                 X
1/22/2012                                                                                                                                                                    9
GAO “Business Case” Model
                                                PDR        CDR      SIR
                                                                                                                                 Production
                 Pre-         Phase       Phase                                                                                  Rarely Applies
    NASA        phase A         A           B
                                                         Phase C              Phase D                    Phase E
                                                                                                                                 to NASA
                                      KDP-B      KDP-C              KDP-D
                                                                                                                                 Projects
                      KDP-A                                                               KDP-D


    DOD           Technology Development
                                                         Engineering Manufacturing and
                                                                 Development
                                                                                                Production and Deployment
                                                                                                                            Operations and
                                                                                                                               Support


                                              PDR MS-B     CDR                           MS-C


                  Concept and Technology                 Engineering Manufacturing and
    GAO               Development                                Development
                                                                                                       Production



                                          Knowledge Knowledge                        Knowledge
                                            Point 1   Point 2                          Point 3
•   Knowledge Point 1: Resources and Requirements Match
     – Indicated by a high level of technology maturity
            •   Heritage is technology is appropriate in form, fit and function
            •   New Technologies at least at TRL-6 (for NASA)
•   Knowledge Point 2: Product Design is stable “about midway through development”
     – Best practice is to achieve design stability by midpoint of development
     – GAO cites 90% of engineering drawings releasable as figure of merit
•   Knowledge Point 3: Manufacturing Processes are mature
     – Product shown to be producible within cost, schedule, and quality targets and have demonstrated
       reliability and Design demonstrates needed performance through realistic system-level testing
1/22/2012                                                                                                                                         10
Data Requested Annually
                                                            Reporting Year
Data Category                Element               2008     2009     2010      2011      Frequency
                        Design Stability             X        X         X        X     Annual + Updates
                      Critcal Technologies           X        X         X        X     Annual + Updates
  Technical         Heritage Technologies                     X         X        X     Annual + Updates
                     Software Complexity                                X        X          Annual
                      Quality Parts Issues                              X        X          Annual
                         MPAR Baseline               X        X         X        X        As Occurs
                         ARRA Funding                                   X        X       Semi-annual
                    KDP-B Est. LCC Range             X        X         X        X       Semi-annual
                        KDP-C Baseline                                  X        X        As Occurs
    Cost
                               ICEs                                     X        X        As Occurs
                               JCLs                                     X        X        As Occurs
                       Project-held UFE                                 X        X     Monthly (in MSRs)
                     MD-held UFE by year                                         X          Annual
                        Key Milestones               X        X         X        X       Semi-annual
  Schedule
              Agy vs Mgmt Schedule Comparison                                    X       Semi-annual
                       Basic Information             X        X         X        X       Semi-annual
  Contracts         Award Fee Strucuture                      X         X        X       Semi-annual
                     EVM (CPR Formats)                                  X        X         Monthly
                            FAD/PCA                  X        X         X        X        As Occurs
                          Project Plan               X        X         X        X        As Occurs
                          Control Plans              X        X         X        X        As Occurs
                      PDR/CDR Packages               X        X         X        X        As Occurs
Documentation               CADRes                                      X        X        As Occurs
                          SRB Reports                                   X        X        As Occurs
                   KDP-C Decision Memos                                          X        As Occurs
                      KDP-C Datasheets                                           X        As Occurs
                      MSR Presentations                                 X        X         Monthly

   1/22/2012 GAO   reviewed this list for 2011, and agreed to reduce frequency of some requests    11
So, my project is in Quicklook…

   WHAT SHOULD I EXPECT?


1/22/2012                            12
The Quicklook Annual Cycle (2011)

               DCI-                  Action Item Response               Addt’l Actions
             contract
                                                          Site Visits                                     Agency
                                                                                         2-Pgr Cmts
              DCI-          Site Visits –                                                                  Cmts
                                                           – MSFC
              C&s           JPL &GSFC                        &JSC
              DCI-                                                      DCI-                                       DCI-
                                    Other Data Requests                            Addt’l Data
              proj                                                      C&s                                        C&s




 Entrance                                                                                       Exit          Final Report
Conference                                                                                   Conference




                     DCI = “Data Collection Instrument” Three types:
                        1.Project DCI – for technical data
                        2.Cost and Schedule (C&S) DCI – provided by OCFO
                        3.Contracts DCI – provided by OP
1/22/2012                                                                                                                 13
Process for Working with GAO
                                                                        Extra explanation may be provided
      Quicklook Technical POC (TPOC)
                                                                        to help contextualize the data or to
      discusses request with GAO to
                                                                        better assist the GAO in answering
          understand purpose for
                                                                                  their questions
             requested items


GAO transmits                 Project drafts            Response reviewed             QLB TPOC
request                       response                  by HQ & Ctr Mgmt              transmits response
                                                                                      to GAO

     • The TPOC and                 • Provide               • Possible                    • ALR, PM, PE,
       relevant Audit                 document                reviewers                     and reviewers
       Liaisons (ALR)                 and/or                  include PE, MD,               will be copied
       should be                      explanation             PCFO,
       copied on all                                          Procurement,
                                    • When
       requests                                               OCE, OSMA, LSP
                                      document or
     • PM please                      data is not           • QLB TPOC and
       forward                        available,              MD ALR will
       requests to ALR                explain why             coordinate the
       and QLB TPOC                                           appropriate
                                    • Draft response
       when necessary                 provided to PE,         reviews.
                                      ALR, and QLB
                                      TPOC




                   Standard 3 week turn-around when not otherwise stipulated by GAO
   1/22/2012                                                                                               14
GAO Data Access Rights
• Legislatively, the GAO has right of access to the data, regardless
  of procurement sensitivity or classification
• NASA can negotiate with GAO regarding the provision of data; if
  there is a disagreement, GAO’s recourse is legislative action
• GAO is bound by the same rules as NASA for handling the data,
  and NASA can specify data restrictions
• NASA actively works with GAO on data formats, types and
  context for interpretation


NASA provides the data requested and works with the GAO to add
context and help the GAO analysts to understand NASA’s data.
  1/22/2012                                                        15
Data Collection Instruments
                                                            Example Technical DCI
GAO Data Collection Instrument:             Design Stability
                                                                                                                                                                                        •   GAO asks PMs to fill in
Date Submitted to NASA Project Office:
NASA Project Office Response Required by:
                                            5/3/2011
                                            5/6/2011
                                                                                                                                                                                            a Data Collection
GAO Contact:
Project:
                                            Joe Analyst
                                            Awesome
                                                                                                                                                                                            Instrument (DCI)
NASA Project Office Instructions            Please provide or update the total number of releasable engineering drawings and the total number expected at key dates listed below. The
                                            term "releasable" refers to those drawings (physical hard copy or computer-generated packages) that are sent to be manufactured. Please
                                                                                                                                                                                            annually and following
                                            calculate the number of drawings by using the number sheets within a package, not just the number of packages. Sometimes these
                                            drawings are considered the "build to" drawings. These drawings include details on the parts and work instructions needed to make the           PDR and CDR
                                            product and reflect the results of testing. Change any dates or data as needed.


Key Event Dates Defined: The PDR column should reflect information as of the Preliminary Design Review prior to KDP C (NAR). The CDR column should reflect information as of the
                                                                                                                                                                                        •   Data collection
Critical Design Review. The "Latest" column should reflect the the number of drawings releasable and expected as of the date this spreadsheet is returned to GAO.
                                                                                                                                                                                            includes
Overall Design Knowledge
KP ESTIMATE                                 PDR                          CDR                              September 2010 update        Latest                                                – Critical and heritage
Date
No. of Releasable
                                                    7/12/2010               Scheduled for July 2011              9/16/2010                      5/3/2011
                                                                                                                                                                                               technology TRLs
                                                                         1537 sheets expected for
                                                                                                                                       947 spacecraft sheets
                                                                                                                                       released to date. 299                                 – Drawing release
                                                                                                                                                                                             – Parts Quality issues,
                                                                         spacecraft, 818 drawings         10 released for spacecraft, drawings released for
Drawings                                                               0 expected for instruments.        16 released for instruments. instruments.
No. of Total
                                                                                                                                       2437 sheets of spacecraft                               including impact ($)
                                                                                                                                                                                             – Software TLOC and
                                                                         2437 sheets expected for                                      drawings planned, 1023
                                            1879 sheets expected for     spacecraft, 1023 drawings                                     drawings expected for
Drawings Expected                           spacecraft.                  expected for instruments.                                     instruments.
Source                                      Project Office               Project Office                   Project Office               Project Office                                          Blocks
If the total number of drawings is not known or if the total number of drawings has changed, please provide an explanation in the space below. If the project has                            – Partners
                                                                                                                                                                                        •
completed CDR, please describe any measures and metrics used in addition to engineering drawings to assess design stability. If the project has yet to complete its
CDR, please identify the measures and metrics the project intends to use to assess design stability.
See below for Spacecraft and Instruments comments. The project assesses design stability primarily thru the Systems Engineering organization. A weekly Systems Design Team
                                                                                                                                                                                            GAO requests data as
regularly addresses design, design margins, analyses, interfaces, subsystem and instrument design topics, and proposed changes. Proposed design changes are then processed thru
standard Configuration Control Boards. In addition, weekly status reviews as well as a Monthly Management Review take place that addresses overall status and issues.                       of PDR, CDR, and
                                                                                                                                                                                            current
Comments & Issues: Enter comments and issues below or on a separate sheet.
                                                                                                                                                                                        •   Procurement and
  PMs should NOT leave any sections blank; if the data is                                                                                                                                   OCFO fill in similar
                                                                                                                                                                                            DCIs on contracts,
  not available or not applicable, say so and explain why.                                                                                                                                  cost, and schedule
               1/22/2012                                                                                                                                                                                         16
Site Visits
• GAO will prepare for site visits with the projects by reviewing project monthly
  status review (MSR or PSR) packages; discussion questions will be transmitted
  in advance of the visit.
• GAO may not visit every project included in the Quicklook; projects not visited
  will receive written questions based on the content of their monthly review
  packages.
• PMs are invited to bring any charts or personnel that they need to help answer
  the questions; however, GAO prefers to have an informal discussion – NOT a
  lecture.
• When preparing responses, keep in mind that Quicklook is about cost growth
  against the Agency’s MPAR Baseline; responses should keep in mind the full
  mission cost – including launch vehicle and UFE
• Refer questions about things you are not responsible for to correct organization
     – LV questions should generally be directed to LSP
     – MD-held UFE questions should be directed to the MD, although it is ok to state the
       value
• When in doubt, it is ok to take an action to provide a response after the visit.
  This is the project manager’s opportunity to add context and explain
  all the good work their team is doing.
  1/22/2012                                                                          17
2-Page Project Summary Anatomy
       Project Challenges   Overview             Project Timeline       Project Update




Project
Essentials




Contract
Info




     Project Performance               Summary                      Project Office Comments

       1/22/2012                                                                              18
Draft 2-Page Project Summaries
              What to look for                                How to Respond
• Overview and Project Essentials                1.  Project comments
  – Is the information correct?                      – Will be printed in the “Project Office
• Project Update and Summary                            Comments” section of the final report
                                                     – Limited to 500 characters (appr 100 words)
  – Does the GAO accurately portray the
    status?                                      2. General Comments
  – Are there any errors in the analysis?            – Focus on errors in understanding or
                                                        interpretation of information previously
• Project Challenges                                    provided to GAO as well as points of
  – GAO’s challenge categories are fixed and            disagreement with the GAO’s analysis.
    will not change
                                                     – These comments will not appear in the
  – Assertion of a challenge should be                  report but may help drive the discussion with
    supported by the analysis described in the          the GAO about how they can revise
    Project Update
                                                 3. Technical Comments
• Contract Info, Project Performance,                – Comments provided in this section will be
  and Project Timeline                                  used to correct any factual errors in the
  – no action from PM required; Procurement             report.
    and OCFO will review to ensure consistency       – Corrections must be accompanied by
    with other external reporting
                                                        evidence

GAO will transmit draft summaries directly to project managers with response requested in
5 business days; Response must go through full agency review process in this time.
  1/22/2012                                                                                     19
So we gave GAO all this data…

   WHAT DID THE DATA SAY?


1/22/2012                          20
GAO Quicklook – Mar 2011
•     Maintained recommendation for GAO’s “Business Case Approach,” but recognized
      improvements due to JCL policy
•     Found average cost growth of 14.6% and schedule growth of 8 mo; nearly all growth
      attributed to projects baselined before 2009
•     Main Observations:
        – Technology Challenges:
            • NASA begins development with insufficient technology maturity
            • Evidence: Majority of projects passed PDR with critical technologies at TRL < 6
        – Design Challenges:
            • Projects pass CDR with unstable design
            • Evidence: Nearly all projects passed CDR with <90% releasable drawings
•     Additional Observations
        – Funding Challenges: JWST, Ares, Orion funding “did not align with plan”
        – Launch Vehicle Challenges: mainly medium class – addressed more fully in another GAO
          report
        – Development Partner Challenges
        – Parts Challenges – addressed more fully in GAO Quality Parts Audit
        – Contractor Management Challenges
•     Report stated that GAO will conduct a more thorough analysis of NASA’s EVM data in
      future audits and that [cost] transparency is insufficient during formulation
    1/22/2012                                                                                   21
Mar 2011 QLB - Cost Growth
                                                                      The majority of
                                                                      NASA’s cost and
                                                                      schedule growth is
                                                                      attributed to
                                                                      projects baselined
                                                                      prior to 2009




            NASA agrees with GAO’s cost and schedule growth figures
1/22/2012                                                                           22
Challenges Summary
                                                            Quicklooks published in 2009, 2010, 2011
                                        12
                                                                                                                                            2009
                                                                                                                                            2010
                                        10
 # Projects with GAO-identified Issue




                                                                                                                                            2011

                                        8


                                        6


                                        4


                                        2


                                        0
                                             Funding Issues Launch Issues   Parts Issues      Technology   Design Issues   Contractor   Development
                                                                                                Issues                       Issues     Partner Issues

                                                                                           Subjects of other GAO audits
1/22/2012                                                                                                                                            23
Data Provided During 2011
                              Technology Maturity and Drawing Release
                     150.0%                                                                                 150.0%
Percentage Development Cost




                     130.0% JWST                                                                            130.0%
                                                                                                                                                 JWST




                                                                                   Percentage cost growth
                     110.0%                                                                                 110.0%
                                                                    Class A                                               GAO hypothesis
                      90.0%                                                                                  90.0%
          Growth




                      70.0%                        MSL              Class B                                  70.0%

                      50.0%                                         Class C                                  50.0%                         NPP
                      30.0%              NPP                                                                 30.0%                 SOFIA
                                                                    Class D
                      10.0%                                                                                  10.0%
                                                                                                                            Juno
                     -10.0%                                                                                 -10.0%
                                                                                                                  0.00% 20.00% 40.00% 60.00% 80.00% 100.00%120.00%
                            0%           50%            100%                                                                    GPM
                     -30.0%                                                                                 -30.0%
                             % Immature Technologies at PDR                                                             % releaseable drawings at CDR

MSL did not provide drawing data; MAVEN is excluded from CDR analysis because CDR was not complete at time of DCI submission



• Self-reported data from DCIs submitted to GAO appear to support the hypothesis that
  immature technologies at PDR leads to cost growth for NASA missions
                        – The distribution does not support a strong correlation
                        – JWST and MSL drive the correlation, but have cost growth clearly attributable to other factors.
• The data shows low correlation between releasable drawings at CDR and cost growth

            1/22/2012                                                                                                                                       24
Data Provided During 2011
                                    Drawing Release
                                          % of Drawings Releasable at CDR
100.00%
 90.00%
 80.00%
 70.00%
 60.00%
 50.00%
 40.00%
 30.00%
 20.00%
 10.00%
  0.00%
          Aquarius   GPM   GRAIL   Juno     JWST    LADEE     LDCM     MMS       MSL      NPP   OCO-2   RBSP   SOFIA   TDRS

                                             GAO Calculation Assumed   NASA calculation



                           CDR     Now        • There appears to be some disagreement between how NASA
# Releasable                 A       C          and GAO calculate the percent of releasable drawings
                                              • NASA calculates the percentage of releasable drawings at CDR
Total # anticipated          B       D
                                                based upon the total number of drawings anticipated as of CDR
NASA: % releasable = A/B                      • Drawing growth after CDR typically occurs as a result of risk
GAO: % releasable = A/D                         realization; it is impossible to predict which risks will come to
                                                pass and which of those will require drawing changes
NASA’s 90% guidance is based                  • By NASA’s calculation, several projects did meet the best
on what is known at the time                    practice guideline of 90% releasable drawings at CDR
  1/22/2012                                                                                                             25
Data Provided During 2011
                                                                        Software
• 13 projects provided complete software data
                   – 4 projects - Aquarius, MSL, NPP, and JWST - reported growth in TLOC greater than 25%
                   – 5 projects reported a decrease in TLOC since PDR – Juno, LDCM, RBSP, SOFIA, TDRS
• After removing projects whose cost growth is clearly attributed to other
  factors, the mathematical correlation between development cost growth in
  growth in TLOC is high (.91) but the lack of significant distribution does not
  support a relationship between cost growth and software growth

                                         All Quicklook Projects                                                MSL and Aquarius Excluded
                               200.0%                                                                         200.0%
 % Development Cost Growth




                                                                                 % Development Cost Growth
                               150.0%                                                                         150.0%
                                                   JWST                                                                                             JWST

                               100.0%                                                                         100.0%
                                                   MSL
                                50.0%                                                                           SOFIA
                                                                                                               50.0%
                                             NPP                                                             LDCM                    NPP
                              SOFIA                                Aquarius
                                 0.0%       GPM                                                                 0.0%
            -200%                        0%    200%       400%   600%     800%         -50%                            0%      50%         100%       150%
                                                                                                                         GPM
                 -50.0%                                                                                       -50.0%
                                     % Growth in TLOC, PDR to Current                                            % Growth in TLOC, PDR to Current

                             1/22/2012                                                                                                                26
What is Headquarters saying to GAO?

   AGENCY MESSAGES


1/22/2012                                27
Formulation
                                      • Formulation is an iterative
    Resources         Requirements      process that aims to balance
                                        requirements and resources.
                                      • At KDP-C the Decision
                                        Authority agrees that the
                                        design is sufficiently mature
                                        and the requirements match
                                        the resources
                                      • The confirmation decision
                                        results in an agreed-to cost
                                        and schedule baseline; budget
                                        is committed to the project as
                                        part of the PPBE process
                                      • The balance of requirements
                                        and resources is continuously
                                        assessed throughout a
                                        project’s lifecycle
Changes in schedule or projected LCC during formulation should
not be treated as slips or cost growth; rather, this is a normal part of
the formulation process.
 1/22/2012                                                            28
PDR                  Road to Confirmation
                                                            (For SMD Projects to date)
      Project Report                   Project Report                                                  Project Report
             JCL                               JCL                                                            JCL
      SRB C&S (JCL)                     SRB C&S (JCL)                                                  SRB C&S (JCL)
       Assessment                        Assessment                                                     Assessment
      SRB Technical                     SRB Technical                                                  SRB Technical
       Assessment                        Assessment                                                     Assessment
    Project Response                  Project Response                  Approval                     Project Response
                                                                                                                                        Approval
         to SRB                            to SRB                      to Proceed                         to SRB                       to Proceed
       ICEs or ICAs1                     ICEs or ICAs1
                                                                       (Cat 2&3)2                       ICEs or ICAs1
                                                                                                                                         (Cat 1)
                                         Center                                                          Center
                                     Recommendation                                                  Recommendation
                                      Program Office                                                      SMD
                                     Recommendation                                                  Recommendation
            CMC                      Division (Theme)                     DPMC                       OCFO/SID Analyst                      APMC
                                     Recommendation                                                    Assessment


           The Confirmation decision takes into account many inputs.
                          There is no “right answer”
1. when available
     1/22/2012                                                                                                                                    29
2. Cat 2 and 3 missions do not require an APMC, but Cat 1 missions do. With 7120.5E, the AA will approve entry to Implementation for all projects
JCLs and UFE
• The use of JCL in decision making represents a shift in the paradigm
  of how project cost estimates, budgets and schedules are
  established.
• This method of estimation moves beyond prior practices of
  calculating a “point estimate” and adding an additional level of
  reserves.
• Because the UFE is a product of the probabilistic basis of
  estimates, any reduction to the UFE will reduce the probability of
  achieving the project cost and schedule targets
• Usage of UFE should not be interpreted as a “funding issue;” the
  UFE is part of the full amount that NASA budgets for a mission.
• The term “UFE” should be used rather than “reserves” to denote
  the more rigorous cost estimating policy NASA now uses
   With the release of 7120.5E, NASA no longer uses the term
    “reserves.” PMs should update charts to use the term
                       “Project-held UFE.”
1/22/2012                                                            30
Management vs Agency Commitments
                   Cost                                     Schedule
Agency Baseline                                                Agency Baseline
Commitment                                       Agency        Commitment
                               MD-Held
                                                Schedule       • Based upon JCL and
• Based upon JCL and            UFE              Margin          other inputs
  other inputs
                                Project-         Project       • At 70% JCL or as
• At 70% JCL or as
                                                Schedule          determined by the DA
   determined by the DA        held UFE
                                                 Margin        • Difference between ABC
• Difference between ABC                                          and MA is additional
   and MA is budgeted as                                          schedule margin held by
   MD-held UFE                                                    the Agency

Management
                                                               Management
Agreement
                                                               Agreement
• Managed by PM
                                                                Managed by PM
• Based upon Project’s
                                                                Based upon Schedule
  own Cost Estimate
                                                                 developed by PM
• Should be at 50% JCL or
                                                                Should be at 50% JCL or
  greater                                                        greater
                                      Not to scale
In some cases, the Decision Authority deems it prudent for the Agency to hold
additional schedule margin above the project’s schedule. In these cases, the
Management Schedule should not be viewed as acceleration of the Agency’s
schedule
   1/22/2012                                                                        31
What happens when funds are not
             appropriated or performance is poor?
o When requested funds                          Notional Example
  are not
  appropriated, or          700
  performance is                   FY10 Budget request
  poor, work is pushed      650    FY10 Appropriation + FY11 request
  to future years, and             FY11 Appropriation + FY12 request
  budget requirements
                                   FY12 Appropriation + FY13 request
  increase in the out       600
  years.
                            550
o This may lead to cost
  growth and operating
  plan changes.             500


o If reductions happen in   450
  subsequent years, the
  profile becomes
  unsustainable.            400
                                  FY10         FY11          FY12      FY13   FY14


 1/22/2012                                                                           32
EVM
                 Total Project (Combined In- and Out- of House)
                                          Out-of-House                       NASA projects tend to be complex;
                                     Contracts
IIn-House
                 CP & FPI    CP & FPI      CP & FPI               Partners    EVM is not necessarily applied to
                                                         FFP
                 >$50M       >$20M         <$20M                             all suppliers but it is addressed at
    7            Certified      32
Principles        System     Guidelines
                                             NA           NA        NA             the total project level.
      7 Principles applied across all elements of Total Project

  • NASA’s projects are accomplished through combined efforts of suppliers and
    NASA in-house efforts
             –       Total In-House with targeted procurements
             –       Prime contract with government providing Level-Of-Effort (LOE) oversight
             –       Many unique mixtures of the above two scenarios
  • NASA applies EVM to contractors at the project level and requirements are
    flowed down to subcontractors
EVM is one of many project management tools that NASA uses. NASA
recognizes EVM implementation as an area of concern and is committed to
improvement in this area
     1/22/2012                                                                                              33
Technical Leading Indicators
• NASA is enhancing the ability to monitor and assess the
  stability of programs and projects as well as the maturity of
  their products through the development process
• Comprehensive 3-Step Approach
     1. The Entrance and Success Criteria for Lifecycle reviews are being
        enhanced to further account for product maturity
            •   These will be documented in the Rev B update to NPR 7123.1
     2. A required set of 3 parameters that will address product stability
        and maturity will be added as a NID to NPR 7123.1A
            • Mass Margins
            • Power Margins
            • Review item closures
     3. A set of recommended Common Leading Indicators is being
        developed
            • Added to NPR 7120.5E Formulation Agreement and Program/Project
              Plan templates

1/22/2012                                                                      34
Any Questions?




1/22/2012                    35

Mais conteúdo relacionado

Destaque

Gonzales.matthew
Gonzales.matthewGonzales.matthew
Gonzales.matthewNASAPMC
 
Lockwood.scott
Lockwood.scottLockwood.scott
Lockwood.scottNASAPMC
 
Bizier.brenda
Bizier.brendaBizier.brenda
Bizier.brendaNASAPMC
 
Matt.leonard
Matt.leonardMatt.leonard
Matt.leonardNASAPMC
 
Sally.godfrey
Sally.godfrey Sally.godfrey
Sally.godfrey NASAPMC
 
Fletcher risk vs_innovation_120220
Fletcher risk vs_innovation_120220Fletcher risk vs_innovation_120220
Fletcher risk vs_innovation_120220NASAPMC
 
Leon.j.douglas.d
Leon.j.douglas.dLeon.j.douglas.d
Leon.j.douglas.dNASAPMC
 
Zimmerman barbier
Zimmerman barbierZimmerman barbier
Zimmerman barbierNASAPMC
 
Gwyn.smith
Gwyn.smithGwyn.smith
Gwyn.smithNASAPMC
 
Backup hesenperger kottmeyer
Backup hesenperger kottmeyerBackup hesenperger kottmeyer
Backup hesenperger kottmeyerNASAPMC
 
Elliott.k.liang.a.sauser.b
Elliott.k.liang.a.sauser.bElliott.k.liang.a.sauser.b
Elliott.k.liang.a.sauser.bNASAPMC
 
Grammier.richard
Grammier.richardGrammier.richard
Grammier.richardNASAPMC
 
Lisa mc cann
Lisa mc cannLisa mc cann
Lisa mc cannNASAPMC
 
Goldstein.barry
Goldstein.barryGoldstein.barry
Goldstein.barryNASAPMC
 
Pollitt.julie
Pollitt.juliePollitt.julie
Pollitt.julieNASAPMC
 
Glenn.perez
Glenn.perezGlenn.perez
Glenn.perezNASAPMC
 
Cook steve
Cook steveCook steve
Cook steveNASAPMC
 
Adesh.jain
Adesh.jainAdesh.jain
Adesh.jainNASAPMC
 
Fred.ouellette
Fred.ouelletteFred.ouellette
Fred.ouelletteNASAPMC
 
Vipavetz.kevin
Vipavetz.kevinVipavetz.kevin
Vipavetz.kevinNASAPMC
 

Destaque (20)

Gonzales.matthew
Gonzales.matthewGonzales.matthew
Gonzales.matthew
 
Lockwood.scott
Lockwood.scottLockwood.scott
Lockwood.scott
 
Bizier.brenda
Bizier.brendaBizier.brenda
Bizier.brenda
 
Matt.leonard
Matt.leonardMatt.leonard
Matt.leonard
 
Sally.godfrey
Sally.godfrey Sally.godfrey
Sally.godfrey
 
Fletcher risk vs_innovation_120220
Fletcher risk vs_innovation_120220Fletcher risk vs_innovation_120220
Fletcher risk vs_innovation_120220
 
Leon.j.douglas.d
Leon.j.douglas.dLeon.j.douglas.d
Leon.j.douglas.d
 
Zimmerman barbier
Zimmerman barbierZimmerman barbier
Zimmerman barbier
 
Gwyn.smith
Gwyn.smithGwyn.smith
Gwyn.smith
 
Backup hesenperger kottmeyer
Backup hesenperger kottmeyerBackup hesenperger kottmeyer
Backup hesenperger kottmeyer
 
Elliott.k.liang.a.sauser.b
Elliott.k.liang.a.sauser.bElliott.k.liang.a.sauser.b
Elliott.k.liang.a.sauser.b
 
Grammier.richard
Grammier.richardGrammier.richard
Grammier.richard
 
Lisa mc cann
Lisa mc cannLisa mc cann
Lisa mc cann
 
Goldstein.barry
Goldstein.barryGoldstein.barry
Goldstein.barry
 
Pollitt.julie
Pollitt.juliePollitt.julie
Pollitt.julie
 
Glenn.perez
Glenn.perezGlenn.perez
Glenn.perez
 
Cook steve
Cook steveCook steve
Cook steve
 
Adesh.jain
Adesh.jainAdesh.jain
Adesh.jain
 
Fred.ouellette
Fred.ouelletteFred.ouellette
Fred.ouellette
 
Vipavetz.kevin
Vipavetz.kevinVipavetz.kevin
Vipavetz.kevin
 

Semelhante a K gallagher tosborne

Anglo American Preliminary Financial Results for 2011
Anglo American Preliminary Financial Results for 2011Anglo American Preliminary Financial Results for 2011
Anglo American Preliminary Financial Results for 2011Anglo American
 
Svarcas.rita
Svarcas.ritaSvarcas.rita
Svarcas.ritaNASAPMC
 
Business Architecture based Performance Transformation
Business Architecture based Performance TransformationBusiness Architecture based Performance Transformation
Business Architecture based Performance TransformationSteve Kerzman
 
NG BB 07 Multi-Generation Project Planning
NG BB 07 Multi-Generation Project PlanningNG BB 07 Multi-Generation Project Planning
NG BB 07 Multi-Generation Project PlanningLeanleaders.org
 
六合彩,香港六合彩
六合彩,香港六合彩六合彩,香港六合彩
六合彩,香港六合彩pcdylwd
 
Understanding challenges of developing transportation PPPs in the Middle East
Understanding challenges of developing transportation PPPs in the Middle EastUnderstanding challenges of developing transportation PPPs in the Middle East
Understanding challenges of developing transportation PPPs in the Middle EastDr Rupert Booth
 
ON TIME | ON TARGET: Leasing for Army Transportation Battalion
ON TIME | ON TARGET: Leasing for Army Transportation BattalionON TIME | ON TARGET: Leasing for Army Transportation Battalion
ON TIME | ON TARGET: Leasing for Army Transportation BattalionCurt Cooper
 
GeoKettle, GeoMondrian et Spatialytics : une suite open source de GeoBI
GeoKettle, GeoMondrian et Spatialytics : une suite open source de GeoBIGeoKettle, GeoMondrian et Spatialytics : une suite open source de GeoBI
GeoKettle, GeoMondrian et Spatialytics : une suite open source de GeoBIACSG Section Montréal
 
Project Control Basics
Project Control BasicsProject Control Basics
Project Control Basicsprojectcompany
 
ROSCo2019 MoveIt Workshop Recap
ROSCo2019 MoveIt Workshop RecapROSCo2019 MoveIt Workshop Recap
ROSCo2019 MoveIt Workshop RecapTomoya Fujita
 
CCS Projects Integration Workshop - London 3Nov11 - ROAD - CCS project integr...
CCS Projects Integration Workshop - London 3Nov11 - ROAD - CCS project integr...CCS Projects Integration Workshop - London 3Nov11 - ROAD - CCS project integr...
CCS Projects Integration Workshop - London 3Nov11 - ROAD - CCS project integr...Global CCS Institute
 
Empirical Evidence Of Agile Methods
Empirical Evidence Of Agile MethodsEmpirical Evidence Of Agile Methods
Empirical Evidence Of Agile MethodsGrigori Melnik
 
Sip Iocl 0921411
Sip Iocl 0921411Sip Iocl 0921411
Sip Iocl 0921411Parulmidha
 
Mary.zimmerman
Mary.zimmermanMary.zimmerman
Mary.zimmermanNASAPMC
 

Semelhante a K gallagher tosborne (20)

Anglo American Preliminary Financial Results for 2011
Anglo American Preliminary Financial Results for 2011Anglo American Preliminary Financial Results for 2011
Anglo American Preliminary Financial Results for 2011
 
Svarcas.rita
Svarcas.ritaSvarcas.rita
Svarcas.rita
 
TEX110308
TEX110308TEX110308
TEX110308
 
Business Architecture based Performance Transformation
Business Architecture based Performance TransformationBusiness Architecture based Performance Transformation
Business Architecture based Performance Transformation
 
TEX110308
TEX110308TEX110308
TEX110308
 
NG BB 07 Multi-Generation Project Planning
NG BB 07 Multi-Generation Project PlanningNG BB 07 Multi-Generation Project Planning
NG BB 07 Multi-Generation Project Planning
 
六合彩,香港六合彩
六合彩,香港六合彩六合彩,香港六合彩
六合彩,香港六合彩
 
Scope creep - cylfe campain
Scope creep - cylfe campainScope creep - cylfe campain
Scope creep - cylfe campain
 
Understanding challenges of developing transportation PPPs in the Middle East
Understanding challenges of developing transportation PPPs in the Middle EastUnderstanding challenges of developing transportation PPPs in the Middle East
Understanding challenges of developing transportation PPPs in the Middle East
 
ON TIME | ON TARGET: Leasing for Army Transportation Battalion
ON TIME | ON TARGET: Leasing for Army Transportation BattalionON TIME | ON TARGET: Leasing for Army Transportation Battalion
ON TIME | ON TARGET: Leasing for Army Transportation Battalion
 
D07 Project Charter
D07 Project CharterD07 Project Charter
D07 Project Charter
 
Premier o il
Premier o ilPremier o il
Premier o il
 
GeoKettle, GeoMondrian et Spatialytics : une suite open source de GeoBI
GeoKettle, GeoMondrian et Spatialytics : une suite open source de GeoBIGeoKettle, GeoMondrian et Spatialytics : une suite open source de GeoBI
GeoKettle, GeoMondrian et Spatialytics : une suite open source de GeoBI
 
Project Control Basics
Project Control BasicsProject Control Basics
Project Control Basics
 
ROSCo2019 MoveIt Workshop Recap
ROSCo2019 MoveIt Workshop RecapROSCo2019 MoveIt Workshop Recap
ROSCo2019 MoveIt Workshop Recap
 
CCS Projects Integration Workshop - London 3Nov11 - ROAD - CCS project integr...
CCS Projects Integration Workshop - London 3Nov11 - ROAD - CCS project integr...CCS Projects Integration Workshop - London 3Nov11 - ROAD - CCS project integr...
CCS Projects Integration Workshop - London 3Nov11 - ROAD - CCS project integr...
 
Empirical Evidence Of Agile Methods
Empirical Evidence Of Agile MethodsEmpirical Evidence Of Agile Methods
Empirical Evidence Of Agile Methods
 
Sip Iocl 0921411
Sip Iocl 0921411Sip Iocl 0921411
Sip Iocl 0921411
 
Managing benefits
Managing benefitsManaging benefits
Managing benefits
 
Mary.zimmerman
Mary.zimmermanMary.zimmerman
Mary.zimmerman
 

Mais de NASAPMC

Bejmuk bo
Bejmuk boBejmuk bo
Bejmuk boNASAPMC
 
Baniszewski john
Baniszewski johnBaniszewski john
Baniszewski johnNASAPMC
 
Yew manson
Yew mansonYew manson
Yew mansonNASAPMC
 
Wood frank
Wood frankWood frank
Wood frankNASAPMC
 
Wood frank
Wood frankWood frank
Wood frankNASAPMC
 
Wessen randi (cd)
Wessen randi (cd)Wessen randi (cd)
Wessen randi (cd)NASAPMC
 
Vellinga joe
Vellinga joeVellinga joe
Vellinga joeNASAPMC
 
Trahan stuart
Trahan stuartTrahan stuart
Trahan stuartNASAPMC
 
Stock gahm
Stock gahmStock gahm
Stock gahmNASAPMC
 
Snow lee
Snow leeSnow lee
Snow leeNASAPMC
 
Smalley sandra
Smalley sandraSmalley sandra
Smalley sandraNASAPMC
 
Seftas krage
Seftas krageSeftas krage
Seftas krageNASAPMC
 
Sampietro marco
Sampietro marcoSampietro marco
Sampietro marcoNASAPMC
 
Rudolphi mike
Rudolphi mikeRudolphi mike
Rudolphi mikeNASAPMC
 
Roberts karlene
Roberts karleneRoberts karlene
Roberts karleneNASAPMC
 
Rackley mike
Rackley mikeRackley mike
Rackley mikeNASAPMC
 
Paradis william
Paradis williamParadis william
Paradis williamNASAPMC
 
Osterkamp jeff
Osterkamp jeffOsterkamp jeff
Osterkamp jeffNASAPMC
 
O'keefe william
O'keefe williamO'keefe william
O'keefe williamNASAPMC
 
Muller ralf
Muller ralfMuller ralf
Muller ralfNASAPMC
 

Mais de NASAPMC (20)

Bejmuk bo
Bejmuk boBejmuk bo
Bejmuk bo
 
Baniszewski john
Baniszewski johnBaniszewski john
Baniszewski john
 
Yew manson
Yew mansonYew manson
Yew manson
 
Wood frank
Wood frankWood frank
Wood frank
 
Wood frank
Wood frankWood frank
Wood frank
 
Wessen randi (cd)
Wessen randi (cd)Wessen randi (cd)
Wessen randi (cd)
 
Vellinga joe
Vellinga joeVellinga joe
Vellinga joe
 
Trahan stuart
Trahan stuartTrahan stuart
Trahan stuart
 
Stock gahm
Stock gahmStock gahm
Stock gahm
 
Snow lee
Snow leeSnow lee
Snow lee
 
Smalley sandra
Smalley sandraSmalley sandra
Smalley sandra
 
Seftas krage
Seftas krageSeftas krage
Seftas krage
 
Sampietro marco
Sampietro marcoSampietro marco
Sampietro marco
 
Rudolphi mike
Rudolphi mikeRudolphi mike
Rudolphi mike
 
Roberts karlene
Roberts karleneRoberts karlene
Roberts karlene
 
Rackley mike
Rackley mikeRackley mike
Rackley mike
 
Paradis william
Paradis williamParadis william
Paradis william
 
Osterkamp jeff
Osterkamp jeffOsterkamp jeff
Osterkamp jeff
 
O'keefe william
O'keefe williamO'keefe william
O'keefe william
 
Muller ralf
Muller ralfMuller ralf
Muller ralf
 

Último

COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 

Último (20)

COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 

K gallagher tosborne

  • 1. Overview of GAO Quicklook PM Challenge February 2012 Katie Gallagher and Tracy Osborne OCFO/SID
  • 2. Outline • Background – Why is this important? • GAO’s Approach – What projects are involved? – Why does GAO request technical data? • What should I expect if my project is in Quicklook? • What does NASA’s data say? • Agency Messages 1/22/2012 2
  • 3. Background • The GAO’s Assessment of Large-Scale [NASA] Projects is Congressionally-mandated • GAO sends its findings to Congress in an annual “QuickLook Book” (QLB) Report focusing on changes in project cost and schedule and providing GAO’s explanations for these changes • GAO is frequently called upon to testify about the results. • Engagement objective: “Identify and gauge the progress and potential risks associated with selected NASA acquisitions.” (i.e., projects) • GAO uses data from QuickLook Book (QLB) in other audits, such as the ongoing High Risk audit, and vice versa. 1/22/2012 3
  • 4. Why the Stakeholder Interest? • NASA’s historical inability to control cost and schedule growth – JWST, CxP, MSL growth cited in several provisions establishing new reports. – A 2004 GAO finding that “NASA lacks discipline in cost estimation.” – Multiple Operating Plans per year to address cost growth or phasing. • Based on past Performance NASA is required to report to OMB, Congress and GAO routinely on Project cost and schedule. – All appropriations since FY 2008 have included direction for GAO to “Identify and gauge the progress and potential risks associated with selected NASA acquisitions.” 1/22/2012 4
  • 5. Typical Results from Quicklook Reports • Most of NASA’s cost growth for the current portfolio is attributed to projects baselined prior to 2009 • NASA projects do not meet GAO-identified best practice standards for technology maturity and design stability • Other challenges that affect NASA project outcomes include Funding, Launch Vehicle, Development Partners, Parts Quality, Contractor Management 1/22/2012 5
  • 6. Why do they ask about TRL’s and drawings? GAO’S APPROACH 1/22/2012 6
  • 7. GAO’s Report is in Two Parts • Agency Observations – Examines adherence to NASA policies such as 7120.5E and 1000.5A – Examines potential systemic reasons for NASA’s cost growth – Assesses NASA’s efforts to improve cost and schedule growth, including JCL policy • Project Assessments – Provides an update on status of each project included in the audit – Outlines challenges specific to each project that GAO believes may lead to cost growth GAO Quicklook assesses performance of NASA’s major projects and the Agency’s management of those projects. 1/22/2012 7
  • 8. How does a project get picked? • GAO evaluates all spaceflight projects in formulation if a contract is released that has development content and is greater than $25 M – Can include projects in Phase A and Pre-Phase A if they have such a contract • GAO evaluates all spaceflight projects in development with a life cycle cost greater than $250 M • GAO continues to track projects in Operations until NASA negotiates removal from Quicklook – Minimal data collection on Operations projects; main focus is cost and scientific results • For projects in formulation, GAO expects NASA to provide cost and schedule ranges • For projects in development, GAO expects NASA to provide baseline cost, schedule, design and technical information 1/22/2012 8
  • 9. Projects Included in Audit 2008 2009 2010 2011 Requirement for Inclusion * Estimated Lifecycle Cost * Estimated Lifecycle Cost * Estimated Lifecycle Cost * Estimated Lifecycle Cost >$250M >$250M close to $250M (if it is called close to $250M (if it is called Lifecycle * In development * In development out in the budget as a out in the budget as a * In formulation if it also has * In formulation if it also has discreet project then it may discreet project then it may Phase as of released a development released a development be included) be included) Project Entrance contract valued at >$50M contract valued at >$25M * In development * In development Conference * If in operations, it is not * In formulation if it also has * In formulation if it also has (April 2011) (consistent with other included released a development released a development external reporting contract valued at >$50M contract valued at >$50M requirements) * In operations, in the prime * In operations, in the prime mission phase mission phase 1 Aquarius Development X X X X 2 Ares Formulation X X X 3 Dawn Operations X X X 4 Fermi/GLAST Operations X X X 5 GPM Development X X X X 6 Glory Development X X X X 7 GRAIL Development X X X 8 Herschel Operations X X X X 9 ICESat II Formulation X X 10 JWST Development X X X X 11 Juno Development X X X 12 Kepler Operations X X X X 13 LADEE Development X X 14 LDCM Development X X X X 15 LRO Operations X X X X 16 MAVEN Development X X 17 Mars 2016 (TGO) Formulation X 18 MMS Development X X X 19 MPCV Pre-Formulation X 20 MSL Development X X X X 21 NPP Development X X X X 22 OCO/OCO-2 Development X X X 23 Orion Formulation X X X 24 Phoenix Closed Out X 25 RBSP Development X X X 26 SDO Operations X X X X 27 SLS Pre-Formulation X 28 SMAP Formulation X X 29 SOFIA Development X X X X 30 Solar Probe Plus Formulation X X 31 TDRS Development X X 32 Wise Operations X X X X 1/22/2012 9
  • 10. GAO “Business Case” Model PDR CDR SIR Production Pre- Phase Phase Rarely Applies NASA phase A A B Phase C Phase D Phase E to NASA KDP-B KDP-C KDP-D Projects KDP-A KDP-D DOD Technology Development Engineering Manufacturing and Development Production and Deployment Operations and Support PDR MS-B CDR MS-C Concept and Technology Engineering Manufacturing and GAO Development Development Production Knowledge Knowledge Knowledge Point 1 Point 2 Point 3 • Knowledge Point 1: Resources and Requirements Match – Indicated by a high level of technology maturity • Heritage is technology is appropriate in form, fit and function • New Technologies at least at TRL-6 (for NASA) • Knowledge Point 2: Product Design is stable “about midway through development” – Best practice is to achieve design stability by midpoint of development – GAO cites 90% of engineering drawings releasable as figure of merit • Knowledge Point 3: Manufacturing Processes are mature – Product shown to be producible within cost, schedule, and quality targets and have demonstrated reliability and Design demonstrates needed performance through realistic system-level testing 1/22/2012 10
  • 11. Data Requested Annually Reporting Year Data Category Element 2008 2009 2010 2011 Frequency Design Stability X X X X Annual + Updates Critcal Technologies X X X X Annual + Updates Technical Heritage Technologies X X X Annual + Updates Software Complexity X X Annual Quality Parts Issues X X Annual MPAR Baseline X X X X As Occurs ARRA Funding X X Semi-annual KDP-B Est. LCC Range X X X X Semi-annual KDP-C Baseline X X As Occurs Cost ICEs X X As Occurs JCLs X X As Occurs Project-held UFE X X Monthly (in MSRs) MD-held UFE by year X Annual Key Milestones X X X X Semi-annual Schedule Agy vs Mgmt Schedule Comparison X Semi-annual Basic Information X X X X Semi-annual Contracts Award Fee Strucuture X X X Semi-annual EVM (CPR Formats) X X Monthly FAD/PCA X X X X As Occurs Project Plan X X X X As Occurs Control Plans X X X X As Occurs PDR/CDR Packages X X X X As Occurs Documentation CADRes X X As Occurs SRB Reports X X As Occurs KDP-C Decision Memos X As Occurs KDP-C Datasheets X As Occurs MSR Presentations X X Monthly 1/22/2012 GAO reviewed this list for 2011, and agreed to reduce frequency of some requests 11
  • 12. So, my project is in Quicklook… WHAT SHOULD I EXPECT? 1/22/2012 12
  • 13. The Quicklook Annual Cycle (2011) DCI- Action Item Response Addt’l Actions contract Site Visits Agency 2-Pgr Cmts DCI- Site Visits – Cmts – MSFC C&s JPL &GSFC &JSC DCI- DCI- DCI- Other Data Requests Addt’l Data proj C&s C&s Entrance Exit Final Report Conference Conference DCI = “Data Collection Instrument” Three types: 1.Project DCI – for technical data 2.Cost and Schedule (C&S) DCI – provided by OCFO 3.Contracts DCI – provided by OP 1/22/2012 13
  • 14. Process for Working with GAO Extra explanation may be provided Quicklook Technical POC (TPOC) to help contextualize the data or to discusses request with GAO to better assist the GAO in answering understand purpose for their questions requested items GAO transmits Project drafts Response reviewed QLB TPOC request response by HQ & Ctr Mgmt transmits response to GAO • The TPOC and • Provide • Possible • ALR, PM, PE, relevant Audit document reviewers and reviewers Liaisons (ALR) and/or include PE, MD, will be copied should be explanation PCFO, copied on all Procurement, • When requests OCE, OSMA, LSP document or • PM please data is not • QLB TPOC and forward available, MD ALR will requests to ALR explain why coordinate the and QLB TPOC appropriate • Draft response when necessary provided to PE, reviews. ALR, and QLB TPOC Standard 3 week turn-around when not otherwise stipulated by GAO 1/22/2012 14
  • 15. GAO Data Access Rights • Legislatively, the GAO has right of access to the data, regardless of procurement sensitivity or classification • NASA can negotiate with GAO regarding the provision of data; if there is a disagreement, GAO’s recourse is legislative action • GAO is bound by the same rules as NASA for handling the data, and NASA can specify data restrictions • NASA actively works with GAO on data formats, types and context for interpretation NASA provides the data requested and works with the GAO to add context and help the GAO analysts to understand NASA’s data. 1/22/2012 15
  • 16. Data Collection Instruments Example Technical DCI GAO Data Collection Instrument: Design Stability • GAO asks PMs to fill in Date Submitted to NASA Project Office: NASA Project Office Response Required by: 5/3/2011 5/6/2011 a Data Collection GAO Contact: Project: Joe Analyst Awesome Instrument (DCI) NASA Project Office Instructions Please provide or update the total number of releasable engineering drawings and the total number expected at key dates listed below. The term "releasable" refers to those drawings (physical hard copy or computer-generated packages) that are sent to be manufactured. Please annually and following calculate the number of drawings by using the number sheets within a package, not just the number of packages. Sometimes these drawings are considered the "build to" drawings. These drawings include details on the parts and work instructions needed to make the PDR and CDR product and reflect the results of testing. Change any dates or data as needed. Key Event Dates Defined: The PDR column should reflect information as of the Preliminary Design Review prior to KDP C (NAR). The CDR column should reflect information as of the • Data collection Critical Design Review. The "Latest" column should reflect the the number of drawings releasable and expected as of the date this spreadsheet is returned to GAO. includes Overall Design Knowledge KP ESTIMATE PDR CDR September 2010 update Latest – Critical and heritage Date No. of Releasable 7/12/2010 Scheduled for July 2011 9/16/2010 5/3/2011 technology TRLs 1537 sheets expected for 947 spacecraft sheets released to date. 299 – Drawing release – Parts Quality issues, spacecraft, 818 drawings 10 released for spacecraft, drawings released for Drawings 0 expected for instruments. 16 released for instruments. instruments. No. of Total 2437 sheets of spacecraft including impact ($) – Software TLOC and 2437 sheets expected for drawings planned, 1023 1879 sheets expected for spacecraft, 1023 drawings drawings expected for Drawings Expected spacecraft. expected for instruments. instruments. Source Project Office Project Office Project Office Project Office Blocks If the total number of drawings is not known or if the total number of drawings has changed, please provide an explanation in the space below. If the project has – Partners • completed CDR, please describe any measures and metrics used in addition to engineering drawings to assess design stability. If the project has yet to complete its CDR, please identify the measures and metrics the project intends to use to assess design stability. See below for Spacecraft and Instruments comments. The project assesses design stability primarily thru the Systems Engineering organization. A weekly Systems Design Team GAO requests data as regularly addresses design, design margins, analyses, interfaces, subsystem and instrument design topics, and proposed changes. Proposed design changes are then processed thru standard Configuration Control Boards. In addition, weekly status reviews as well as a Monthly Management Review take place that addresses overall status and issues. of PDR, CDR, and current Comments & Issues: Enter comments and issues below or on a separate sheet. • Procurement and PMs should NOT leave any sections blank; if the data is OCFO fill in similar DCIs on contracts, not available or not applicable, say so and explain why. cost, and schedule 1/22/2012 16
  • 17. Site Visits • GAO will prepare for site visits with the projects by reviewing project monthly status review (MSR or PSR) packages; discussion questions will be transmitted in advance of the visit. • GAO may not visit every project included in the Quicklook; projects not visited will receive written questions based on the content of their monthly review packages. • PMs are invited to bring any charts or personnel that they need to help answer the questions; however, GAO prefers to have an informal discussion – NOT a lecture. • When preparing responses, keep in mind that Quicklook is about cost growth against the Agency’s MPAR Baseline; responses should keep in mind the full mission cost – including launch vehicle and UFE • Refer questions about things you are not responsible for to correct organization – LV questions should generally be directed to LSP – MD-held UFE questions should be directed to the MD, although it is ok to state the value • When in doubt, it is ok to take an action to provide a response after the visit. This is the project manager’s opportunity to add context and explain all the good work their team is doing. 1/22/2012 17
  • 18. 2-Page Project Summary Anatomy Project Challenges Overview Project Timeline Project Update Project Essentials Contract Info Project Performance Summary Project Office Comments 1/22/2012 18
  • 19. Draft 2-Page Project Summaries What to look for How to Respond • Overview and Project Essentials 1. Project comments – Is the information correct? – Will be printed in the “Project Office • Project Update and Summary Comments” section of the final report – Limited to 500 characters (appr 100 words) – Does the GAO accurately portray the status? 2. General Comments – Are there any errors in the analysis? – Focus on errors in understanding or interpretation of information previously • Project Challenges provided to GAO as well as points of – GAO’s challenge categories are fixed and disagreement with the GAO’s analysis. will not change – These comments will not appear in the – Assertion of a challenge should be report but may help drive the discussion with supported by the analysis described in the the GAO about how they can revise Project Update 3. Technical Comments • Contract Info, Project Performance, – Comments provided in this section will be and Project Timeline used to correct any factual errors in the – no action from PM required; Procurement report. and OCFO will review to ensure consistency – Corrections must be accompanied by with other external reporting evidence GAO will transmit draft summaries directly to project managers with response requested in 5 business days; Response must go through full agency review process in this time. 1/22/2012 19
  • 20. So we gave GAO all this data… WHAT DID THE DATA SAY? 1/22/2012 20
  • 21. GAO Quicklook – Mar 2011 • Maintained recommendation for GAO’s “Business Case Approach,” but recognized improvements due to JCL policy • Found average cost growth of 14.6% and schedule growth of 8 mo; nearly all growth attributed to projects baselined before 2009 • Main Observations: – Technology Challenges: • NASA begins development with insufficient technology maturity • Evidence: Majority of projects passed PDR with critical technologies at TRL < 6 – Design Challenges: • Projects pass CDR with unstable design • Evidence: Nearly all projects passed CDR with <90% releasable drawings • Additional Observations – Funding Challenges: JWST, Ares, Orion funding “did not align with plan” – Launch Vehicle Challenges: mainly medium class – addressed more fully in another GAO report – Development Partner Challenges – Parts Challenges – addressed more fully in GAO Quality Parts Audit – Contractor Management Challenges • Report stated that GAO will conduct a more thorough analysis of NASA’s EVM data in future audits and that [cost] transparency is insufficient during formulation 1/22/2012 21
  • 22. Mar 2011 QLB - Cost Growth The majority of NASA’s cost and schedule growth is attributed to projects baselined prior to 2009 NASA agrees with GAO’s cost and schedule growth figures 1/22/2012 22
  • 23. Challenges Summary Quicklooks published in 2009, 2010, 2011 12 2009 2010 10 # Projects with GAO-identified Issue 2011 8 6 4 2 0 Funding Issues Launch Issues Parts Issues Technology Design Issues Contractor Development Issues Issues Partner Issues Subjects of other GAO audits 1/22/2012 23
  • 24. Data Provided During 2011 Technology Maturity and Drawing Release 150.0% 150.0% Percentage Development Cost 130.0% JWST 130.0% JWST Percentage cost growth 110.0% 110.0% Class A GAO hypothesis 90.0% 90.0% Growth 70.0% MSL Class B 70.0% 50.0% Class C 50.0% NPP 30.0% NPP 30.0% SOFIA Class D 10.0% 10.0% Juno -10.0% -10.0% 0.00% 20.00% 40.00% 60.00% 80.00% 100.00%120.00% 0% 50% 100% GPM -30.0% -30.0% % Immature Technologies at PDR % releaseable drawings at CDR MSL did not provide drawing data; MAVEN is excluded from CDR analysis because CDR was not complete at time of DCI submission • Self-reported data from DCIs submitted to GAO appear to support the hypothesis that immature technologies at PDR leads to cost growth for NASA missions – The distribution does not support a strong correlation – JWST and MSL drive the correlation, but have cost growth clearly attributable to other factors. • The data shows low correlation between releasable drawings at CDR and cost growth 1/22/2012 24
  • 25. Data Provided During 2011 Drawing Release % of Drawings Releasable at CDR 100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% Aquarius GPM GRAIL Juno JWST LADEE LDCM MMS MSL NPP OCO-2 RBSP SOFIA TDRS GAO Calculation Assumed NASA calculation CDR Now • There appears to be some disagreement between how NASA # Releasable A C and GAO calculate the percent of releasable drawings • NASA calculates the percentage of releasable drawings at CDR Total # anticipated B D based upon the total number of drawings anticipated as of CDR NASA: % releasable = A/B • Drawing growth after CDR typically occurs as a result of risk GAO: % releasable = A/D realization; it is impossible to predict which risks will come to pass and which of those will require drawing changes NASA’s 90% guidance is based • By NASA’s calculation, several projects did meet the best on what is known at the time practice guideline of 90% releasable drawings at CDR 1/22/2012 25
  • 26. Data Provided During 2011 Software • 13 projects provided complete software data – 4 projects - Aquarius, MSL, NPP, and JWST - reported growth in TLOC greater than 25% – 5 projects reported a decrease in TLOC since PDR – Juno, LDCM, RBSP, SOFIA, TDRS • After removing projects whose cost growth is clearly attributed to other factors, the mathematical correlation between development cost growth in growth in TLOC is high (.91) but the lack of significant distribution does not support a relationship between cost growth and software growth All Quicklook Projects MSL and Aquarius Excluded 200.0% 200.0% % Development Cost Growth % Development Cost Growth 150.0% 150.0% JWST JWST 100.0% 100.0% MSL 50.0% SOFIA 50.0% NPP LDCM NPP SOFIA Aquarius 0.0% GPM 0.0% -200% 0% 200% 400% 600% 800% -50% 0% 50% 100% 150% GPM -50.0% -50.0% % Growth in TLOC, PDR to Current % Growth in TLOC, PDR to Current 1/22/2012 26
  • 27. What is Headquarters saying to GAO? AGENCY MESSAGES 1/22/2012 27
  • 28. Formulation • Formulation is an iterative Resources Requirements process that aims to balance requirements and resources. • At KDP-C the Decision Authority agrees that the design is sufficiently mature and the requirements match the resources • The confirmation decision results in an agreed-to cost and schedule baseline; budget is committed to the project as part of the PPBE process • The balance of requirements and resources is continuously assessed throughout a project’s lifecycle Changes in schedule or projected LCC during formulation should not be treated as slips or cost growth; rather, this is a normal part of the formulation process. 1/22/2012 28
  • 29. PDR Road to Confirmation (For SMD Projects to date) Project Report Project Report Project Report JCL JCL JCL SRB C&S (JCL) SRB C&S (JCL) SRB C&S (JCL) Assessment Assessment Assessment SRB Technical SRB Technical SRB Technical Assessment Assessment Assessment Project Response Project Response Approval Project Response Approval to SRB to SRB to Proceed to SRB to Proceed ICEs or ICAs1 ICEs or ICAs1 (Cat 2&3)2 ICEs or ICAs1 (Cat 1) Center Center Recommendation Recommendation Program Office SMD Recommendation Recommendation CMC Division (Theme) DPMC OCFO/SID Analyst APMC Recommendation Assessment The Confirmation decision takes into account many inputs. There is no “right answer” 1. when available 1/22/2012 29 2. Cat 2 and 3 missions do not require an APMC, but Cat 1 missions do. With 7120.5E, the AA will approve entry to Implementation for all projects
  • 30. JCLs and UFE • The use of JCL in decision making represents a shift in the paradigm of how project cost estimates, budgets and schedules are established. • This method of estimation moves beyond prior practices of calculating a “point estimate” and adding an additional level of reserves. • Because the UFE is a product of the probabilistic basis of estimates, any reduction to the UFE will reduce the probability of achieving the project cost and schedule targets • Usage of UFE should not be interpreted as a “funding issue;” the UFE is part of the full amount that NASA budgets for a mission. • The term “UFE” should be used rather than “reserves” to denote the more rigorous cost estimating policy NASA now uses With the release of 7120.5E, NASA no longer uses the term “reserves.” PMs should update charts to use the term “Project-held UFE.” 1/22/2012 30
  • 31. Management vs Agency Commitments Cost Schedule Agency Baseline Agency Baseline Commitment Agency Commitment MD-Held Schedule • Based upon JCL and • Based upon JCL and UFE Margin other inputs other inputs Project- Project • At 70% JCL or as • At 70% JCL or as Schedule determined by the DA determined by the DA held UFE Margin • Difference between ABC • Difference between ABC and MA is additional and MA is budgeted as schedule margin held by MD-held UFE the Agency Management Management Agreement Agreement • Managed by PM  Managed by PM • Based upon Project’s  Based upon Schedule own Cost Estimate developed by PM • Should be at 50% JCL or  Should be at 50% JCL or greater greater Not to scale In some cases, the Decision Authority deems it prudent for the Agency to hold additional schedule margin above the project’s schedule. In these cases, the Management Schedule should not be viewed as acceleration of the Agency’s schedule 1/22/2012 31
  • 32. What happens when funds are not appropriated or performance is poor? o When requested funds Notional Example are not appropriated, or 700 performance is FY10 Budget request poor, work is pushed 650 FY10 Appropriation + FY11 request to future years, and FY11 Appropriation + FY12 request budget requirements FY12 Appropriation + FY13 request increase in the out 600 years. 550 o This may lead to cost growth and operating plan changes. 500 o If reductions happen in 450 subsequent years, the profile becomes unsustainable. 400 FY10 FY11 FY12 FY13 FY14 1/22/2012 32
  • 33. EVM Total Project (Combined In- and Out- of House) Out-of-House NASA projects tend to be complex; Contracts IIn-House CP & FPI CP & FPI CP & FPI Partners EVM is not necessarily applied to FFP >$50M >$20M <$20M all suppliers but it is addressed at 7 Certified 32 Principles System Guidelines NA NA NA the total project level. 7 Principles applied across all elements of Total Project • NASA’s projects are accomplished through combined efforts of suppliers and NASA in-house efforts – Total In-House with targeted procurements – Prime contract with government providing Level-Of-Effort (LOE) oversight – Many unique mixtures of the above two scenarios • NASA applies EVM to contractors at the project level and requirements are flowed down to subcontractors EVM is one of many project management tools that NASA uses. NASA recognizes EVM implementation as an area of concern and is committed to improvement in this area 1/22/2012 33
  • 34. Technical Leading Indicators • NASA is enhancing the ability to monitor and assess the stability of programs and projects as well as the maturity of their products through the development process • Comprehensive 3-Step Approach 1. The Entrance and Success Criteria for Lifecycle reviews are being enhanced to further account for product maturity • These will be documented in the Rev B update to NPR 7123.1 2. A required set of 3 parameters that will address product stability and maturity will be added as a NID to NPR 7123.1A • Mass Margins • Power Margins • Review item closures 3. A set of recommended Common Leading Indicators is being developed • Added to NPR 7120.5E Formulation Agreement and Program/Project Plan templates 1/22/2012 34

Notas do Editor

  1. Note: Funding Issues, Launch Issues, and Parts Issues were not addressed in the 2008 Quicklook Report. The 2008 Quicklook Report included issues for several projects that are now in Operations or are completed, and are not listed here (Dawn, Fermi, Herschel, Kepler, LRO, OCO, SDO, WISE).
  2. NASA requires EVM reports to be in compliance with ANSI/EIA-748. CPR is a DOD format.analytical products and risk watch list are provided in MSRs