1. National Aeronautics and Space Administration
Improving Project Success through
Technical Excellence
February 9 – 10, 2010
John D. Marinaro
Mark M. Kowaleski
http://nsc.nasa.gov/
1
Used with Permission

2. Agenda
• SMA Technical Excellence Program (25 min.)
– NSC Background
– Purpose of STEP
– How the STEP Curriculum was Developed
– STEP Curriculum Elements
– Extension to other Agency training needs and risk reduction
– STEP Level 1 Roll-out and Metrics
– STEP Levels 2 – 4 Planning
• System Safety Discipline Overview (20 min.)
– System Safety & Program/Project Interactions
– System Safety Competencies
– System Safety Lifecycle Products and Services
– Community of Practice
– System Safety State-of-the-Discipline
– System Safety and Program/Project Risk Reduction
• Q&A (10 min.)
• Back-up Materials
– SMA Discipline Curriculums
2

3. SMA Technical
Excellence Program
3

4. NASA SAFETY CENTER
Overview
NSC develops SMA personnel, processes, and tools needed for
the safe achievement of NASA’s strategic goals through:
Establishing a learning Benchmarking from the best
environment and bringing their best
practices into our workplace
Learning from our mishaps Ensuring that we are informed
risk-takers
Managing the routine risks of Preserving our resources for
the workplace effectively the NASA mission
http://nsc.nasa.gov/ 4 2/18/2010

5. NASA Safety Center
FOUR AREAS OF SUPPORT
http://nsc.nasa.gov/ 5 2/18/2010

6. THE PEOPLE OF THE NSC
NASA Safety Center
Alan H. Phillips Rafael Sanabria Dennie Gonia Sandra L. Hardy Charlene L. Anderson
Director Deputy Director Secretary Business Manager Administrative Officer
John Marinaro
Karen Meinert Cheevon (Mi-Mi) Lau Kenneth O’Connor
OCE Technical Chief, GRC NESC
Technical Knowledge Management Audits & Assessments Director Mishap Investigation
Liaison Excellence Director OSMA OCIO Liaison
Discipline Systems Manager Support Manager
Fellows
Kerry Remp Diane Mark Kowaleski Homayoon Carolyn Van Sue Otero Sallie Keith Dr. Jeffrey Dawson Michael Milbert Mark George
COTR Chapman Systems Safety Dezfuli Drei IRIS COTR Information Data Analysis & Analyst MIS Administrative Support
Curriculum Technical Systems Safety ST Enterprise Dissemination Trending Analyst
IRIS
Development Support/ Architecture
EX3
Discipline Manager
EDUCATE Dr. Natesan
Alphaport Fayssal Safie Art Lee Michael Dodson
Jambulingam Reliability & Institutional/ Facility Mishap Investigator
Reliability & Maintainability ST Operational Audit Manager Eastern Region (GRC, GSFC,
Maintainability Brian Follen HQ HQ, KSC, LaRC )
PBMACOTR Robert Cross
NASA Safety Analyst
PBMA
Training Center ARES
Ray Kacmar
Brian Hughitt
Quality Paul Boldon
Quality Kristie French
Engineering Institutional/
Engineering ST Mishap Investigator
Brian Jackson Programmatic Support
Auditor Audit Manager HQ Southern Region
(JSC, MFSC, SSC)
Vacant
Software Assurance Audit support
Martha
from the field
Wetherholt
centers, other HQs Dennis Morehouse
Software Brad Kaustinen
offices/SSCs Mishap Investigator
Assurance ST NCAS COTR
Western Region
Vacant NCAS (ARC, DFRC, JPL)
Operational Aviation Honeywell
Safety
Agency Range
Israel Greenfeld
Safety Efforts
IRIS Requirements Liaison
SARD/KSC Steve Lilley with MIBs/MIWG
MIB Technical Report
Reviews & Corrective
http://nsc.nasa.gov/ 6 Action Tracking 2/18/2010

7. Safety & Mission Assurance
Technical Excellence
• Within NASA, SMA professionals
provide a vital role in support of
NASA’s design and operations focus.
• SMA advises mission specialists on
matters related to risk, safety, and
mission success.
• SMA encompasses all Agency safety,
reliability, maintainability, and
quality engineering and assurance
activities.
7

8. STEP was Designed to be…
1. …a way to assist
the Agency
(specifically SMA)
as it transitions
from an
operational focus
back to a design
focus.
8

9. STEP was Designed to be…
2. …both a source of
knowledge and an
ongoing reference tool in
providing SMA
professionals with
structured, advanced
SMA educational
opportunities in support
of that design focus.
9

10. STEP Curriculum Development
Technical Discipline Teams developed a competency-
driven curriculum structure for each discipline that
included Education, Training & Lifelong Learning.
Job of SMA SMA Competency SMA Products
Professional Disciplines Wheel & Services
EDUCATION TRAINING LIFELONG LEARNING
10

11. STEP Curriculum
• The STEP Curriculum is delivered across four
levels with increasing specialization.
3 Intermediate 4 Advanced
2 Introductory
Discipline Discipline
1 Introduction Discipline
Concepts or Concepts or
to STEP Concepts
Specialty Specialty
[Doer]
[Team Leader] [Expert]
Generalized Specialized
11

12. A Typical STEP Program Commitment
Estimated Hours per STEP Level
STEP Training Elements Level 1 Level 2 Level 3 Level 4
Core Training 3 24 40 N/A
Discipline Training 18 100 137 137
Domain Training (by others) 4 40 TBD TBD
Total Course Hours 25 164 177 (+TBD) 137 (+TBD)
OJT N/A 200 400 400
Years of Experience N/A 2 4 8
Test and/or Peer Board N/A N/A Yes Yes
CEU’s N/A 16 24 32
12

13. STEP Training Elements
13

14. STEP’s Benefit to
NASA’s Programs/Projects
• High confidence in SMA’s competence and capability.
• High confidence in SMA’s products & services.
• Improved technical program/project performance.
• Risk Reduction:
– Increased success likelihood.
– Improved risk characterization.
– Improved risk-based decision-making.
• Serves as showcase for engineering excellence.
• Complements Agency training initiatives:
– Academy of Program/Project and Engineering Leadership (APPEL).
– System Engineering Leadership Development Program (SELDP).
– Senior Executive Service Career Development Program (SES CDP).
14

15. STEP Level 1
Curriculum
• One curriculum across all six disciplines
• Total of 25 contact hours
– Each course is one hour on average
• Topics covered:
– SMA Implementation
– Core
– Domain
– Disciplines
– Case Studies
• On-demand and self-paced
15

16. Level 1 Initiatives
• Agency-wide roll-out
• Online Courses
• Handbook
• Quick-start guide
• Course DVD
• Brochures
16

17. Level 1 Participation
Post Rollout
• Over 1,300 individuals have
been assigned or
registered for the curricula
• 361 graduates as of Nov 1
• 26% of all enrollees have
successfully completed
Level 1 – two months after
rollout!
• Feedback: nearly 70% rate
the program as “high
value”
17

18. STEP Levels 2 - 4
Curriculum Features
• Unlike Level 1’s unified curriculum, Levels 2 - 4 provide a unique curriculum
for each discipline
• Participants will choose a discipline and receive a Course of Study to guide
them through each level.
• Levels 2-4 allow for electives & discipline specialization
• The course of study for each discipline includes a variety of course types:
– Online/e-learning via SATERN
– Instructor-led workshops
– APPEL, NSTC and SARD courses
– Commercially-available courses
– Courses created by and for the NASA Safety Center
Levels 2 – 4 also encourage OJT, Enrichment Experiences, Career
Training & Continuing Education.
18

19. Competency Wheel
• The planning process
started with the
Competency Wheel
for each discipline,
from Level 1.
• The Competency
Wheel lists all the
competencies an SMA
professional needs to
perform.
– The inner ring lists the major
competencies.
– The outer ring provides a
further level of detail to each
of the major competencies.
19

20. Performance Objectives
For each competency, detailed objectives were written to describe exactly what
performance is expected at each level. (Typical performance objectives shown)
Level 2 Level 3 Level 4
• Recognize QE Quality • Perform QE Quality • Evaluate Quality
Management functions. Management functions. Management functions
• Recognize Quality • Apply Quality and Systems.
Management as one of Management principles
four (4) second tier to develop Quality
competencies of the Management Systems
Perform Quality (QMS).
Management & Planning
Competency.
• Recognize the six (6) third
tier competencies in
Quality Management.
20

21. SMA Work Products and Services
SMA Work Products and Services were identified for each
discipline. These are the things SMA Professionals
actually do on the job.
21

22. Course of Study
The Result:
• A complete curriculum
plan – a Course of Study –
was developed for each
discipline, by level
• The course of study
includes a list of
– Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning Activities
22

23. STEP – A Complete Curriculum
• Overall, STEP will be a
complete curriculum
tailored specifically for
the SMA Professional.
• STEP is specifically
designed to address the
technical needs of
NASA’s programs and
projects.
23

24. System Safety Discipline
OVERVIEW
24

25. Technical Excellence does pay off…
…and you do get what you pay for.
Coach 1st Class
US Airways Flight # 1549 25

26. SS Discipline – Where are we today?
• Most SS professional’s training is ad hoc training and professional
development (not many opportunities for SMA degrees at universities).
• We “learn as we go” and heavy emphasis on unstructured OJT.
• We do a lot of “reviewing” but not a lot of actually “doing” and this is
perceived as a negative thing.
• Don’t address new hires’ training needs.
• Don’t have mentoring for entry or mid-level employees.
• Training needs to apply to “real” work vs. perceived or ideal work that we
think we should be doing.
• Don’t have structured OJT with a clear understanding of what it is that is
expected to be learned.
• Training does not address softer SS discipline areas like resource
estimation, safety in contracts, and SS integration with system and
program lifecycle.
• Training is presently focused on qualitative methods and we would benefit
as a community from more analytical-based training.
26

27. What Do System Safety Engineers Do?
Fundamentally, NASA System Safety Engineers, at some point in
their careers, during any phase of the system life cycle, do any
combination of three basic activities:
1. Assess any situation with safety consequences.
2. Manage safety in acquisitions.
3. Manage, leverage, and interact with safety organizations and NASA
programs.
From this premise, all competencies and products and services are
derived.

28. System Safety’s Interactions
with Programs/Projects
System Safety Products (E.g., Hazard
SS products and
SS Engineer applies Analysis, FTA, PRA, CRM, etc.)
Framework
competencies to
influences
develop SS products
decisions
2
3
SS Engineer Decision Maker
Competencies define
1 how SS Engineers do SS products support milestones
their job at NASA 4 in the Program lifecycle Decisions
influence the
5 outcome of
Programs
System Safety
monitors lifecycle
Safety Performance
6

29. Overall System Safety Framework
Draft 0 of Overview of System
Safety – Not to be Distributed
Outside of SS Working Group

30. System Safety Student Body Estimates
System Safety STEP participants
Civil Servant SMA & SMA's Support Contractor
SMA - Civil SMA -
Center Servant Contractor Total
GRC 14 6 20
JSC 110 233 343
MSFC 32 35 67
LaRC* 10 20 30
DFRC 2 0 2
JPL n/a 10 10
ARC* 5 10 ?
SSC 4 10 14
GSFC 31 73 104
KSC 28 25 53
HQ/NSC* 10 2 12
* Estimated
Total 246 424 670
30

31. SS Competency Wheel
A SMA System Safety
professional needs to possess
these 34 competencies:
Major Competencies
Technical
• System Safety Mathematical Skills
• System Safety Analytical Methods
• System Safety Rationale
Managerial
• System Safety in Acquisition Management
• System Safety in Organizational Management
31

32. Competency Performance Objectives
and Training, OJT, and Reference Material Allocations
(1 of 34 competencies)
Competency: Hazard Analysis
Description: The analytical method and approach used to identify, mitigate, and manage
safety hazards found in systems.
Performed by: System Safety Engineer
STEP Level 2 STEP Level 3 STEP Level 4
• Understand and explain the • Prepare a quantitative HA on • Monitor the performance of
basic engineering principles, a NASA space subsystem or HA management systems and
steps, and elements of a system. processes used by the
Hazard Analysis (HA). • Prepare a Hazard Report (HR) organization.
• Identify and explain NASA with all required content. • Conduct analysis of
policy and procedures • Identify, document, and supporting HA data for trends,
governing the conduct of HA. monitor requirements indicators, and pitfalls.
• Prepare a qualitative hazard associated with a HR. • Interpret, apply, communicate,
analysis on a simple, isolated • Monitor the implementation and monitor requirements
NASA space subsystem. and performance of Hazard related to HA for NASA
• Document and present a HA Controls throughout the programs and projects.
to a senior official. lifecycle. • Influence policy or procedures
• Understand and explain the • Evaluate HR/HA prepared by to improve the conduct of
relationships of HA to fault contractors. HA/HR.
tree analysis and the • Present a HA/HR to decision
relationship of the HA with makers.
other SMA disciplines.
Training and/or Reference and/or32 OJT equals COMPETENCY

33. Core and Advanced
System Safety Products
Core System Safety Products & Services
• Are products & services that all NASA’s System Safety Engineers are
expected to be able to produce at any time.
–E.g., Create a Hazard Analysis.
• Are generally independent of specific job location, project, or job function.
Advanced System Safety Products & Services
• Are products & services that NASA’s System Safety Engineers are expected
to be able to produce if their job assignments demand it or as their careers
progress and their responsibilities escalate.
–E.g., Support a SEB for a major procurement/acquisition activity.
• Are generally dependent on specific job location, project, and job function.
33

34. SS Products & Services
Distributed across Levels
STEP Level 2 STEP Level 3 STEP Level 4
• Conduct Hazard Analysis • Conduct Hazard Analysis • Conduct Hazard Analysis
• Conduct Fault Tree Analysis • Conduct Fault Tree Analysis • Conduct Fault Tree Analysis
• Prepare Safety Rationale • Prepare Safety Rationale • Prepare Safety Rationale
• Prepare Safety Data Package • Prepare Safety Data Package • Prepare Safety Data Package
• Support Anomaly Resolution • Support Anomaly Resolution • Support Anomaly Resolution
• Provide Decision Maker Input • Provide Decision Maker Input • Provide Decision Maker Input
• Communicate Safety • Communicate Safety • Communicate Safety
Requirements Requirements Requirements
• Conduct Surveillance & • Conduct Surveillance & • Conduct Surveillance &
Compliance Verification Compliance Verification Compliance Verification
• Safety Performance Metrics • Safety Performance Metrics • Safety Performance Metrics
Monitoring Monitoring Monitoring
---------------------------------------------- ---------------------------------------------
• Safety Acquisition Management • Safety Acquisition Management
• Probabilistic Risk Assessment • Probabilistic Risk Assessment
• Risk Analysis • Risk Analysis
• Integrated Safety Data Package • Integrated Safety Data Package
• Support Certification of Flight • Support Certification of Flight
Readiness Readiness
----------------------------------------------
Green = Core • Safety Organizational
Management
Red = Advanced • Support Lessons Learned
• Manage Policy/Standards &
Requirements
• Support Special Studies
• Support Mishap/Close-call
investigations
34 • Support Audits & Assessments

35. SS Community of Practice
Q: What is the SS COP?
A: Anything associated with the conduct
of System Safety activities that folks
need to know, share, keep, or access.
Examples:
1. Agency “go-to” SS Website.
2. Mass email notification list.
3. Guest Lecture Series.
4. Webcasts, Podcasts, etc.
5. Blogs/Papers with ratings.
6. Clearing House Concept
Tools, Documentation, Reference,
Resources, Conferences, Speakers,
3
Lessons Learned, Training, Etc. 5
35

36. SS Website
36

37. STEP enables Program/Project
Risk Reduction through…
• Providing a competent and highly capable System Safety
workforce.
• Providing predictable, consistent, and value-added System
Safety products & services to programs/projects.
• Analytical tools proliferation (e.g., SAPHIRE/QRAS).
• Enabling risk-informed decision-making (RIDM).
• Providing safety throughout the entire program acquisition
lifecycle.
• Providing a forum for a community of practice to encourage
lessons learned and communications.
• Proactive human capital management.
37

38. State-of-the-Discipline (SOD)
Asks: “What is the health status
of the SS Discipline?” and
“What are we doing about it?”
Sample SOD Metrics
• STEP Qualification distribution.
• Training metrics/rates/distribution.
• Integrity of SS products over time.
• Analytical tools proliferation.
• Proactive Human Capital Management
• Resources, staffing, work-load.
• Rate of Hazard Controls violations.
• Safety-related Technology Readiness.
38

39. State-of-the-Discipline
Example
Agency CMS Competency Level
Agency CMS Competency Level 1 Level 2 Level 3 Level 4 Total
"SAFENG" 181 321 288 197 987
"SYSTEMENG" 325 742 930 503 2500
506 1063 1218 700 3487
Combined CMS Competencies Levels
Level 1 Level 2 Level 3 Level 4 Total
Combined SAFENG and
17 20 21 26 84
SYSTEMENG
3% 2% 2% 4% 2%
Metric: Proactive Human
Capital Management Desired
Desired SOD
metric
• What is the competency improvement
relationship between
System Safety and System
Engineering at NASA?
• Data supports observation
that SMA and SE could be
better integrated.
• Is an overall 2% cross
competency relationship
acceptable?
• What are we doing about it?
39

40. STEPPING FORWARD
• Providing Education for NASA’s Future…
… one STEP at a time.
40

41. Back-up
41

42. SMA Technical Excellence Program
CURRICULUM BY DISCIPLINE
42

43. STEP Internal Review
SOFTWARE ASSURANCE
KAREN MEINERT
43

44. SA Technical Discipline Team
• NSC TDT Lead – Karen Meinert
• NSC SSC – Jennifer Jones
• HQ – M. Wetherholt, M. Bodeau
• ARC – C. Chow, G. DelCarmen
• DFRC – D. Tran
• GRC – T. Luu • JPL – S. Flanagan, J. Wilf
• GSFC- S. Harris • JSC – R. Hugger
• IV&V – L. Montgomery • KSC – D. Victor
• LaRC – L. Johnson
• MSFC – R. Strickland
• SSC – C. Fallo

45. SA Competency Wheel
• Software Assurance is
comprised of the
following competencies:
– Software Quality
– Software Safety
– Software Reliability
– Software Verification &
Validation
– Software Independent
Verification & Validation
– Software Assurance
Planning & Management
45

46. Competencies by Level
Competency: Software Safety
Description: Knowledge, skills and abilities in identifying, analyzing, tracking, mitigating and controlling
software hazards and hazardous functions to ensure safer operation within a system. Activities performed
ensure that safety is designed into the software.
STEP Level 2 STEP Level 3 STEP Level 4
Performance Expectation Performance Expectation Performance Expectation:
Observe and participate with supervision Under general awareness of supervisor, o Determines criteria for safety critical
in software safety activities throughout perform software safety activities software classification
the lifecycle including: throughout the lifecycle including: o Develop/review safety standards
o Identify safety critical software
o Perform software safety litmus test o Identify safety critical software o Complete hazard reports with Systems
o Assist in reviewing: system PHA o Review system hazard analyses Engineering
requirements, designs, code, analyses, o Perform or review software safety o Develop software hazard reports
change requests, testing documents, analyses o Assure hazards verifications are followed
planning documents, lessons learned, and o Verify compliance to appropriate o Assess adequacy of safety requirements
traceability of safety requirements safety standards and software safety documentation
o Review software safety plans o Plan and manage software safety efforts
o Participate in peer reviews o Assure V&V requirements are met o Perform software safety activities
o Attend/observe milestone reviews o Review/assure software safety throughout the lifecycle, including:
and safety reviews requirements are documented and o Develop/review/assess
o Assist in assurance of CM of safety tracked throughout the life cycle requirements, designs, code, and
requirements o Participate in reviews of designs and test documents;
o Assist in software acquisition code o Assess development and test
activities and evaluation of o Review test documentation environments;
contractor deliverables. o Identify potential risk/safety issues o Determine adequacy of V&V
and verify issues are identified and activities;
tracked to resolution (…continued)
(…continued)
46
This is one of six competencies for this discipline

47. Mapping Training to Competencies
47

48. Current Status
Software Assurance Levels 2-4
Discipline/
Training
Course Type Needs Modification New Development
Available
(30 Courses)
Instructor-Led
93% 7% 17%
Courses
Online Courses 7% .5% 1%
Total 100% 7.5% 18%
Hours 374 28 68

49. Course of Study
• The Course of Study
includes a list of
– 30 Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
49

50. STEP Forward
QUALITY ENGINEERING
RAY KACMAR
50

51. QE Technical Discipline Team
• NSC TDT Lead – Ray Kacmar
• NSC SSC – Jerome Anderson
• HQ – Brian Hughitt
• ARC - Steve Jara, Damon Flansburg
• DFRC - Gary May, Kevin Reilly • JPL - Thomas Smith, Tony Gutierrez
• GRC - Bob Makovec • JSC - Dave Dyer
• GSFC - Louis Thomas, Mike Kelly • KSC - Robert Ellison
• LaRC - Elijah Kent, Charlie Greenhalgh
• MSFC - Ken Crane
• SSC - Tim White
• NCAS - Charles Dionne
51

52. QE Competency Wheel
An SMA Quality Engineering
professional needs to possess
these major competencies:
• Perform Quality Assurance
Management & Planning
• Assure Adequate Design &
Development Controls
• Conduct Procurement QE
• Define and Implement Production
Material & Process Controls
• Define and Perform Inspection
Testing & Acceptance Activities
• Implement Problem Reporting &
Corrective Action
• Conduct Quality Evaluations & Data
Analysis The inner ring lists the major competencies. The outer ring provides
a further level of detail to each of the major competencies.
52

53. Competencies by Level
Competency: Quality Management
Description: Recognize, Perform and Evaluate Quality Management Functions
STEP Level 2 STEP Level 3 STEP Level 4
• Recognize QE Quality Management • Perform QE Quality Management • Evaluate Quality Management
functions. functions. Functions and Systems.
• Recognize Quality Management is one • Apply Quality Management principles • Perform a complete quality
of four (4) second tier competencies of to develop Quality Management management system evaluation to
the Perform Quality Management & Systems (QMS). ensure compliance to all applicable
Planning Competency. • Based on a review of the quality QMS requirements defined in the
• Recognize the six (6) third tier requirements, plan the QMS (to include planned evaluation.
competencies in Quality Management. objectives, scope, schedule, and • Become a Certified Quality Engineer
• Understand the concepts of quality technical expertise). (Elective).
management and quality management • Document the results of the QMS, to • Evaluate human factor issues and
systems. include any findings, observations, or human error prevention techniques.
• Understand the concepts of strategic assessments of risk. (Elective).
planning for quality. • Perform strategic planning (Elective). • Provide instruction to operators and/or
• Understand communication • Develop effective communication inspectors in the methods of
mechanisms. mechanisms (Elective). product/service verification, throughout
• Recognize the factors that influence • Identify human factor issues and the value-stream.
the work environment. human error prevention techniques
• Recognize NASA polices on Human (Elective).
Factors.
• Recognize the importance of NASA’s
training & certification requirements
related to quality and identify related
training.
• Recognize related sources of
information.
This is 1of 48 competencies for this discipline

54. Mapping Training to Competencies
(CIL ems Lis d
nt
te
ning t and
) an
sis
eme
As su Quality
d
De si Adequa
ent
ent
t
Revie puts an
lysis ects
s
Inte g Qua lity
sis
Analy
Critic (FMEA
r
t rols t
stem
Qua ment
awa rd/P ost
Orde
agem
agem
en
es & r e
ana g
en
ring
ns &
Stan nship
Man r ation
ool s
nal y
ent
ent
nts
ilu
Man ce
ent
Eff
l opm
s ed
Man ion
agem
es
n
gn &
al S y
s
ur
n Fa
Tree
,
Proc t
ran
Asse l
agem
agem
r atio
n In
ensio
n of
inee
li ty M
r ime
ings
hase
orm
ss m
M an
Man
ure
nic a
rd A
dard
duc
li ty T
al It
ranc
ram
ws
re
lity
ui sit
kma
rd S
wa
figu
k Ba
w
As su
De ve
Desig
Desig
Desig
Critic
Training
Pla n
Pr e-a
Fault
Draw
Expe
Haza
rev ie
Tech
Perf
Pr og
)
Con
Con
E ng
Data
Purc
Risk
Mod
Tole
Co n
Wor
Qua
Qua
An a
Dim
Acq
Ris
Training Course Hours
Qualit y Management Overview 3 2 2
Qualit y Management Syst ems 8 3
Workforce Development , T raining
& Cert ificat ion 0.5
P rogram Qualit y Int egrat ion
Overview 1 2
P rogram Qualit y Int egrat ion 4 3
Qualit y Assurance P lans 3 2
Lessons Learned 1 2
T echnical Assessment Overview 1.5 2
T echnical Assessment 4 3
Audit s 0.5 2
AS 9100 Lead Audit or Class 40 4
Qualit y Assurance Surveillance P lan 1 2
Qualit y T ools Overview 1.5 3
Qualit y T ools 5 4
Qualit y T ools - NASA Addendum 2 4
Lean Six Sigma 8 4
Qualit y Funct ion Deployment 6 4
Design Input s and Reviews Overview 4 2 2
Design Input s and Reviews 8 3 3
Mist ake-P roofing Execut ive
Overview 4 2
Mist ake P roofing 16 3
Workmanship St andards Overview 1 2
Design of Experiment s Execut ive
Overview 4 3
Design of Experiment s 32 4
Drawings, Dimensions, T olerances
Overview 1 2
Drawings, Dimensions &
T olerancing - S-44 40 3
Failure Modes Effect s Analysis &
Crit ical It ems List (SAT ERN) SMA-
017-01 1 2
Syst em Safet y I 8 3 3
Cont inuous Risk Management -
AP P EL - CRM 24 4
Configurat ion Management 4 2
Funct ional Configurat ion Audit /
P hysical Configurat ion Audit 0.5 2
Dat a Management 2 2
P art s Cont rol Overview 1
P rocurement Qualit y Overview 8 2 2 2 2
NASA Let t er of Delegat ion 2
54

55. Current Status
Quality Engineering Levels 2-4
Discipline/
Training
Course Type Needs Modification New Development
Available
(66 Courses)
Instructor-Led
58% 0% 0%
Courses
Online Courses 18% 14% 10%
Total 76% 14% 10%
Hours 282.5 53.5 38

56. Course of Study
• The Quality Engineering
Course of Study includes
a list of
– 66 Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
56

57. STEP Forward
RELIABILITY & MAINTAINABILITY
DR. NAT JAMBULINGAM
57

58. R & M Technical Discipline Team
• NSC TDT Lead – Nat Jambulingam
• NSC SSC – Maria Havenhill, Dan Deans, Bob Loomis
• HQ – Frank Groen
• HQ Tech Fellow – Fayssal Safie
• ARC – N/A
• DFRC – N/A
• GRC - Edward Zampino
• GSFC - Tony Diventi, Matthew Samuel
• JPL - Chau Brown • JSC - Samadar Roshan-Zamir
• KSC - Chi Yeh, Tim Adams
• LaRC – Pete Parker
• MSFC - Richard Stutts
• SSC - Mike Rewis
58

59. R & M Competency Wheel
A Reliability & Maintainability
professional needs to possess
these major competencies:
• Reliability Management
• R&M in Acquisition
• R&M Methods in Design &
Development
• Reliability Testing and
Demonstration
• Use of Database and Reports for
R&M
• Mathematical Tools
The inner ring lists the major competencies. The outer ring provides
a further level of detail to each of the major competencies.
59

60. Competencies by Level
Competency: FMEA/CIL, FMECA
Description: Systematically identifies system’s potential failure modes, and evaluates
their causes and effects on the system’s ability to perform its functions.
STEP Level 2 STEP Level 3 STEP Level 4
• Define failure modes, integration • Interpret and apply FMEA or FMECA • Perform/evaluate FMEA or FMECA on
characteristics, propagation program requirements. increasingly complex systems.
• Describe the contents of FMEAs, Critical • Perform a FMEA or FMECA on assigned • Accurately & thoroughly document
Items List (CIL) and FMECAs. system/subsystem/component. FMEA and Critical Item List analysis
• Describe how and why Critical Items are • Evaluate a FMEA and Critical Item List performed & describe the implications
created out of the FMEA analyses. analysis & coordinate comments with the conclusions may have.
• Locate and read program requirements. the subsystem engineer. • Explain nuances not readily evident to
• Describe the FMEA, Critical items and • Assess compliance to the reliability less experienced personnel.
FMECA process flow. design requirements verified by the • Assess compliance of submitted
• Describe the relationship of CIL to FMEA or FMECA. FMEA/CIL, FMECA to requirements
hazard reports. • Identify common cause failure modes in document.
• Identify simple generic failure modes addition to system failure modes. • Determine compliance to Reliability
and postulate the effects based on an • Identify failure modes that are specific design requirements verified by the
understanding of component to the design configuration under FMEA or FMECA.
functionality. analysis. • Lead a team in developing a FMEA and
• Identify simple failure mode effects at • Trace the effects of failure modes to a Critical Item List.
the system level the system level. • Coordinate the team activity on a
• Coordinate with design engineering to complex system so that the interfaces
develop retention rationale for the between subsystems have coverage
critical items. and the Critical Item retention rationale
• For those engineers involved in meets all the requirements of the
diagnostics, describe the relationship Program FMEA/CIL or FMECA
between FMEA, FMECA and fault requirements document.
detection and isolation.
This is one of 41 competencies for this discipline 60

61. Mapping Training to Competencies
Competencies
R&M into Govern
contracts, ment-
Potential Source SOWs, Industr Comme
specifications, Defining and y Data rcial
deliverables, controlling Parts NASA Exchan Materia and
R&M R&M Managing and award R&M Reliabil Data and Lesson ge l Contra other
Uncert Sensiti Decisio Boolea Progra Starate Evaluating the structure, and requirements ity Fault Root Probab Parts Reliabil Maintai Availab Mecha Softwa Analysi Materia Reliabil Design Reliabil s NASA Progra Review ctors Givern
ainty vity n n Data Markov R&M m gic elements of a other for Block Tree FMEA/ Cause ilistic Risk Stress ity nability ility nical re Human s and l ity Test of ity Test Produc Reliabil NASA Learne Internal m Board Test ment
Training Probab Statisti Analysi Analysi Analysi Algebr Analysi Analysi Planni Manag Manag contractor's acquisition program/projec Diagra Analysi CIL, Analysi Analysi Analysi Analysi Modeli Modeli Modeli Reliabil Reliabil Reliabil Trendi evaluat Planni Experi Develo t Life ity PRACA d Report (GIDEP Report Report agenci
Training Courses Hours ility cs s s s a s s ng ement ement R&M elements t acquisition m s FMECA s s s s ng ng ng ity ity ity ng ion ng ment pment Testing Growth System System s ) s s es data
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
1 OSMA+NSC R&M Principles 24 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 NSC Design for Reliability 32 2 2 2 2 2 2 2
3 NSC Design for Maintainability 32 2 2 2 2 2 2 2
NSC Reliability, Availability, and Maintainability (RAM) Planning for
4 Programs and Projects 14 2 2 2
5 Commercial Data Collection and Analysis I* 16 3 3 3 3 3 3 3 3
NSC Reliability, Availability, and Maintainability Modeling (including
6 success tree analysis) I 16 3 3 3
GSFC/JPL Parts and Materials Assessment (EEE and Mechanical), Parts
Stress/Derating and also incl. Thermal Analysis and Structural
7 Analysis I 16 3 3
NSC
8 Maintainability and Supportability Analysis and Integration* 16 3
9 SATERN/JPL FMEA/CIL, FMECA 16 3
10 NSC R&M Testing and Demonstration I* 16 3 3 3 3
11 Commercial Reliability Growth 16 3
12 NSC Dependency and Common Cause 16 3 3 3
JPL
13 Single Event Upset/Effects Analysis, Radiation Effects I 16 3 3 3 3
14 Commercial Sneak Circuit Analysis 16 3 3 3
OSMA
15 Root Cause Investigation and Corrective Action/Failure Analysis 8 3
16 NSC GIDEP 8 3 3
17 Commercial Certified Reliability Engineer (CRE) Refresher 16
18 SS Probabilistic Risk Assessment (SS) 24 3 3 3
19 QE Problem Reporting (QE) 4 3 3 3
20 SS Hazard Analysis (SS) 8 3 3 3 3
21 SS Fault Tree Analysis (SS) 24 3
22 NSC Data Collection and Analysis* II 16 4 4 4 4 4 4 4 4
NSC Reliability, Availability, and Maintainability Modeling (including
23 success tree analysis) II 16 4 4 4
NSC Parts and Materials Assessment (EEE and Mechanical), Parts
Stress/Derating and also incl. Thermal Analysis and Structural
24 Analysis II 8 4 4
25 NSC R&M Testing and Demonstration II* 16 4 4 4 4
26 Commercial Reliability Growth 16 4
27 SA Software Reliability (SA) 16 4
28 Commercial Physics of Failure 16 4 4 4 4 4
JPL
29 Single Event Upset/Effects Analysis, Radiation Effects II 16 4 4 4 4
30 JPL Worst Case Analysis 24 4 4
Commercial (NRC)
31 Human Reliability 28 3
NSC/ Commercial
32 Design of Experiments 40 4
Cross Discipline
Course Probability and Statistics Basics : Source Document: NASA SP-
** 2009-659 (SS) 3 3
TOTAL COURSES ADDRESSING COMPETENCY 2 3 1 1 1 1 5 1 3 3 3 3 3 3 3 1 4 2 1 2 2 8 8 9 4 1 1 3 2 3 1 3 3 5 1 2 2 3 2 2 1
61

62. Current Status
R & M Levels 2-4 Discipline
Discipline/
Training
Course Type Needs Modification New Development
Available
(32 Courses)
Instructor-Led
19% 0% 15%
Courses
Online Courses 25% 9% 32%
Total 44% 9% 47%
Hours 166 32 176
Total Discipline Hours = 374 [302 hours are required; 72 hours are elective]

63. Course of Study
• The R & M Course of
Study includes a list of
– 32 Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
63

64. STEP Forward
SYSTEM SAFETY
MARK M. KOWALESKI
64

65. SS Technical Discipline Team
• NSC TDT Lead – Mark Kowaleski
• NSC SSC – Maria Havenhill, Diane
Chapman, Jennifer Baumeister, Kathy Kraft
• HQ – Homayoon Dezfuli
• ARC – N/A
• JPL - Kirk Barrow
• DFRC - Jonathan Brown
• JSC - Gerald Readore, Larry Gregg
• GRC - Bill Schoren, Carrie Green
• KSC - Ronald Long, Robert Schwader
• GSFC - Bo Lewis, Angela Melito
• LaRC - John Greco, Jose Caraballo
• MSFC - Herb Shivers, Keith Layne
• SSC - Robert Gargiulo
65

66. SS Competency Wheel
An SMA System Safety professional
needs to possess these major
competencies:
Technical
• System Safety Rationale
• System Safety Analytical Methods
• System Safety Mathematical Skills
Managerial
• System Safety in Acquisition Management
• System Safety in Organizational
Management
The inner ring lists the major competencies. The outer ring provides
a further level of detail to each of the major competencies.
66

67. Competencies by Level
Competency: Hazard Analysis
Description: The analytical method and approach used to identify, mitigate, manage and communicate safety hazards found
in systems.
STEP Level 2 STEP Level 3 STEP Level 4
• Understand and explain the basic • Prepare a quantitative HA on a • Assess the performance of HA
engineering principles, steps, and NASA space subsystem or system. management systems and
elements of a Hazard Analysis • Prepare a Hazard Report (HR) with processes used by the
(HA). all required content. organization.
• Identify and explain NASA policy • Identify, document, and monitor • Compare actual vs. predicted
and procedures governing the requirements associated with a HR. system performance and formulate
conduct of HA. • Monitor the implementation and and manage corrective actions, as
• Prepare a qualitative hazard performance of Hazard Controls necessary, to ensure efficacy of
analysis on a simple, isolated throughout the lifecycle. hazard control.
NASA space, ground, and/or test • Evaluate HR/HA prepared by • Conduct analysis of supporting HA
hardware/operations subsystem. contractors. data for trends, indicators, and
• Document and present a HA to a • Present a HA/HR to decision pitfalls.
senior official. makers. • Interpret, apply, communicate, and
• Understand and explain the • Utilize anomaly resolution input to monitor requirements related to HA
relationships of HA to fault tree evaluate impacts to system-level for NASA programs and projects.
analysis and other relevant hazard risk through the use of • Influence policy, procedures, and/or
analytical techniques, and the qualitative and/or quantitative tools. requirements documents to
relationship of the HA with other • Utilize Fault Tree Analysis (FTA) or improve, standardize and
SMA disciplines. an equivalent tool to map hazard encourage the conduct of HA/HR
causes and controls to other safety for NASA programs and projects.
and reliability data products (e.g.
FMEA/CIL).
This is 1 of 34 competencies for this discipline. 67