An introduction to the standard on reliability assessment.
THE last decade of the twentieth century witnessed a rapid
globalization across the business spectrum. Competitive pressures
have driven all types of electronics manufacturers to adopt
low-cost manufacturing and to evolve a worldwide supply chain.
Today, external sourcing of components and contract manufacturing is
widespread. Electronics manufacturers nowadays are dependent upon
worldwide suppliers who provide them with parts and subassemblies.
Reed Electronics Research predicted in 2005 that by the end of that
year China would have accounted for 16% of global electronics
output, up from 6% in 2000 and under 3% in 1995. In that same ten
year period, electronics output in China would have risen from $28
billion to $210 billion. By contrast, electronics output in the United
States would have reached $342 billion in 2005, up from $285 billion
ten years earlier—a much lower growth rate [1]. A large portion of
the manufacturing growth in Asia, in particular in China, is a result
of outsourcing by multinational electronics manufacturers based in
Europe, Japan, and North America. These statistics cover all levels
of electronic products and services, including components, boards,
assemblies, enclosures, and interconnects.
System integrators, who are at the top of the supply chain, generally
set the requirements for system reliability. Parts and manufacturing services
purchased on the market as commodities are selected based on information
provided by suppliers. However, system integrators usually
know very little about the reliability practices of their suppliers. Often
the organizations require that the suppliers prove reliability of the products
by using outdated and discredited handbook-based reliability predictions
as the first and principal way of measuring the expected reliability
of products. It is only after they receive the parts or subassemblies
that they can assess their reliability. This can be an expensive iterative
process that has to be repeated for each new product. A solution to this
problem is to identify the organizational traits that lead to high-reliability
products and seek out suppliers possessing those traits to partner
with. Therein lies one of the core advantages of reliability capability
assessment.
In a business scenario involving global supply partners, there may be
several options from which to choose. Traditionally, supplier selection
is based on cost, logistics, technical capabilities, production volume,
support locations, and other contractual factors. One of the reasons why
reliability does not typically enter into the decision-making process is
the lack of an accepted methodology to quantitatively measure the capability
of an organization to develop and build reliable products. An
organization’s capability to design for reliability and to implement a
reliable design through manufacturing, testing, and support is i
2. 426 IEEE TRANSACTIONS ON COMPONENTS AND PACKAGING TECHNOLOGIES, VOL. 29, NO. 2, JUNE 2006
involves monitoring and learning innovative analysis techniques and
design or manufacturing technologies that can improve reliability.
Reliability analysis incorporates a group of activities used to con-
duct analysis of a product design to identify potential failure modes
and mechanisms and to make reliability predictions. The analysis re-
sults can include the quantification of risks for each component or
sub-system. The results of this analysis can lead to establishing war-
ranty schemes and making arrangements for spares provisioning at ser-
vice sites for expected failures during and after the warranty period.
Reliability testing is required to explore the design limits of a
product, stress-screen products for design flaws, and demonstrate
the reliability of products. The tests may be conducted according
to some industry standards or required customer specifications. The
reliability testing procedures may be generic—that is, common for all
products—or the tests may be custom-designed for specific products.
Fig. 1. Key organizational reliability practices.
The tests may or may not be used for the verification of known failure
modes and mechanisms. Detailed reliability test plans can include the
3) The supplier should include activities that assure the customer sample size for tests and corresponding confidence level specifications.
that reliability requirements and product needs have been satis- Supply-chain management is required to identify sources of parts
fied. or processes that may be used to satisfy reliability requirements for
Reliability capability assessment is a process for evaluating the ex- a product and to manage suppliers (vendors and subcontractors) for
tent to which the organizational traits exist that help meet these objec- long-term business association. Activities like tracking product change
tives. The need for reliability capability assessment as a supply chain notices and changes in the part traceability markings, and managing
development tool has been recognized by IEEE, which has approved part obsolescence are also included under this key practice. These ac-
development of a standard under project P1624. This project is in the tivities are essential for sustaining product reliability throughout the
process of finalizing a guide for defining reliability capability. It will life cycle. They are also useful for making changes to product specifi-
be an independent guide for defining the criteria for assessing organi- cations, as well as for making design changes during a product’s life
zational reliability capability. Reliability capability will be defined by cycle.
key processes and associated metrics. The guide will be usable by all or- Failure tracking is required to collect manufacturing, yield, and field
ganizations that design, manufacture, or procure electrical/electronics failure data. Failure data analysis is needed to analyze failures, iden-
components or products. The proposed standard does not seek to create tify the root causes of manufacturing defects and field failures, and
or propose creation of certifying bodies that assess whether a com- generate failure analysis reports. The documented records for each re-
pany meets the definitions of reliability capability, but can be used for port can include the date and lot code of the failed product, the failure
self-assessment by companies or for supplier/customer relationship de- point (quality testing, reliability testing, or field), the return date, the
velopment between members of a supply chain. This standard is being failure site, the failure mode and mechanism, and recommendations for
developed under the auspices of the Reliability Society. The active avoiding the failure in existing and future products. For each product
working group includes participants from contract manufacturing, in- category, a Pareto chart of failure causes can be created and continually
dustrial controls, avionics, consumer electronics, telecommunication, updated.
computers, defense, standards organizations, and academia. Develop- Verification through an internal review or audit of reliability plan-
ment and validation of the technical content has been performed by ning, testing, and analysis activities ensures that planned reliability ac-
CALCE, University of Maryland. tivities are implemented so the product fulfills the specified reliability
IEEE Reliability Program Standard-1332 identifies three reliability requirements. Validation is also required to compare the reliability as-
objectives to ensure that every reliability program activity adds value to sertions made at the design stage against the tracked and observed re-
the final product. The three objectives of IEEE 1332 can be addressed liability during operation. The field information on products can be
through eight key practices [4], as illustrated in Fig. 1. Each of these used to update reliability estimates, reliability test conditions, warranty
practices is associated with a number of specific tasks on which the cost estimates, and other logistics specifications, including spares pro-
assessment is based. Altogether, a total of 88 reliability tasks have been visioning.
defined. Improvements can be made in product reliability by using lessons
Reliability requirements and planning comprises the group of learned from testing, reported field failures, technological improve-
activities to evaluate customers’ requirements, to generate reliability ments, and so on. This key practice primarily involves implementing
goals for products, and to plan reliability activities to meet those goals. corrective actions based on failure analysis. It also involves initiating
The inputs for generating reliability requirements for products include design changes in products or processes as a result of changes in re-
customer inputs, reliability specifications for competitive products, liability requirements for products or changes in life-cycle application
and lessons learned from the reliability experience of previous prod- (operating and nonoperating) conditions of products.
ucts, such as test results and field failure data. Constraints like budget,
schedule, and maturity of technology can also affect requirements and
III. METHOD FOR RELIABILITY CAPABILITY ASSESSMENT
subsequent planning.
Training and development is required to enhance the technical, busi- An organization’s capability to supply reliable products is quantified
ness, and specialized skills and knowledge of personnel so that they can using a maturity level metric. This multilevel metric allows comparison
perform their roles in manufacturing a reliable product effectively and of different organizations and also provides a baseline against which
efficiently. The aim is to ensure that employees understand the relia- to measure an organization’s improvement over time. The maturity ap-
bility plans and goals for products, and to improve employee expertise proach to determining organizational abilities has roots in quality man-
in methods required for achieving those goals. This key practice also agement. Crosby’s Quality Management Maturity Grid [5] describes
Authorized licensed use limited to: University of Maryland College Park. Downloaded on May 5, 2009 at 18:32 from IEEE Xplore. Restrictions apply.
3. IEEE TRANSACTIONS ON COMPONENTS AND PACKAGING TECHNOLOGIES, VOL. 29, NO. 2, JUNE 2006 427
the typical behavior of a company, which evolves through five phases testing, qualification, stress analysis, failure analysis, failure tracking,
(uncertainty, regression, awakening, enlightenment, and certainty) in warranties, parts selection, and supplier assessment, as well as any
its ascent to quality management excellence. Since Crosby’s grid was others who provided answers to the questionnaire. These personnel
published, maturity models have been proposed for a wide range of should bring to the meeting “objective evidence” in support of their
activities, including software development, supplier relationships, re- responses to the questionnaire. The evidence may consist of data,
search and development effectiveness, product development, innova- reports, policy drafts, or current documents.
tion, collaboration, and product design. The most commonly used ma- The evaluation team offers an overview of reliability capability to
turity model is the capability maturity model (CMM) developed by the provide an understanding of the rationale and the process. After the
Software Engineering Institute of Carnegie Mellon University. CMM presentation, the company presents an overview of the business and
provides a method for assessing the capability of software contractors operations at its facility, followed by its vision of reliability. This in-
[6]. The reliability capability assessment model is analogous to the cludes, but should not be limited to, reliability objectives for the var-
CMM, but focused on hardware reliability. We define the reliability ious product categories, and a description of its reliability organization
capability maturity metric to be a measure of the practices within an and practices. Specifically, the presentation should include information
organization that contribute to the reliability of the final product, and on the following items:
the effectiveness of these practices in meeting customers’ reliability re- • reliability tasks performed for products;
quirements. • list of test and failure analysis equipment;
Although a set of key practices and associated reliability tasks are • reliability test plan and process guidelines and/or standards;
used in an assessment, reliability capability maturity is more than just • list of reliability tests and some examples;
performing a list of reliability-related tasks. Two organizations may • failure analysis methods and examples;
have similar products and implement similar tasks. The more mature • supplier assessment guidelines;
organization uses the tasks in an integrated fashion within the product • part selection guidelines;
lifecycle. The mature organization implements tasks that provide value • reliability input during product development;
and reduce risk. A less mature organization may only implement tasks • failure tracking strategy and examples;
when required by a customer. By its nature, less mature organizations • warranty determination.
tend not to have an institutional memory and any lessons learned from The evaluation team then assesses responses to the questionnaire and
improvements made in response to a customer request or to address a the supporting evidence, asking follow-up questions as necessary. At
reliability problem are not carried over to other product lines or future the conclusion of the meeting, the company is provided an informal
products. summary of the findings, including recommendations for corrective ac-
Independent of who is conducting the assessment and for what pur- tions.
pose, the process of assessing reliability capability need not be onerous The third and final phase involves documentation of the assessment.
and time-consuming. One approach that has been developed consists of The company is provided with a draft report summarizing the evalua-
a review of documentation and responses to a questionnaire, followed tion team’s observations and recommendations for reliability improve-
by an on-site assessment and preparation and presentation of results ment. The company is typically given an opportunity to review the draft
and recommendations. This process allows a team to determine a reli- report and provide comments. A final report is then issued to the com-
ability capability maturity level of reliability practices for a facility or pany and to the organization that requested the assessment that high-
department. The assessment of capability level helps to identify those lights the areas of strengths and weaknesses, with recommendations
practices that the company is performing to a high standard and also for improvements to approach best-in-class standards. The report also
indicates the opportunities for improvement in reliability achievement. includes the maturity level of the company along with explanation of
The final product is a report that summarizes the assessment process the significance of that level.
and findings. There can be other alternative ways of performing the as-
sessment that may be equally effective. IV. CONCLUSION
The procedure for reliability capability assessment consists of three
phases. In the first phase, a questionnaire is submitted to the company A reliability capability assessment process can assist OEMs and
which consists of nine sub-sections—one section on background infor- system integrators in assessing prospective suppliers for their ability
mation about the company and eight sections pertaining to each of the to design and manufacture reliable products before they are delivered
key practices essential to reliability achievement. The company is re- for use, and on an ongoing basis, help a company in identifying
quested to identify the personnel who are best qualified to answer these shortcomings in its reliability program, which can be rectified by
questions and obtain their responses. The responses should be returned subsequent improvement actions.
to the evaluators before the proposed on-site visit, with sufficient time The assessment can also help to establish reliability management
to study the responses. practices for use by designers, suppliers, customers, and independent
authorities. The assessment method may be used to evaluate the reli-
The company also needs to send additional information before the
ability capability of all types of electronics-related industries that per-
evaluators visit:
form activities influencing the reliability of a product. It can produce
• an organizational chart for the company, which includes the re- increased customer satisfaction, provide competitive opportunities, and
liability functions; shorten the product development cycle. In summary, a reliability capa-
• a list of key products and their reliability requirements; bility assessment can be used for:
• a list of reliability standards and handbooks used in product de- • specifying or planning reliability practices if product develop-
velopment process. ment is implemented internally;
For the second phase, on a mutually accepted date, an evaluation • evaluating reliability practices to determine the extent to which
team visits the company. Company personnel participating in this a supplier is capable of providing a product that meets the relia-
on-site evaluation meeting should include the reliability manager bility requirements/needs; and
and engineers who are involved in activities like defining reliability • improving reliability practices if the current reliability practices
requirements, reliability predictions, derating, manufacturing yields, have been evaluated and improvement is desired or required.
Authorized licensed use limited to: University of Maryland College Park. Downloaded on May 5, 2009 at 18:32 from IEEE Xplore. Restrictions apply.
4. 428 IEEE TRANSACTIONS ON COMPONENTS AND PACKAGING TECHNOLOGIES, VOL. 29, NO. 2, JUNE 2006
REFERENCES Diganta Das (M’00) received the B.Tech. degree
(with honors) in manufacturing science and engi-
[1] A. Fletcher, “All eyes on China, Asia/Pacific,” in Movers and Shakers,
neering from the Indian Institute of Technology,
6th ed. New York: Reed Business Information, 2005. Kharagpur, the M.S. degree in mechanical engi-
[2] IEEE Standard Reliability Program for the Development and Production
neering from the University of Missouri, Rolla, and
of Electronics Systems and Equipment, IEEE Std. 1332-1998, Jun. 30, the Ph.D. degree in mechanical engineering from the
1998. University of Maryland, College Park.
[3] M. Pecht and A. Ramakrishnan, “Development and activities of the
He is a Researcher at the CALCE Electronics Prod-
IEEE reliability standards group,” J. Rel. Eng. Assoc. Jpn., vol. 22, no. ucts and Systems Center, University of Maryland. He
8, pp. 699–706, Nov. 2000.
is coauthor of books on electronic parts obsolescence
[4] S. Tiku and M. Pecht, “Auditing the reliability capability of electronics and electronic parts uprating and contributor of sev-
manufacturers,” Adv. Electron. Packag., vol. 1, pp. 947–953, 2003.
eral chapters on a book on electronic parts selection and management. He is
[5] P. B. Crosby, Quality is Still Free: Making Quality Certain in Uncertain a member of the Editorial Advisory Board of the Microelectronics Reliability
Times. New York: McGraw-Hill, 1996.
Journal. His primary research interest are environmental and operational ratings
[6] M. C. Paulk, C. V. Weber, S. M. Garcia, M. B. Chrisis, and M. Bush, of electronic parts, uprating, obsolescence prediction and management, tech-
“Key Practices of the Capability Maturity Model , Version 1.1,” nology trends in the electronic parts and their effects on the parts selection and
Tech. Rep. CMU/SEI-93-TR-025, ESC-TR-93-178, Software Eng.
management methodologies.
Inst., Carnegie Mellon Univ., Pittsburgh, PA, Feb. 1993. Dr. Das is a member of IMAPS, a Six-Sigma Black Belt, and is the Technical
Editor of Standards Coordinating Committee SCC37, IEEE Standards Associ-
ation.
Louis J. Gullo (M’02) received the B.S. degree in
electrical engineering from the University of Con-
necticut, Storrs.
He has 26 years experience in military, space,
and commercial applications involving electrical
system design, analog and digital circuit design,
design/product assurance, reliability/maintainability
(R/M), systems safety, component engineering, and
production engineering. He is currently employed
with Raytheon Integrated Defense Systems in
Portsmouth, RI, working on the DD(X) Program. Fred Schenkelberg (M’99) received the B.S. degree
Previously, he was the Director of Product Assurance and Reliability Engi- in physics from the United States Military Academy,
neering for Flextronics International. Prior to that, he was the Manager of West Point, NY, and the M.S. degree in statistics from
Product Assurance at Sensormatic/Tyco Safety Products responsible for Elec- Stanford University, Stanford, CA.
tronic Article Surveillance (EAS) and Radio Frequency Identification Devices He is a Technical Director/Consultant at Ops A La
(RFID) product development analysis and testing. Worked at Honeywell for 12 Carte, Saratoga, CA. He left Hewlett Packard (HP)
years. He managed a commercial avionics Reliability Engineering department in 2004 to join the ranks of independent consultants
in Phoenix, Arizona. He developed the Honeywell In-service Reliability As- focused on reliability engineering. He is currently
sessment Program (HIRAP) as a new reliability assessment method alternative working with clients using reliability assessments as
to MIL-HDBK-217, for which he was awarded a patent in January 2004. He a starting point to develop detailed reliability plans
retired from the U.S. Army Reserve Signal Corps as a Lieutenant Colonel. He and programs. Also, he is exercising his reliability
was deployed for one year in the U.S. Army in Operations Enduring Freedom engineering and statistical knowledge to design and conduct accelerated life
and Noble Eagle as the S3, Operations Officer for the U.S. Army Information tests. He joined HP in February 1996 in Vancouver, WA. He moved with HP
Systems Engineering Command (ISEC), Fort Huachuca, AZ. to Palo Alto, CA, in January 1998 and cofounded the HP Product Reliability
Mr. Gullo is a Member of the IEEE Reliability Society ADCOM, the IEC Team. He was responsible for the community building, consulting, and training
TC56, Working Group 2; IEEE SCC-37 Reliability Prediction Working Group; aspects of the Product Reliability Program. He was also responsible for research
and the RMS Partnership Board of Directors. and development on selected product reliability management topics. Prior to
joining HP Corporate, he worked as a Design for Manufacturing Engineer on
deskjet printers. Before HP, he worked with Raychem Corporation in various
positions, including research and development of accelerated life testing of
polymer based heating cables.
Michael H. Azarian (M’06) received the B.S. degree Mr. Schenkelberg is an active member of the RAMS Management Committee
in chemical engineering from Princeton University, and currently the IEEE Reliability Society Santa Clara Valley Chapter Vice Pres-
Princeton, NJ, and the M.S. degree in metallurgical ident and ASQ Reliability Division Treasurer.
engineering and materials science and Ph.D. degree
in materials science and engineering from Carnegie
Mellon University, Pittsburgh, PA.
He is an Assistant Research Scientist at the
CALCE Electronic Products and Systems Center,
University of Maryland, College Park. He has over
13 years of professional experience in the data
storage, advanced materials, and optics industries,
having worked for Philips Research Laboratories, Eindhoven, the Netherlands,
W.L. Gore and Associates, Inc., Elkton, MD, and Bookham Technology,
San Jose, CA, as well as several startup companies. He was most recently
Manager of Quality and Reliability at Bookham Technology where he was
responsible for qualification of optoelectronic products for telecommunications Sanjay Tiku (M’05) received the M.S. and Ph.D. degrees in mechanical engi-
applications. He has published in the fields of electrochemical migration, neering from the University of Maryland, College Park.
capacitor reliability, creep corrosion, nanotribology, structure and properties He currently works for Microsoft Inc., Redmond, WA. Previously, he worked
of thin films, and colloid science. He holds five U.S. patents for inventions in at the Research Center of Tata Motors in India, and he also held a Lecturer
data storage and contamination control. His current research interests include position in Mechanical Engineering at Government College of Engineering and
failure mechanisms in electronic components and circuit boards and reliability Technology, Jammu, India. He has written several papers and book chapters.
of photonic devices. His research interests include quality and reliability of electronic products and
Dr. Azarian has been an Invited Conference Speaker and Guest Lecturer on electronic parts selection and management.
nanotribology and reliability. Dr. Tiku is a member of IMAPS and Phi Kappa Phi.
Authorized licensed use limited to: University of Maryland College Park. Downloaded on May 5, 2009 at 18:32 from IEEE Xplore. Restrictions apply.