This document discusses the importance of defect prevention in software development. It notes that testing schedules are longer and more costly for low-quality projects compared to high-quality projects. The document advocates shifting investments from failure activities like testing to prevention activities earlier in the lifecycle like requirements reviews and static analysis. A framework is presented for establishing a defect prevention program that includes establishing teams, providing training, and measuring prevention efforts. Specific prevention techniques discussed include code reviews, static analysis, and addressing requirement ambiguities.
1. Presented by: Scott Aziz
24 October 2013
Develop a Defect Prevention Strategy – Or Else!
2. 2
Agenda
• Why is Defect Prevention Critical?
• Cost & Quality / Time to Market Or Else!
• Current Landscape & How Defect Prevention Can Help
• Prevention Framework
• Prevention Methods
• Requirements Reviews
• Ambiguity
• Pre-Test Defect Removal Activities
• Summary
3. 3
Premise
Why is Defect Prevention Critical?
• Testing schedules for low-quality, large software projects are 2-3X
longer and more than 2X as costly as testing for high-quality
projects.
• If defects remain undetected and unremoved until testing starts, it is
too late to bring a software project back under control. It is much
more cost-effective to prevent defects or to remove them prior to
testing.
• Prevention and Appraisal activities remove many more defects per
Engineer-Hour than Failure activities, but organizations often invest
very little in the time-proven methods.
5. 5
The Concern
• About 40-50% of the effort on current software
projects is spent on avoidable rework.1,2
– $0.50 out of every $1.00 for
new/maintenance.
– Repair and rework is the major software
cost driver.
– This is a restrictive business model.
• 1. [Shull et al. 2002] Shull, F., V. Basili, B. Boehm, A.W. Brown, P. Costa, M. Lindvall, D. Port, I. Rus, R.
Tesoreiro, and M. Zelkowitz. 2002.What we have learned about fighting defects. Proceedings of the
8th International Symposium on Software Metrics:
• 2. Jones, Capers and Bonsignour, Olivier; The Economics of Software Quality; Addison Wesley; 2011.
As an engineering discipline, we can do better. Much better.
6. 6
Cost of Quality / Poor Quality Defined
• Prevention Costs
─ Cost of activities to
prevent poor quality
• Appraisal Costs
─ Cost of activities to
detect quality issues
• Internal Failure Costs
─ Costs incurred to fix
quality issues before
delivery to customer
• External Failure Costs
─ Costs due to failures
after delivery to
customer
Prevention Costs
• Training of development team
• Requirements Analysis
Appraisal Costs
• Static Tests
─ Peer Review
(also Prevention)
─ Code Walk-through
• Dynamic Tests (Manual/
Automated)
─ Functional Testing
─ Performance Testing
• Test Equipment
Internal Failure Costs
• Churn – Characterize/Fix/Re-Test
• Opportunity cost of delayed delivery
External Failure Costs
• Bug fixing after delivery (All Types)
• Customer support
• Loss of business
• Litigation
• Fines
7. 7
Why Should We Care about CoPQ?
It’s an accounting problem – right?
Not my job…
Reduction of Cost is not the only factor
• Quality, Time To Market, Customer Delight
Isn't this just a cost of doing business?
We have to take a leadership stance.
• Manage IT as a business within a business.
• Business growth enabled through lower costs.
• Build the best product for lowest cost to best compete.
Poor quality costs are too large to ignore.
8. 8
Align IT Goals / Business
1 1
2 2
3 3
4 4
5
6
7
8
9
10
5
6
7
8
9
10
13
Not in top 20
Forrester Results From Survey Of Business And Technology Decision-Makers
KPI Business rank IT rank
IT cost per business service supported
Internal customer satisfaction survey score
IT spend by business objective (margin, revenue, growth, compliance, etc.)
Percentage of IT projects that meet or exceed expected benefits
Percentage of IT spend on run, grow, change the business
Yr/Yr IT budget growth vs. revenue growth
Yr/Yr unit cost of infrastructure, systems, apps maintenance
IT spend as a percentage of business revenue
Percentage of IT budget spend on R&D, emerging tech, pilots, innovations, etc.
External customer satisfaction (your organization's customers) survey score
Base: 16 CIOS, CMOs, and CFOs
Source: A commissioned study conducted by Forrester Consulting on behalf of The Technology Business Management Council, September 2013
9. 9
External Failures
• Knight Trading Debacle in 2012 (Took 30 minutes
to stop programmed trading; $440M loss)
• Nasdaq FaceBook IPO (Pre-Market Trading
System)
• $62M Compensation Fund
• Race Condition – only could have been discovered under
heavy volume.
2012.
• Investment firm AXA Rosenberg shelled out $217 million to cover investor
losses from what it called a "significant error“
in the computer code for one of its investment models.
• Issues are not limited to just financial companies…
2012.
10. 10
Software Testing Profession Must Evolve
Why Are These Costly Failures Occurring?
• Minimal prevention activities
– When more defects enter testing, the productivity of testing decreases
due to engineering churn; i.e., the costs of testing is higher and the time
to complete testing increases.
• Casual test case design
– multiple thorough studies show approximately 85% of defects in
production could have been detected by simply testing all possible pairs of
values.1
• Testing by untrained/uncertified test personnel
What’s Next -> Regulation
• The SEC is considering writing regulations that would require trading firms
and other market participants to disclose issues with
their trading programs and test them before they are used on the open
market. 2
1. Source: IEEE Computer: Combinatorial Software Testing Aug, 2009 by Richard Kuhn, Raghu Kacker, Jeff Lei, and Justin Hunter.
2. Financial Times Newspaper, August 2012.
12. 12
Internal & External Failures Example
Engineering Cost per Hour $96
Developer effort spent on QA, rework, process, etc. 20%
Management effort spent on QA, rework, process, etc. 10%
Average hours spent to correct a defect that resulted in a
change
14
Average hours to close a defect with a no-change resolution
(duplicate, etc.)
3
Support costs attributed to poor quality 30%
Group Change
No
Change
Group 1 5,675 2,481
Group 2 8,410 2,231
Group 3 912 326
Group 4 7,056 4,243
Group 5 2,258 1,330
Group 6 6,795 2,251
Group 7 3,205 1,564
TOTAL 39,938 14,426
Rework Support Total Staff Cost Total COPQ
TOTAL $53,076,827 $79,230,223 $444,129,231 $177,383,877
13. 13
Minimal Early Activities = Technical Debt
Requirements
Injection Phase
Design
Injection Phase
Coding
Injection Phase
Technical Debt
Prevention and Appraisal activities remove many
more defects per Engineer-Hour than Failure
activities.
Every defect that we can remove more economically
leads to more money available to re-invest in better
coverage methods and tools for testing.
Early Lifecycle Testing Activities are not well planned
due to:
• Minimal Recognition
• No Training
• Few Tools
• Project Inertia
• Change Management
15. 15
What Do We Do?
• The only way to be in control of Quality is to shift it from the uncontrollable Failure Costs to
the controllable Prevention/Appraisal Costs.
• With each incremental increase in Appraisal activities, such as reviews, we can expect a
corresponding and larger reduction in our Failure activities.
Take Control -> Invest Upfront
Change Management is difficult
• Investments are needed to shift the majority of our Cost of Quality to the prevention
and appraisal side of the equation.
─ CoQ will not only be reduced significantly, but it will also be more predictable and more
manageable.
• Mind-set change
• Inspections are not the most enjoyable engineering task compared to designing and
coding.
• Inspections are labor intensive and low tech, but they do work.
• The importance of removing defects pre-test; organizational goals.
16. 16
Framework for Prevention
Causal Analysis and Resolution: part of many process improvement models
(CMMI, ISO, SixSigma)
• An organization-level team to coordinate
defect prevention activities exists.
• A team to coordinate defect prevention
activities for the software project exists.
• Adequate resources and funding are provided
for defect prevention activities at the project
and organization levels.
• Members of the software engineering group
and other software related groups receive
required training to perform their defect
prevention activities.
17. 17
Prevention Activities
Pre-Code
Reusable requirements, architecture, design,
and code
Requirements / Design Reviews
Requirements Testing / Modelling
QFD, Kaizen for software
Using quality-strong methodologies such as
RUP and TSP
Agile: Test First, Then Code; Tests are the
Requirements.
Post-Code
Code Review
Static Analysis
Eliminate
difficult work
Eliminate
waste in
process
Identify areas
of
improvement
Experiment
with solutions
to problems
18. 18
Measuring Prevention
Function Points
• Without prevention: Function Points ^1.2.
• With Prevention: Function Points ^1.05-1.15
Number of Defects = 250 @ 100 Function Points.
Quality Strong Methods = 199 = ^1.15
Requirements Testing = 150: ^1.05-1.10
20. 20
Requirement Error Categories
From literature survey of software engineering, psychology and human cognitive fields
G.S. Walia, J.C. Carver
A systematic literature review to identify and classify software requirement errors
Inform. Softw. Technol., 51 (2009), pp. 1087–1109
If one person wrote it with
one intent and another person
reads it differently, it is
ambiguous.
22. Requirements: What can go wrong?
• Ambiguity of reference
• Dangling Else
• Omissions
─ Causes without effects
─ Missing effects
─ Effects without causes
• Ambiguous logical operators
─ OR, AND, NOR, NAND
─ Implicit connectors
─ Compound operators
• Negation
─ Scope of negation
─ Unnecessary negation
─ Double negation
• Ambiguous statements
─ Verbs, adverbs, adjectives
─ Variables, unnecessary aliases
Fact: If something is ambiguous in the
specs it will nearly always result in a
defect(s) in the code.
Examples:
It, This, The above, The previous, Them, These, They.
“Add field A to field B.
This number must be positive.”
Must be, will be, is one of, should be, could be.
“The value must be either A, B, or C.”
Else? An error condition?
23. 23
Code Reviews
• Keep it Small – Large teams are not productive and have diminishing
returns.
• Rigor and scheduling must be adhered to.
• Effectiveness of review depends on experience of reviewer.
• Can eliminate coding defects by 40-50% or more.
• Collaborative code review platforms
like Gerrit, CodeFlow, Collaborator, ReviewBoard or Crucible.
24. 24
Static Analysis
• Is this the way we engineer software?
• What is static analysis?
• Rules Engine: Code Patterns for Reliability, Performance,
and Security.
• How can it help?
• Leaks, crashes, deadlocks, security vulnerabilities, etc.—
problems that might otherwise take weeks to find.
25. 25
Review Points
• Defect prevention and removal is a proven method to cost and schedule reduction.
• The point at which most failing projects first show signs of serious trouble is when
testing starts. Many projects that are cancelled or have major delays for delivery
showed no signs of distress until testing started.
• Although prevention and pre-test defect removal activities have been utilized since the
software industry began, the literature on software quality is sparse: there is a 20 to 1
ratio between books on testing and books on prevention and pre-test removal
activities.
• From an economic viewpoint prevention and pre-test defect removal is even more
important than testing because it raises testing efficiency, lowers testing costs,
shortens testing schedules, and generates a very solid return on investment.
• From both an economic and quality assurance standpoint, defect prevention, pre-test
defect removal, and formal testing are all necessary to achieve a combination of low
costs, short schedules, and low levels of defects present in the software when it is
delivered to customers.
