Wouldn’t it be nice if you had more insight in product quality, during development? To estimate defects inserted in the product, and check how effective test is in capturing defects? We developed a model to measure defects from requirements to customer feedback after release. It is used to control projects (putting quality next to planning/budget), evaluate risks, and decide on release and support. Baseline data is used to measure and improve process performance.

1. Controlling Project Performance
by Using a Defect Model
SEPG 2008, Tampa, FL
Ben Linders
Ericsson Telecommunicatie B.V., Rijen The Netherlands
Affiliate Software Engineering Institute, Pittsburgh, PA

2. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-312
Overview
Business Needs
Project Defect Model
Experiences
Conclusions
Product quality and process effectiveness

3. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-313
Ericsson, The Netherlands
Market Unit Northern Europe & R&D Center
R&D: Value Added Services
– Strategic product management
– Marketing & technical sales support
– Development & maintenance
– Customization
– Supply & support
+/- 1300 employees, of which +/- 350 in R&D

4. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-314
Business Need for Quality
Multimedia functionality
Stability & Performance
Customizations, flexibility
Outsourcing

5. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-315
Target
Business: Increased R&D Efficiency
R&D Scorecard
Lead-Time, Cost & Quality
Quality: Lower Fault Slip Through (FST)
FST = Number of defects detected in integration &
customer test that should have been detected earlier
“Should” implies that the defect is more cost effective to find earlier.
The test strategy defines what is cost effective

6. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-316
Measurement Values
Use Available Data over Collecting More
Analyze over Measuring
Give Feedback over Micro Managing
Take Actions over Reporting

7. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-317
Required
Control of Quality:
Clear requirements
Quality planned & tracked.
Fact based decisions
Known release quality
Deliver on time
Lower maintenance

8. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-318
Project Defect Model
Why?
– Control quality of the product during development
– Improve development/inspection/test processes
Business Benefit:
Better planning & tracking
Early risks signals
Save time and costs
Happy customers!

9. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-319
Measuring quality
Insertion: Where are defects made? How to prevent?
Detection: Where are defects found? Early/economic removal?
Quality: How many defect are left in the product at release?

10. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3110
Quality Management
Plan
– Documents/code (nr defects made)
– Inspection & Test effectiveness (% detection rate)
Quality consequence of project approach
Track
– Actual nr. defects found
– Estimate remaining defects
Quality status, steer daily work
Project decisions, early escalation
Steer
– Toll Gates, Quality Doors, Product Release
Product Quality figures, quantitative decisions

11. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3111
Project Status Deviation Report regarding QualityProject Status Deviation Report regarding Quality
WhoWhoWhen (due date)When (due date)WhatWhat
Corrective actionsCorrective actions (Mandatory for targets with Minor or Major deviations.)
xxxxx
xxxxx
xxxxx
200y-mm-dd
200y-mm-dd
200y-mm-dd
…………………………..
…………………………..
…………………………..
Status
Analysis of current situationAnalysis of current situation
Targets – ………………………..
Fact – …………………………..
Reason – ………………………….
Consequence – …………………..
TG2 baseline actual Min Max
actual
estimate
0
10
20
30
40
50
60
70
80
# FST to Test # GA Defects DR %
20%
30%
40%
50%
60%
70%
80%
90%
100%
[#]FST,GADefects
[%]DetectionRate
Reporting

12. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3112
History
2001 Defined, pilot project started
2002 Evaluated, 2 new projects
2003 Industrialized, used in all major projects
2004 Integrated in Project Steering Model
2005 Corporate process, Pilot Cost of Quality
2006 Corporate Good Practice
2007 R&D Efficiency, reduce Fault Slip Through, Agile

13. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3113
Functional Test
Project:
– Incremental
– Function Test Team
– Weekly analysis
Functional Testing:
More defects then
estimated
Root Cause Analysis:
– Missed Inspection
– Design Rules
Defect Detected Function Test
Increment 2 Increment 3 Increment 4 Increment 5 Increment 6 TRF Increment 7 Increment 8
Defect Detected Inspections
Increment 2 Increment 3 Increment 4 Increment 5 Increment 6 TRF Increment 7 Increment 8

14. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3114
Improve Inspections
Re-introduce Design Rules
Coach Inspections
More defects inspection
Additional defects in test
DetectionRate
Inspection
Increment2Increment3Increment4Increment5Increment6
TRFIncrement7Increment8
Actualtotal
Target
DetectionRate
FunctionTest
Increment2Increment3Increment4Increment5Increment6
TRFIncrement7Increment8
Actualtotal
Target
Improved Inspection
and Function Test

15. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3115
Release defect prediction
Number of defects predicted
at release (General Avail.)
Actual defects tracked in first
6 months of operation
Accuracy:
– Mostly within 150% range
– Only 1 product > 100% off
– Only 1 product more defects
Maintenance dimensioning
Reduce Cost of Poor Quality
Definition: Defects predicted at GA / Actual defects (%)
Product Release
Expected
GA
Actual
GA
GA
Estimate
Accuracy
R1 21 20 105%
R2 32 18 178%
B R7 2 2 100%
C R1 5 5 100%
D R1 6 1 600%
R2.1 18 15 120%
R3 13 17 76%
R2.2 84 52 162%
R3.0a
R3.0b
R3.0c 60 60 100%
R3.0d 9 9 100%
G R1 66 41 161%
R2a 25 25 100%
R2b 0 0 100%
E
A
GA- 6MOP Defects
H
104 71
146%
F

16. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3116
Agile Approach
Planning game:
Analyze Quality
Demo:
Deliver
Network test:
Verify
Team meeting:
Feedback
Balance Quality - Time – Costs
Early Risk signals
Optimized process

17. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3117
Agile experiences
Planning game:
Investigate solutions
Define Test strategy
Agree with Product Manager
Estimate remaining defects
Reduce Quality risks
Team feedback:
Root Causes: Test coverage, configuration problems
Process update: Inspection, test strategy, delivery test

18. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3118
Key Success Factors
Management Commitment
Everybody involved
Defect classification
Frequent feedback

19. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3119
Management Targets
Design ManagerFault Slip Through
Project Office ManagerDefect Detection Rate
Strategic Product ManagerGA Defects
Target OwnerTarget

20. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3120
Estimation
Analysis
Report
Defect
Modelling
Target
Setting
+
-
Start Pre-study Execution Finish
Data
Collection
• Design
• Test
Target
Commitment
Bridging the gap

21. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3121
Defect Classification
Fault Slip Through: Could have been found?
Orthogonal Defect Classification Triggers
Test Matrices Focus
Discipline maps Proces Flow
Agree & deploy consistently

22. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3122
Feedback
Frequent, short
At the workplace
All data available
Design/test leaders
Show data
ask questions
form conclusions
take needed actions
Feedback: Collected data delivered to the
people that have done the work, in order
to support understanding of the situation
and help them to take needed actions

23. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3123
Benefits
Qualitative
Earlier risk signals: Deliver on time
Incremental Development: Collaboration design-test
Better decisions: Release quality
Process adherence: Increased efficiency
Less defects after release: Maintenance Reduction
Less disturbances: Employee motivated
Quantitative
Higher quality
Reduced lead time
Lower costs
ROI 5:1

24. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3124
Learnings
Estimation & analysis with Design & Test Leaders:
Valuable quality feedback
All defect information in 1 excel sheet:
Detailed insight, easy root cause analysis.
Feedback sessions with project members:
Essential for analysis, conclusions, and actions.
Quality data next to planning and budget.
Deployment and optimizing processes & methods.
Risks reduced: delivery date, budget & quality!

25. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3125
Quality Prediction
Current Model: Estimation
– Extrapolate past performance
– Based on inserted/detected defects
– Plan & track
Wanted: Prediction
– Causes of defects
– What if Scenarios
– Decision taking
All models are wrong
Some models are useful
Deming

26. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3126
SEI Affiliate Research
Quality Factor Model
– Expert opinion, with data
– Quick Quality Scan
– Rough Prediction
– Improvement Areas
Defect Prediction Model
– Data, tuned with expert opinion
– Detailed Prediction
– Improvement Business Case Process
Inputsandoutputs
Influencingfactors
Measurement
DefectsInserted
(documentation,
code)
DefectsDetected
(Inspection, test)
(Un)happycustomers
DesignProcess
Competence, skills
Tools, environment
TestProcess
Competence, skills
TestCapacity
Tools, environment
ResidentDefectsin
DeliveredProduct
ResidentDefectsin
DesignBase
DetectionRate
DefectDensity
FaultSlipThrough
DefectLevel
DefectClassification
Process
Inputsandoutputs
Influencingfactors
Measurement
DefectsInserted
(documentation,
code)
DefectsDetected
(Inspection, test)
(Un)happycustomers
DesignProcess
Competence, skills
Tools, environment
TestProcess
Competence, skills
TestCapacity
Tools, environment
ResidentDefectsin
DeliveredProduct
ResidentDefectsin
DesignBase
DetectionRate
DefectDensity
FaultSlipThrough
DefectLevel
DefectClassification

27. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3127
Pilot Agile: Prevention
Determine defect insertion & detection costs
Predict savings due to less defects inserted
Phase Quality Factor Detected defects Defects left Cost
Req 4.5
Arch 5.1
Impl 5.1
Total development 49
Inspection 5.3 12 36 72
Early Test 5.0 12 25 132
Late Test 6.2 11 14 1136
Customer Test 5.0 5 10 516
Total development 1856
Maint 4000
Total 5856
Phase Quality Factor Detected defects Defects left Cost Savings
Improvement 50
Req 4.9
Arch 5.1
Impl 5.1
Total development 49
Inspection 5.3 12 35 72
Early Test 5.0 11 24 121
Late Test 6.2 10 14 1033
Customer Test 5.0 5 2 516
Total development 1792 3%
Maint 800
Total 2592 56%

28. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3128
Conclusions
Quality has Business Value
You can Measure & Manage Quality
Estimate, Analyze, and Feedback:
– Prevention
– Early detection
– Risk Management
Why not start today?
– Inspections & test
– Release & maintenance
– Agile

29. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3129
Further reading
Papers
– Controlling Product Quality During Development with a Defect Model, in
Proceedings ESEPG 2003 & ESEPG 2004 conferences
– Make what’s counted count, in Better Software magazine march 2004
– Measuring Defects to Control product Quality, in Measure! Knowledge! Action! The
NESMA anniversary book. Oct 2004. ISBN: 90-76258-18-X
– A Proactive Attitude Towards Quality: The Project Defect Model, in Software Quality
Professional Dec 2004 (with Hans Sassenburg)
– Controlling Project Performance Using the Project Defect Model, in Proceedings
Practical Software Quality & Testing 2005 conference
References
– Managing the software process. Watts Humphrey.
– Metrics and models in Software Quality Engineering. Stephen H. Kan.
Ben Linders
Ericsson Telecommunicatie B.V., Rijen, The Netherlands
ben.linders@ericsson.com, +31 161 24 9885

30. © Ericsson Telecommunicatie B.V., Rijen, The Netherlands 2007-08-3130