Software Testing is a very time consuming activity and consumes enormous amount of effort in any software project. It makes sense to improve productivity of software testing as well as to reduce the defect density in the software, so that overall economy in the project is achieved. In order to do this, we need to understand the defects, their root causes and be able to predict their outcome in advance during estimation. This presentation by Oaksys is an attempt to share its experience of over 10 years (1998-2008) with the practitioners.

1. OAK~TQ Seminars 2008
Celebrating 10 years of Quality Life
Test Defect Metrics and Analysis
Pradeep P. Oak
Bangalore
[January 24, 2008]

2. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/2
“Works are always
associated with the
defects ..
just as the fire is
enveloped with
smoke”
Bhagavad-Gita Ch. XVIII, 48

3. 2989/1B, 12th
Main, 80 Feet Road,
HAL II Stage,
BANGALORE – 560 008
http://www.oaksys.net/

4. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/4
Who we are
• Specialist Independent Software
QA/Testing company
• Established in 1998
• ISO 9001:2000 certified since 2003
• Over 150 Test specialists
• Multiple domain expertise
• Offices in Bangalore, Pune
• Operations in India, USA, Europe

5. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/5

6. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/6
Key Members
• Mr. Pradeep Oak - M Tech., IIT Kanpur
– worked with TCS, Index, …
– consulted with HPISO, Philips, DRDO, Satyam, …
• Mr. Narendranath Pai - B Tech
– expertise in Business & Web Applications
– worked with ANZIT, SAP Labs, PSI, …
• Ms Tejaswini - B Tech
– expertise in Embedded, Real Time & Telecom projects
– consulted with DRDO, Motorola, …
• Ms Asmita Oak - B Tech
– expertise in Quality Management Systems
– worked on V & V projects at DRDO, Synopsys, …
AND .. 150+ others

7. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/7
Services Offered
Managed Testing Service
Software Test Planning & Strategy
development
QA for the complete SDLC
Test case design and driver development
Test Environment setup & Test Execution
Development of Test Automation Framework
User Acceptance Testing
In short, everything in Software Testing

8. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/8
What makes us different?
• Flexible approach
To us, each client is unique. So is each client solution.
• Adaptable delivery
We know that the key to a successful project is the ability to adapt. If
anything is likely to have an impact on a project, we identify and
communicate the issue quickly and take a constructive approach to
problem resolution.
• Attention to detail
• Superior technical ability
• Team-based delivery
We don’t supply individuals - we supply the knowledge and backup of a
specialist organisation. We have the depth to offer a full range of specialist
skills, ranging from project management to automation scripting, focused on
your requirements.
• Long term relationships, not "Touch and Go"

9. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/9
Our research initiatives
• OAK-BVB Center of Excellence in V&V (2003)
• In association with BVB college of Engineering, Hubli
• Focusing on research and propoagation of new techniques in software
Verification & Validation, deriving new methods of analysing test
metrics, ..
• OAK-PDA Center of Excellence in Software
Automation (2005)
• In association with PDA College of Engineering, Gulbarga
• Focusing on test automation, tool evaluation, productivity
improvement etc.

10. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/10
Projects’ Snapshot ..
Testing & Test Automation in Insurance, Banking, CRM domains
System Testing, Automation and UAT of products under Insurance domain
• Cross-platform Testing of a Network Management Product
• Testing WAP based internet software in Medical services domain
Testing and reverse engineering of a B2B software (Automotives)
Test design, development and execution of ISDN v110 protocol
Internet/Intranet based Pension administration software
System Testing of Internet Payment Security System
Test Automation of Telecom applications
IV & V of of Mission Critical Avionics Software (LCA and others) under 2167A,
DO-178B, IEEE 12207, …
Test automation of telecom data validation system
Requirement analysis of Avionics system using formal methods (ESTEREL)
UAT of HRMIS product
Managed Testing of a Global Core Banking Application with CMMi L5 processes

11. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/11
Partial Client List• Aeronautical Development Agency, Min. of Defence, Govt. of India
• ANZ Information Technology Ltd.
• CEMILAC (Centre for Military Airworthiness & Certification)
• CGI India
• CyberCash India
• Dell International Services Ltd
• Electronics & Radar Development Establishment (LRDE)
• First Apex Technologies
• GE India Technology Center (GEITC)
• IBM Global Services India Pvt. Ltd.
• iFlex Solutions Ltd
• India Life Hewitt Ltd. (leading Pension consulting ASP)
• Infosys Technologies Limited
• Price Waterhouse Coopers (PwC)
• SupportSoft
• Symantec (Veritas) India
• Yodlee Inc.

12. OAK~TQ Seminars 2008
Celebrating 10 years of Quality Life
Test Defect Metrics and Analysis
Pradeep P. Oak
Bangalore
[January 24, 2008]

13. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/13
Seminar Agenda
• Tester – y’day – today - tomorrow
• Defect metrics & Analysis
Discussions & experience-sharing of all
• Questions & Comments

14. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/14
Where are we in Testing
• 1980 .. Nothing called s/w tester
• 1990 – QC/QA sneaks in
• Late 90’s – Tester as a role
• 2000’s – Independent testing
• 2005 – Testing as a career option
• many conferences, seminars,
trainings, symposia, ...
• …
• 2010 - ????
. . . .

15. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/15
Industry Expectations
In the last few years .. We have seen lot of
changes ….Expectations are building
up
• Need the right attitude
• Act as Virtual Users
• Job is quite demanding as the profile is
yet to get defined
. . . .

16. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/16
Tester is required to ..
• Find all defects
• Be responsible to own all the
onsite bugs
• Know programming very well
• Be a domain expert
• Be analytical
• Be flexible
• Be available for a 9-6 job
(9 PM to 6 AM !!)
• ….

17. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/17
Can we be a Thinking Tester ?
Goals
• Be an efficient tester/test group
• Improve effectiveness of testing
• Do complete testing [goal accomplishment]
• Be predictable [repeatable]
Let us try to discuss these thru sample metrics

18. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/18
Goal – Improve Test efficiency
Question: What is Test efficiency ?
• Faster
• Cheaper

19. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/19
Test Efficiency
Cost of finding a defect
• No. of TCs designed v/s effort spent
• No. of TCs executed v/s effort spent
• No. of TCs automated v/s effort spent
• Test effort v/s Devp effort
• Waiting time v/s Test time

20. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/20
Cost of a defect
• Cost = effort spent/defects found
• Avg. Effort per defect per project is plotted
Analysis..
Cost of a defect
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
PersonHours

21. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/21
Cost of a defect - Analysis
Why some projects have high costs of testing
• Is it because of good quality of development?
• Is it ineffective testing / inefficient testing ?
• Is the product complex ?
• Is it new technology ?
• Is it new domain / platform ?
• Is the team new ?
• Is a different process followed / process not followed

22. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/22
Test Effort distribution
0
5
10
15
20
25
30
35
Test
Strategy
STP ITP UTP UT IT ST
Testing Activity
%TestEffort
P1
P2
P3
Analysis..

23. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/23
Test Effort distribution - Analysis
• Is there a relationship between planning & execution ?
• Is more planning leading to reduced execution time
(telescopic)?
STP v/s ST
ITP v/s IT
UTP v/s UT
Related Question: Is there a relationship between test effort &
test effectiveness

24. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/24
Cost v/s Value of a Test
• What is the COST of each test ?
• What is the VALUE of each test ?

25. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/25
Cost v/s Value of a Test
• When you test what you gain ?
• When you do not test, what you save/gain ?
• Cost benefit analysis or ROI
– Tangible benefits
– Intangible benefits –
• increased confidence, better image
Why not .. Value of Quality (VoQ) instead of CoQ !!

26. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/26
Goal – Improve effectiveness
Question: What is Test effectiveness ?
• Finding all defects [till current stage]
• No defects reported in next stages
• Finding critical defects more
• Presenting Testing work well to others

27. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/27
Goal – Improve effectiveness
What are the metrics ?
• Defect leakage
• Defect distribution
• Defect prediction
• Defect acceptability by affected teams
• Feedback / Survey
• …..

28. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/28
Defect Leakage Matrix (sample)
SRS Dsgn CUT IT ST UAT Total
SRS 23 5 1 8 2 39
Dsgn -- 56 7 1 64
CUT -- -- 234 6 34 4 278
IT -- -- -- 32 3 2 37
ST -- -- -- -- 128 26 154
UAT -- -- -- -- -- 11 11
Total 23 61 242 39 173 45 583
Defectinduction
 Defect detection (reviews/testing)  Analysis..

29. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/29
Defect Leakage Matrix - Analysis
• Ideal matrix should have values only on the diagonal cells
• Any values above the diagonal indicate defect leakage
Your Comments on defect leakage please ?? ..
• How is the testing process ? UAT / ST / IT / UT

30. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/30
Defect classification
Defects by Severity
0.00%
5.00%
10.00%
15.00%
20.00%
25.00%
30.00%
35.00%
40.00%
Showstopper High Medium Low Misc
Severity
%defects
Analysis..

31. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/31
Defect classification - Analysis
Are important defects being identified ?
It is also important to look at ..
Who is assigning severity ?
Important in whose eyes ?
Are defect classification guidelines defined and agreed

32. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/32
Defect Prediction Model
Defect distribution across SDLC
11
13.5
16.5
17.25
10.5
21
10.25
7
17
0
5
10
15
20
25
SRS Design Coding UT IT ST UAT
Phase
%Defects
Avg %
No. of defects NOW
U R
here
UAT
Analysis..

33. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/33
Defect Prediction Model - Analysis
From past experience, 10% defects got skipped to UAT
Suppose, We have committed to deliver with less than 10 defects
at UAT.
And, found 7 defects at SRS & 17 defects at Design
• What do we infer ?
• What is expected at UAT ?
• Quality of SRS development v/s quality of SRS review
• Quality of Design v/s quality of design review
Suppose, Till the time of delivery, we had already found 400
defects.
• What do we predict at UAT ?

34. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/34
Defect Acceptability
0
50
100
150
200
250
#ofdefects
Code Rev UT IT ST
Phase
TotalDefects
Accepted
Rejected
Withdrawn
Analysis..

35. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/35
Defect Acceptability - Analysis
Defects accepted
Defects rejected
Probable Defects withdrawn
• Comments please ..
• How early defects are reported ? Is there enough time to
fix them?
• Tester’s Image & Pride are at stake

36. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/36
Goal –Test completeness
Question: What is Test completeness ?
• Functional & nonFunctional
• Against standards / objectives

37. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/37
Goal Accomplishment (FURPS+)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Functionality
Usability
Reliability
Perform
ance
SecurityM
aintainabilitySupportability
TC not Appl
TC not Exec
TC Failed
TC Passed
Analysis..

38. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/38
Goal Accomplishment - Analysis
Test cases failed & not executed are important
to look at
Is there a need to replan ?

39. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/39
Completeness / Coverage
Feature coverage
Code coverage – statement, condition, loop, ..
Requirements coverage
Standards/protocol coverage

40. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/40
Goal – Create right visibility
• Planned presentations to management
– what
– How often
• Conduct survey among affected groups
• Collect feedback – internal / external

41. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/41
Justify Ourselves
Developers feel-
“ Testers write lot of tests, but it is a wasted
effort. Real bugs are not uncovered.”
Is this true?
– May be!
– May be not!!
How do we justify?
Measurement - Metrics - Analysis

42. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/42
Testers must …
• Know what they are doing
• Have pride in their work
• Present their work to the right people
• Plan, Measure, Analyse, Refine, ….
Industry responds with rewards
It is an opportunity to be a leader

43. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/43
Going Forward ...
• Industry will learn & mature
• Demand & value of Testing is
bound to increase
• Will have matured models to support
• Focus may then shift more
towards preventive from
corrective

44. Thank You
Your feedback is valuable to us
- We can be contacted at -
OAK SYSTEMS PVT. LTD.
http://www.oaksys.net/
oakTQ@oaksys.net
Phone- (91)-(80)-4126 7681

45. Celebrating 10 years of Quality Life Test Defect Metrics & Analysis Oak~TQ Seminars/1/45
Q & CQuestions and Comments

46. OAK~TQ Seminars 2008
Celebrating 10 years of Quality Life
Test Defect Metrics and Analysis
Pradeep P. Oak
Bangalore
[January 24, 2008]