As a software tester, you may often face a situation in which your customer requires completing testing faster than you can handle given your effort and the amount of test. For example, in order to complete testing 2000 test cases for a build, you need at least 10 days to complete all testing. However, your customer needs to test and release the build within 5 days. You need to make a tough decision to handle this request. This presentation offers you one of the approaches that you can pursue. The presentation discusses an approach to prioritizing test cases using the principles of value-based software engineering. The approach is based on the principle that not every test case is equally importantly, e.g., not each of the 2000 test cases has the same value. A simple Excel tool will also be provided to allow you quickly prioritize test cases and select the ones that generate best value for your customer.

1. Value-Based Software Testing:
An Approach to Prioritizing Tests
STC 2014
VU NGUYEN
Faculty of Information Technology
University of Science, VNU-HCMC

2. Agenda
• Introduction
• Value-based software engineering
• Value-based software testing
2

3. Introduction - 1
3
This release is very
important for our users,
we want your team to
test it within 5 days
The release includes
many changes and new
features. We have to
test 2000 test cases,
and that takes at least
10 days to complete
Well, we understand
that. But we cannot
release later than 5 days
As a QA engineer,
what should you do?
Your client
You

4. Introduction - 2
• Pareto Principle or 80-20 rule
– 80% of the land in Italy owned by 20% of
population
– 80% of the wealth and income controlled by 20% of
population
– 80% of accidents committed 20% motorists
– 80% of Facebook posts made by 20% users
4

5. Introduction - 3
• In software development
– 80% of MS Office users use only 20% of its features
– 80% of the business value come from 20% of the
features
– 80% of the defects come from 20% of the modules
– 80% of the defects come from 20% of the test
cases
5

6. Introduction - 4
• Value-based solution to the test request above
6
% of
value
from
testing
Test cases
100
80
60
40
20
500 1000 1500

7. Agenda
• Introduction
• Value-based software engineering
• Value-based software testing
7

8. Value-neutral software
practices
• Value-neutral software practices
– Every requirement, use case, module, test case is
equally important
• Value-neutral practices are risky because
– project is constrained by time, budget, staff, quality
– software lifecycle is ever reduced
8

9. Value-based SE
• Value considerations are important to
software project success
• Value
– An equivalent in goods, services, or money
– Relative worth, utility or importance
• Non-monetary values are important
– Fairness, customer satisfaction, trust
• Values vary by stakeholders
9

10. Value-based SE - 2
• VBSE: software development practices take
into account value considerations
• Examples
– Value-based requirements
• Defining objectives by taking into account stakeholder
value propositions
• Feature prioritization based on business value, cost, risk
– Value-based testing
• Test case prioritization based on business value, cost, risk
10

11. Value-based SE - 3
• Some important VBSE practices
– Expectations management
• Manage stakeholders’ expectations focusing on their value
propositions
– Risk management
• Manage most important risks (eg., top ten risks)
– Prioritization
• Prioritize requirements, use cases, test cases, etc. based
on value
11

12. Agenda
• Introduction
• Value-based software engineering
• Value-based software testing
12

13. Value-based testing practice
• Test management and execution based on
value
• A value-based testing approach
1. Derive test cases from prioritized requirements
2. Determine risk exposures and risk reduction
leverage related to each test case
3. Prioritize test cases using risks
13

14. Risk management
• Risk is a potential problem
• Risk Exposure (RE) = Loss x Probability
– Loss: the value lost if the risk occurs
– Probability: the chance of the risk to occur
• Risk Reduction Leverage (RRL)
RRL = (REbefore – REafter)/Cost
Cost: can be the effort spent for testing
REbefore, REafter: Risk Exposure before and after testing
is done
14

15. Test case prioritization - 1
• For each test case, determine RE using
– Business value
– Defect criticality
– Defect-prone components
and developers
RE = Size of Loss x Probability of Loss
– E.g., Size = 20 person-hours, Probability = 20%
 RE = 20 * 20% = 4
15
Size of Loss
Probability of Loss

16. Test case prioritization - 2
• Value-based goal: maximizing the value
gained in testing
– by reducing risk associated with tests
– meaning, maximizing Risk Reduction Leverage
(RRL)
• Greedy approach: prioritize test cases based
on RRL (focusing on tests with high RRL first)
RRL = (REbefore – REafter)/Cost
16

17. Research evidence
• Qi Li et al, 2012
– Results from experiment at Galorath Inc.
17
74.07%
77.78%
83.95%
90.12% 93.83%
95.06%
100.00%
4.94%
6.17%
9.88%
16.05%
22.22%
25.93%
58.02%
35.80%
39.51%
45.68%
51.85%
58.02%
61.73%
87.65%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
8 10 12 14 16 18 20 22 24 26 28 30Stop Testing
Value-based
Value-neutral
Value-inverse

18. Tool support
• Calculations can be done easily in Excel
• Test cases are prioritized/sorted according to
RRL
18

19. Making it simple to use
• It can be time-consuming to estimate risks for
all test cases
• We can approximate the process by
– ignoring risks after testing
• prioritizing based on REbefore/Effort
– prioritizing features/functions to test instead of
test cases
19

20. Making it simple to use - 2
• Ignoring risks after testing
– Assuming we get the same risk reduction rate after
testing
20

21. Making it simple to use - 3
• Prioritizing at feature level
– Prioritizing features instead of test cases
– Link test cases with features
– Run test cases according to prioritized features
21

22. Conclusions
• Value considerations important to success of
software projects
• Values dependent on stakeholders
– Monetary, satisfaction, importance, cost, trust,
happiness, etc.
• Value-based testing: prioritizing tests using
risks is important to maximize values of
testing
22

23. References
• Qi Li, Barry Boehm, “Value-Based Software Test Prioritization”,
Annual Research Review, CSSE, USC, Mar 2012
(http://csse.usc.edu/csse/event/2012/ARR/presentations/QLi_ARR
_March2012.pdf)
• Barry Boehm, “Value-Based Software Engineering”. ACM
Software Engineering Notes, 2003; 28(2)
• Barry Boehm, and Victor Basili, "Software Defect Reduction Top
10 List," Computer, vol. 34, no. 1, pp. 135-137, Jan. 2001
(http://csse.usc.edu/csse/TECHRPTS/2001/usccse2001-
510/usccse2001-510d.doc)
23

24. © 2014 HCMC Software Testing Club
THANK YOU