2. Reference
Specification-based regression test selection with
risk analysis
Yanping Chen, Robert L. Probert, D. Paul Sims
Source IBM Centre for Advanced Studies
Conference archive
Proceedings of the 2002 conference of the Centre
for Advanced Studies on Collaborative research
table of contents
3. Outline
1. Introduction
2. Background
3. Specification-based test case selection
4. Risk-based test case selection
5. Evaluation
6. Conclusions
4. 1. Introduction
Regression testing is essential to ensure software
quality.
Regression test selection attempts to select a
cost-minimized subset of test cases.
5. White-box (code-based) V.S. Black-
box (specification-based)
Code-based regression test selection is good for
unit testing, but it has a scalability problem.
Code-based regression techniques are not
suitable for testing larger components at more
abstract level.
Code-based regression techniques require testers
to access and understand the code.
Code-based regression techniques are language-
dependent.
6. Two kinds of regression tests
Targeted Tests, which ensure that important
current customer features are still supported
adequately in the new release.
Safety Tests, which are risk-directed, and ensure
the potential problem areas are properly handled.
8. 2.1 Regression testing
Given a program P, a modified version P’, and a
set T of test cases used previously to test P,
regression analysis and testing techniques
attempt to make use of a subset of T to gain
sufficient confidence in the correctness of P’
with respect to behaviors from P retained in P’.
With selective re-test techniques, we only return
those test cases that test the affected entities of
the modified program.
9. 2.2 Controlled Regression Testing
Assumption (CRTA)
When P’ is tested with T, we hold all factors that
might influence the output of P’, except for the
code in P’, constant with respect to their states
when we tested P with T.
Ideally, regression tests should be run while
CRTA holds.
Testers must identify and document
uncontrollable factors and the test cases that are
potentially affected by them.
10. 2.3 Risk Analysis
Risk is anything that threatens the successful
achievement of a project’s goals.
Using risk metrics to quantitatively measure the
quality of a test suite.
Risk Exposure model by Stale Amland.
RE(f) = P(f) * C(f)
11. 1. P(f), the probability of a fault being present.
2. C(f), the cost (consequence or impact) of a
fault in the corresponding function if it occurs
in operation.
12. 2.4 Requirement traceability
Traceability is a simple, common-sense book-
keeping property that can prevent a wide range
of problem.
Know which test case verifies a given requirements
attribute.
We use the activity diagram to represent the
desired system behavior.
16. 3.1 Changes in code
Changes only happen in the implementation
without affecting the specification.
For a developer, any changes in code should be
documented in a code change history.
For a tester, traceability needs to be established
between the test case and the log of defects.
Test profile
The tester can choose test cases using the
traceability matrix.
17. 3.2 Changes in specification
Changes happen in requirements or design, the
activity diagram need to be modified accordingly.
In a control flow graph (CFG), nodes are used
to represent statements, and edges represent the
control flow between the statement within a
procedure.
We find that the activity diagram is quite similar
to the CFG except that the nodes of activity
diagram describe activities instead of statements.
20. Regression analysis with the activity
diagram
1. We traverse the activity diagram to identify
affected edges.
2. Then we select test cases that execute the
affected edges based on the traceability matrix
to create our Targeted Tests.
21. 4. Risk-based test case selection
4.1 Motivation for risk-based selection.
4.2 Model-based selection of Safety Tests.
4.3 Risk-based end-to-end test scenario selection.
22. 4.1 Motivation for risk-based
selection
If the developer changes code that cannot be
identified by a test profile, and he does not
update the code change history, the tester might
miss some defects.
To achieve enough confidence, we would like to
include some test cases in addition to the
Targeted Tests for our regression test suite.
23. Useful in two situations
1. After running the Targeted Tests, if time and
resources are available, testers may want to
return some test cases to gain more confidence
in the modified system.
2. The time to delivery may be extremely short,
Safety Tests provide some assurance that the
remaining defects in the release will not bring
about serious failures.
24. 4.2 Model-based selection of Safety
Tests
Uses Risk Exposure model for risk analysis.
There are four main steps in our approach.
1. Assess the cost of each test case.
2. Derive severity probability for each test case.
3. Calculate Risk Exposure for each test case.
4. Select Safety Tests.
25. Step 1. Assess the cost of each test case.
Cost means the cost of the requirements
attributes that this test case covers.
Two kinds of costs will be taken into
consideration.
1. The consequences of a fault as seen by the
customer, that is, losing market share because of
faults.
2. The consequences of a fault as seen by the
vendor, that is, high software maintenance cost
because of faults.
29. Step 4. Select Safety Tests
We choose test cases that have the highest value
of RE(t) from the non-targeted tests to reach
our coverage target.
30. 4.3 Risk-based end-to-end test
scenario selection
End-to-end scenarios simulate common user
profiles of system uses. Some are equivalent to
use cases.
More customer-directed.
Highly effective at finding regression faults.
31. One end-to-end consists of several test cases.
Every test case tests one or more requirements
attributes.
Selection strategy obeys two rules
1. Select scenarios that cover the most critical test
cases.
2. Have the suite of scenarios cover as many test
case as possible.
33. Step 1. Calculate Risk Exposure for each
scenario
Step 2. Select scenario that has the highest RE(s)
Step 3. Update table and rebuild table.
Step 4. Repeat Step 2 and Step 3 until we run
out of time and resources.
34. Step 1. Calculate Risk Exposure for
each scenario
RE (s) = ΣRE (t)i , {1 ≤ i ≤ n | test case i is
covered by this scenario}
35. Step 2. Select scenario that has the
highest RE(s)
The scenario with the highest RE(s) covers the
most critical test cases, that is, has the highest
coverage of test cases.
According to our selection rules, it should be
included in the regression test suite.
For example: s001 is put first into the regression
test suite.
38. Step 4. Repeat Step 2 and Step 3 until
we run out of time and resources
The size of our test suite of scenarios is
dependent on the time and resources available.
Regression testing is terminated whenever we
run out of time and resources.
The final selected regression test suite is the
union of the Targeted Tests and the Safety Tests.
39. 5. Evaluation
We applied our approach to three components
of IBM WebSphere Commerce 5.4. Each
component was owned by one experienced
tester. (306 test cases)
65 defects were opened in the first pass.
There were 9 defects found, which failed 28
test cases in the second pass.
41. 6. Conclusions
This technique is Customer-oriented and also Risk-
based.
It provides methods to obtain both Targted Tests
and Safety Tests.
Future work
To decide when to stop regression testing.
Implementing our approach in a production test
environment.