201008 Software Testing Notes (part 1/2)

Syllabus 8/19/10 3:52 PM

CSE 565
Software Verification, Validation and Testing

Department of Computer Science and Engineering
Ira A. Fulton School of Engineering
Tempe, Arizona 85287-4411

Topics

01. Week 01 - Testing background
02. Week 01 - Testing process activities
03. Week 02 - Requirements-based testing techniques
04. Feature interaction testing
05. Exploratory testing
06. Structure-based testing techniques
07. Integration testing
08. System testing
09. Regression testing
10. Usability testing
11. Performance testing
12. Security testing
13. Design of Experiments
14. Testing tools
15. Software Reliability Engineering
16. Reliability models
17. Statistical testing
18. Test planning
19. Risk based testing
20. Tracking testing progress
21. Testing measures
22. Test documentation

23. Testing standards
24. Test process improvement
25. Test process patterns

[week 01 – Class 01 – Thursday] 8/19/10 3:52 PM

Testing Background

1. Objective of Testing

a. - Minimize risks
b. - Find faults (things that not works)
c. - How robust is the software: Do not fail and if fails it recover very fast
d. - Assess performance

2. When To Stop Testing?

a. - Demonstrate requirements are met
b. - When it is good enough????

Monthly Meetings Of IEEE-CS and ACM
- Sponsor the main conferences
- Phoenix (search for it!)

3. Definitions IEEE

- RELIABILITY: Probability that a give SW application will operate for
some given time period without SW failure.

MTBF - mean time between failure. It is a related concept with
Reliability.

Agencies which certified SW. Each of them have standards:

FAA. Aviation Agency. Standard D0178b. Probability (10^-9).
B787 is another standard (Boeing)
FDA is another agency.

- VALIDATION: Tends to answer the question "are we building the
right product? Is this software do what it needs to do?
(cumple requerimientos de usuario)

- VERIFICATION: Are we building the product right? Does this meets it
requirements. It is the easy one.
(esta implementado acorde a diseño)

Industry talks about V&V concepts which stands for Validation &
Verification.
Other concept is IV&V which stands for Independent V&V

- TESTING: Examination of the behavior of a program by executing it
on sample data sets.

Testers VS Pollsters Analogies

[Poll - encuestas]

[Scientific pollsters such as CNN, NBC News. They take a
representative group for the poll, where the demographics are well
defined which means that the MOE (margin of error) is less than 3.]

Pollsters are paid to make predictions, while Testers …. does tester
make predictions?.

Testers are expected to do predictions about the reliability of
the SW.

Both Testers and Pollsters works with significant constrains, mainly
money and time.

Pollsters cannot do an exhaustive poll. Testers cannot either do
exhaustive testing. Some cases maybe when the SW is very simple
and has very specific inputs.

If you cannot make a exhaustive poll then you have to assure you
have a good sample (representative and valid statistically). The same
happened for a test process.

4. Effective Testing

- Examination of the behavior of a program by executing it on
REPRESENTATIVE sample data sets.

- ROI (Return of Investment): Is a development effort but also a V&V
effort. How many effort we put on doing the development and
the V&V.

Time used for different activities
1/3 - A&D
1/3 – Coding
1/3 - V&V

Cost = f(productivity)…The SW productivity is 1 LOC / hr.

ROI, on software is all about the effort we put on doing the V&V.

5. Measures Of Reliability

- CRUD - Customer Reported Unique Defects. It is expressed by
defects/KLOC. In 2010 the average is 1/KLOC.

defects/Kloc = 1 or 0.1 (2010)

- Another way to measure this is using 6sigma =).

(six sigma) = 3.14/million. Even here is A LOT!

See The Risk Digest
http://catless.ncl.ac.uk/risks


Testing Process Activities

1. Software Testing Process vs. Development Process

- Development process
a. Requirements [What to build – use case scenarios]
b. Design [Pick the Arch patterns. How to meet requirements]
c. Implementation
d. V&V

These 4 activities on development has a analogy on testing process.

- Testing process
a. Test objectives (plan, conf management)
b. Sampling strategy [Test Design]
c. Write test cases and test scripts
d. Execute test

Testing is a mature process and requires specifics skills.

There are some other activities that are analogous and that affects
both, testing and development:

a. Planning
b. Configuration Management
c. Changes of requirements
d. Review

2. Testing Limitations And Constraints

1. Constraint by the requirements (missing or bad reqs by example)
2. exhaustive testing is not possible
3. Limited time and money

3. Testing Attitudes

- Independence. IV&V, independence test, or independence team.

- Customer perspective: understand the customer; know about the
domain.

- Demonstrate that the system works. Go through all the
requirements and be sure that we meet them all. This is kind of clean
process.

- Dirty testing (try to break the system)

- Professionalism. Testing should be an add value to the product.
Should be something where a person can have a career. The different
paths of career on SW could be: development, mgnt, testing. It could
be a good idea to get a “Licensing of SE”, there is a certification on
Texas (fee, test, and educational background). The test covers:
development and reason about correctness (testing, implementation,
prove of correctness).

Note:
A good test engineer has:
-
- an ability to take the point of view of the customer,
- a strong desire for quality,
- a 'test to break' attitude,
- and an attention to detail.

[week 02 – Class 02 – Tuesday] 8/24/10 2:48 PM

0. From the last class

Reading Papers
 Classic Mistakes in Testing
 Testing Best Practices
 Crosstalk about Quality

Testing Attitudes
 Professionalism: testing is a carrier, as develop and as management.
 License as SE in Texas! :-0 (develop and reasoning about correctness).
Include testing, Proof of correctness and Implement
 Testing is not a course into university :-0 but there are a lot of books!

1. Testing Principles

a) Do not throw away your test.
They are going to be useful in the future. Reuse, maintain your product
(the product will be out for a while, it is better to save your case of
testing) regression analysis.

b) Probability of the existence of more errors in a section of code
is proportional to the # of errors already found.
 Remember we have 1 error/Kloc
 Errors tend to generate clusters (because complexity, dependences of
the code, skills of the developer of that section of code, poor or
changing requirements that affect that part).

c) A necessary part of a test case is the definition of the expected
result
A test case consist on: input and expected results.

 Example: Testing a new optimized C++ compiler. The input would be a
file1 with code, and the exit a file2 with code that satisfied defined
characteristics.

 level of independence

d) Faster and Cheaper
Lean manufacturing, lean testing, lean software development :-o

 This has to be with technology, techniques, and reuse.

2. Paper 1. Classic testing mistakes by Brian Marick

• Brian Marick is a king of leader in the area
 [Page 24] Some classic testing mistakes

 2. (Role of Testing). Thinking that the purpose of testing is to
find bugs. There is not problem in the state. However we should
remind that the objective of testing is to find “important bugs”.

 4. (Role of Testing). Not reporting usability problems. It is
common to report only functionality errors. Remain that the user do
not use a tool functionally but by tasks.

 7. (Role of Testing). Starting testing too late (bug detection not
bug reduction). Testing should begin in front. We should write test at
the same time we are writing requirements. You write your test cases
before you write your code. The idea is that developer have more
information about what it is expected.

 10. (Planning the complete testing effort). Putting stress and
load testing off to the last minute. It is a problem due the fact
there would not be enough time to fix any issue.

 11. (Planning the complete testing effort). Not testing the
documentation. Normally the testing is done to the requirements,
but it is also necessary to test the documentation (installations
procedures and configuration instructions).

 19. (Personnel issues). Testers are not domain experts. We need
to have the people who really understand the application.

 23. (Personnel issues). A physical separation between
developers and testers. Separated groups. It is not good to have
this separation.

3. Paper 2. Testing Best Practices by Ram Chillarege

• Report 28 best practices

• 1. (Incremental) Asses software reliability via statistical testing.
One way to talk about reliability is MTBF (Mean Time Between
Failures), the only way to measure this is through statistical testing.

• 2. (Basis) Develop an agile test design. Agile means: flexible, short
cycles, small pieces at the same time.

• 3. (Foundational) Utilize model-base testing-techniques

Whle working on the development we use Model Base Software
Engineering (MBSE) and Software Development, this means have
models such as UML, state machine, Petri nets, and then from here
move to the code.

UML - "state machine" go to test and code!

Something similar happen for testing. We need models and the move
from there to testing cases.

• 4. (Incremental) Cross-functional teams. It is a very good practice
to have tester and developers together at the same time.

Use (IPDT) - Integrated Product Development Team

• 5. (Foundational) Use ODC (Orthogonal Defect Classification).
This technique is used to improve test effectiveness. For each founded
defect we will have attributes associated with it.

Defect Attributes – Defect classification.

 6. (Basis) Automated Test

This is the first homework 

 7. (Foundational) Perform scenario-based testing. Scenario comes
from Uses cases and Use-case modeling.

 8. (Foundational) Emphasize usability testing

 9. (Basis) Test over multiple platforms

4. Paper 3. Software Quality Challenge by W. Humphrey

• Some analogies: Compare SW with other human generated media
(paper/book) 1 defect per page.

(software) 1,000,000 LOC – 40,000 pages / errors

• [Page 4, paragraph 3] The problem with the software is that there is a
lot of combinations and permutations of possible errors.

So many permutations to test all cases software options

State of the art in SW development. 100 defect / 1000 lines

Then we filter errors: first the compiler, then static analyzer (search
for error patterns), then inspection, then unit testing (UT), then
Integration Testing(IT), then System Testing (ST)

And we can obtain 10-20/1000 error per LOC

And then finally some how 1 /1000 error per LOC

Then worry about variation en configuration

• Review the Table 1 and Table 2. (testing variations and possible paths
through a network).

Think on how to test all possible options on the software.

• [Page 5] The eight elements of software quality management

1. Establish quality policies, goals and plans. (V&V).

2. Properly training, couch and support. Make sure that all the
team understand the principles, objectives and tools for testing.

3. Establish and maintain a requirements quality-management
process (V&V).

4. Establish and maintain statistical control of the Software
Engineering process.

5. Review, inspect and evaluate all artifacts (test).
Evaluate the products and process!

6. Evaluate all defects for correction and prevent other similar
problems. (ODC - Orthogonal Defect Classification)

7. Establish and maintain Configuration Management (CM)
System (testing).

8. Continually improve the Process. (testing), postmortem and root
case analysis (CMMI, PSP)

This course is going to be focused on Software Reliability!


1. Testing Techniques

There are two types of testing techniques

 Functional testing: (black box oriented = tests the functionality of an
application as opposed to its internal structures or workings).
This are applicable on any level.

Typical black-box test design techniques include: Decision table
testing, All-pairs testing, State transition tables, Equivalence
partitioning, Boundary value analysis.

 Structural Testing: (white box oriented = tests internal structures or
workings of an application as opposed to its functionality).

The tester chooses inputs to exercise paths through the code and
determine the appropriate outputs. It is analogous to testing nodes in
a circuit

White-box test design techniques include: Control-flow testing, Data
flow testing, Branch testing, Path testing.

We are going to review some of the Functional Testing techniques.

2. Functional Testing (black box)

Useful for all levels of Testing!

a) Scenario-base Testing: directly tined to use-cases (use case driven
testing).

The requirements are expressed on use-cases. (histories with several
steps)

Example: software for AMT system; examples of use-cases: withdraw,
deposit, check balance.

For the use-case withdraw could be different scenarios:
• normal (sunshine case)
• alternative 1(no cash)
• alternative 2(wrong PIN)
• alternative 3 (bad something)

Sampling Strategies for used for case-based testing

b) Requirement-base testing. Is an alternative for Scenario-base
testing.

The requirements are in a document as a list of Shall’s (system shall
do this), then for each of the req. we match one or more testing (we
use a matrix of reqs vs test).

This are functional requirements

Make a table with requirements and testing associated with each
requirement. (a table)

3. Structural Testing (white box)

4. Testing Levels

 Unit Testing
 Integration Testing
 System Testing


1. Functional Testing
1.1. Sampling Strategies (Scenario| Use Case)-Based
1.1.1. Equivalence Partitioning

 technique that divides the input data of a software unit (function,
service, component, program) into partitions of data from which test
cases can be derived. In principle, test cases are designed to cover
each partition at least once. This technique tries to define test cases
that uncover classes of errors, thereby reducing the total number of
test cases that must be developed.

 Example:

Domain = integer 32 bits
-

0

+

Test Cases:

1. -5 -> 5
2. 0 -> 0
3. 5 -> 5
4. max
5. min

Partition: negative, positive and zero.
Take a sample of each partition (valid and invalid partition)

Invalid Partition

- greater than max
- less that min
- characters
- float

2. Equivalent Partitioning Technique

1. Identify Partition.
2. Write test covering as many uncovered valid partitions as possible.
3. For each invalid partition write a test that cover one and only one at
time.

 Assumption: Input are Independent.
BUT this assumption is normally not True.

 Example One:

Test a function f(x, y) that receive two input values in the ranges of
X: 1 .. 10; Y: 50 .. 100

Partitions Test cases
1 2 3 4 5
X
1...10 V V V V
<1 I I
>10 I I

Y
50…100 V V V V
<50 I I
>100 I I

remember step 3, only one invalid value at one time

Test X Y
1 5 55
2 -1 60
3 5 45
4
5

EP requires at least 5 test cases for this example (according with the
three steps described before)

 Example Two:

(scenario testing) ATM system:
scenario one –withdraw; scenario two –deposit; scenario three - atm
pin change.

Let us see a simple use case for amt pin change. And try to do
scenario-base testing:

User System
1. validate the user
2. display options
3. Pin change
4. Prompt new pin
5. enter pin
6. Validate pin
7. Reenter
8. Confirm/Update

Scenarios:

a) normal one
b) alternative 1 (format problem)
c) alternative 2 (pin not match)

How to test normal and alternative scenarios?
We need extra requirements

What is a valid PIN?
4-8 digits
1st and last must be different
cannot have 3 or more digits the same

For this create equivalent partitions (EP).
Our INPUT is PIN

Think about ATTRIBUTES like:

length
- 1. 4-8 is (v)
- 2. <4 is (i)
- 3. >8 is (i)

first/last
- 4. They are different (v)
- 5. They are the same (i)

number of same digits
- 6. <3 (v)
- 7. >=3 (i)

AS developer you think in architecture, design and algorithm
AS tester you identify partitions and make sampling

Then create coverage matrix:

1 2 3 4 5
Length
4-8 V V V V
<4 I I
>8 I I

First/Last
Different V V V V V
Equals I I

# same digits
<3 V V V V V
>=3 I I

Then create test data:

Test PIN
1 1234 (VVV)
2 123 (IVV)
3 123456789 (IVV)
4 1231(VIV)
5 244443(VVI)

Testing is all about sampling!

For the normal scenario 1 test {t1}

For alternative 1 (invalid pin) {t2, t3, t4, t5}

 Problem (in class):

A Payroll System with 2 input:
- Id (5 digits)
- Hours Worked

And three Output
- Id (5 digits)
- Type employee, is based on the Id
00000 - 29999 engineer
30000 – 99999 support
- Gross pay
$10 hr for 1st 40 hr
$15 hr after 40 hr

Additional information:

- max number hours a person can work – 100 hr.
If exceeded then send error.
- min number – 0 hr
- format of the hrs is integer and round down!

Testers need Requirements as Development!

Create coverage matrix:

1 2 3 4 5 6 7 8
Id
5 digit V V V V V V
Engineer V V V V
Support V V V
<5 I I
>5 I I
~digit I I

Hours
0-40 V V V V
41-100 V V V
<0 I I
>100 I I
~integer I I

Remember the steps: 1 the matrix; 2 the valid as many as possible; 3
one invalid at time.

Valid cases are tested with two test cases

Then go for the invalids (only one each time).



 We talk about Functional Testing – Scenario-Based (Sampling
Strategies) Equivalence Partitioning and work with some examples

 Let us check the homework about the “Registration System”

1 2 3 4 5 6 7 8
Line #
5 digit in DB V V V V
<5 I I
>5 I I
~digit I I
5 digit not in DB I I

Semester #
1 V V
2 V V
3 V V
other I I

Output (status)
Cancel V V
Open V V
Closed V V

Output are test variations. We want to try each of them. They are like
States.

Create samples for each semester and each Status. But not
combinations, because we assume are independents.

1. Paper 4. An Introduction to Scenario Testing
by Cem Kaner
 For each scenario 1 or more test

 Scenarios are histories that are motivating, and credible, complex, and
easy to evaluate.
 Differences between requirements analysis and create test cases:
Tester exploits disagreement.

The main idea is to create histories (original and also taken from others
similar systems) with objects inside, with user (good and bad) interacting
with the objects, with system events

1.1.2. Boundary Value Analysis

 Each of the inputs of the scenario must be tested with max, min and
values around the max and min.

(test the edges and around the edges of EP)

• Example (Payroll system from the previous session):

Partitions

Hours
0-40 V
41-100 V
<0 I
>100 I
~integer I

(1) Test the Edges means: <0 ; 0-40 ; 41 -100 ; >100
(2) Test around the Edges:

-1, 0, 1
39, 40, 41
99, 100, 101

3. Paper 5. A review of Boundary Value Analysis
Techniques by David J. Coe.

 Fault – (in Spanish: defecto) incorrect step , process or data definition
in a computer program. A fault causes Failure. Detect fault before
cause a Failure.

 Failure – (in Spanish: falla) inability of a program or component to
meet its function.

 Software testing identifies Failures, which indicate the presence of one
or more faults. (The goal of testing is to find faults!)

 {FAULT - tolerant} Software – Detect the fault and avoid the Failure!

 Single Variable and Multi-Variable

4. Merging EP and BVA

 EP where input independent.
 BVA where
 EP x BVA (both at the same time)

Go back to the problem:

Partitions

Hours
0-40 V

41-100 V
<0 I
>100 I
~integer I

EP defines the following sections
<0
0-40
41-100
>100

BVA defines the followings values for each section
-1
0, 1, 39, 40
41, 42, 99, 100
101

Hit the border and around there.

BVA is done with INPUT but can also be about OUTPUT

Example:

Suppose that the maximum check amount that the system should
generate is $9,999.99 [max]

if the program is Input – Process – Output then create input that generate
$9,999.99

9,999.90 OK --- NO .97

10000.00 reject --- NO .03

Output BVA is testing (validation and verification) but is more about
Review & Inspection.

1.1.3. Decision Trees, Decision Tables and DOE

 DOE = Design of Experiments

 This family of testing techniques are for test combinations of
DEPENDENT Input!

 Example

Inputs:
- Customer [A,B,C]
- Order [1- 1000]

Output:
- Discount

Rules:
1. Customer A
gets 0% discount to < 10
gets 5% discount to 10 – 99
gets 10% discount >= 100

2. Customer B
5% < 10
15% 10-99
25% >= 100

3. Customer C
0% < 10
20% 10-99
25% >=100

With EP, we can do 3 testing cases

Partitions Test Cases
1 2 3

Customer
A V V
B V V
C V V

Order
1-9 V V
10-99 V V
100-1000 V V

The question is, are those three cases enough to test the whole
system. Is it possible to ship the system with only this test? The
answer is NO!

But, in this case the type of customer and the amount of the order are
not independent. So it is not possible to use EP.

Decision Tables.

1 2 3 4 5 6 7 8 9
A X X X
B X X X
C X X X

1-9 X X X
10-99 X X X
10-100 X X X

No think about a problem where you have five inputs values that are
not independent.

F (A, B, C, D, E) and A have 4 partitions, B 5, C 10, D 4 and E 3… this
makes 2400 test cases table, which is huge.

One way to reduce this is reduces the terms. For instance reviewing
the partitions of C we can reduce from 10 to 3.

Other way is to reduce the number of inputs, and instead of testing all
5 inputs at the same time, we look for which inputs should be really
tested together. And we might find out that A,B,C has to be tested
together but not with D and E.

 Example

In the previous example:
If we add the D (I) case to the customer, and we add the values out of
the boundaries, this make 4 partitions for customer and 5 to order
amount, what makes 20 test cases.

But in fact instead of having the 20 of them, we can go with only 12,
the 9 basics and then only three extras WITH ERRORS
INDEPENDENTS!

1 2 3 4 5 6 7 8 9 10 11 12
A V X X X X
B V X X X X
C V X X X
D (I) I X

1-9 V X X X X
10-99 V X X X
10- V X X X
100
<1 I X
>1000 I X

This technique becomes practical doing “assumptions”

Columns becomes test cases!

Decision Tree

 Customer

o A
 1-9
 10-99
 99-100
o B
 1-9
 10-99
 99-100
o C
 1-9
 10-99
 99-100

NOTICE that the decision tree generates the same 9 test cases as the
table decision.

Each PATH correspond with one Test Case!

Trees and Tables are analogous!



 Last class we talk about different forms of testing and use 3
techniques, combinatorial, decision table and trees.
 We finish with a simple problem about online order system and we
obtain 4 test cases with decision tree and 8 with the table.

 Putting all Together: EP+BV+DTT

The tree in the product node -> featured -> ship USA split in two
cases: Yes and No.
Yes -> Free
No -> Standard Shipping

The same happends to Qty node

 The tree have 6 paths

• customer
o 1 time
 free earth shipping (1st case)
o ~ 1 time
 product
 featured
• Shipping at USA
o free us shipping (2nd case)
• Nota USA
o Standard Shipping
 ~featured
• qty
o >100
 Shipping at USA
 Free US Shipping
 Not a USA
 Standard Shipping
o <=100

 Standard shipping (6th case)

 Decision table

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
st
1 time

Yes V V V V V V V V

No V V V V V V V V

Product

Featured V V V V V V V V

Not F. V V V V V V V V

USA Add

Yes V V V V V V V V

No V V V V V V V V

QTY

<100 V V V V V V V V

>100 V V V V V V V V

We have 16 vs. 6?

Remember that for “first time user”, I do not care about Anything?

Most of the time 6 test cases would be sufficient, however we might
want to do the 16 just to be sure that the programmers do not do
anything wrong.

1. First Problem in Class

Scenario-based Testing to a Tuition System

Inputs
- Status (instate, outstate)
- Hours #
- Dependent of university employment

Rates
- In-state: $50/hr for first 12hrs and $600 >12
- Out-state: $500/hrs for first 12hrs and $6000 > 12
- Dependant: Receive 50% discount not matter if it is in-state or out-st

 Decision Tree

• Status
o In-state
 Dependant
 Yes
• Hours <=12
o $25/hr
• Hours >12
o 300/hr
 No
• Hours <=12
o $50/hr
• Hours >12
o 600/hr
o Out-state
 Dependant
 Yes
• Hours <=12
o $250/hr
• Hours >12
o 3000/hr
 No
• Hours <=12
o $500/hr
• Hours >12
o 6000/hr

 Decision Table

1 2 3 4 5 6 7 8
Status

In X X X X
Out X X X X
Hours
<=12 X X X X
>12 X X X X
Depend
Yes X X X X
No X X X X

In this case both tree and table generate the same amount of test
cases.

When building a tree it could get different number of test cases
depending on with which input you start.

The table is recommended to force us in think about combinations!

But Trees show sequences!

2. Second Problem in Class

Combining EP x BVA x DT

See the document: Data Base Exercise. (entry validation)

First step: look the inputs. All the inputs are the ones underlined.

1 2 3..5 6 7 7 8 T
Part #

In list V V V V V

No in list I

Part status code

New V V

Old V

Unknown V V

Req. Inventory

Positive Integer V V V

Spaces V V

Supplier

In DB and PP V

In DB not in PP V

NO in DB

Auto Reorder Flag

Y V

N V

Reorder Qty

Positive Integer V V

Other

Remaining Inv T

Positive Integer V V

Other

Purchase Terms

Positive Integer V

Blank V

Part price

Dollar value V V V

Other

(same color means dependency)

They are not independent so not use EP

Decision Table could generate 1152 combinations :-( But they are not
all dependent. So we can combine!

Independents: Part # y Part Price.
Dependents Part Status Code y Req. Inventory.
Dependents Supplier y Purchase Term.
Dependents Auto Reorder Flag, Reorder Qty y Remaining Inventory.

 We are going to have 3 to 5 valid Test Cases, creating valid situations
 Case 6 is an invalid option for input independent 1
 Case 7 is an invalid combination of input 2 and 3 that are dependents

 Create Valid Inputs and Valid Combinations
 Then Crete Invalid inputs and Invalid Combination separately.

The final table has approximately 5 test cases for valid and a total of
13 to 14 test cases.

And at the end BVA :-o
This three techniques together in real problems



 We have been working with sampling data and synchronous testing
techniques.

 REQ BASED TESTING TECHNIQUES are applicable to all levels: Unit
Testing, System Testing, etc.

 For sampling we see: EP, BV, Decision tree and Decision table

1. Asynchronous Testing

 The king of testing that we will when there are input that can take
place in any point in time during the execution.

 Start wit a simple example

 TIME LINE: system engineering tool for looking in a scenario.

 Example One:

ATM system in a “normal” scenario.

Sequence:
1. Customer insert a card
2. Customer request withdraw
3. Update account
4. Dispose cash
5. Take cash

But, what happens if:

1. lost of power (in any point of time)

 Example Two:

The “call waiting” feature in a cel phone.

A talking with B, C tries to call A. We will have a time line for a normal
scenario.

1. A enter digits for B
2. A presses send button
3. Network connects to B
4. B line is ringing
5. B answer
6. A and B taking

During this time line could happen synchronous events. For instant, C
calling A.

BUT what happened if:

C could call A during different point in the time line.

 C calls when A is entering the digits. A’s phone will ring.
 C calls just in the same time A hit the send button. A will answer C’s
calling.
 C calls when the network is trying to connects B. You might lost the
call.
 C calls when B line is ringing. If A get the call, does B get on hold ?

These examples are to illustrate these techniques scenario-based. And
see what happen when an asynchronous event happen during a time-line.
These asynchronous events could just happen or interrupt the time-line.


1.2. Requirement-Based Techniques
1.2.1. Model-Based Testing

(Paper 6,7,8)

Use the model to generate the solution
UML tool to create the model of the system
And we can create the implementation from the model!

2. Paper 6. Gold Practice, Model Based Testing

 MBT is the automatic generation of efficient test procedures using
models.

 Model, expected input, expected output, run test, compare output.

 Techniques to build the model: decision table, finite state machine,
grammars, Markov chains, state charts (UML).

3. Paper 7. A survey of Model-Driven Testing Techniques

 UML for models
 UML Testing Profile (extension of UML)
(BPEL) – Business Process Execution Language. Used for services!
Both describe behavior!

 Survey:
1. Based on languages used
2. Degree of automatic generation of test scenario
3. Testing target:
- implementation “code”

- abstract model (simulation)
4. Tools supports

 SUT (system under test)

 For Software Architecture:
Requirements -> Design in UML

 For Software Testing:
Requirements -> Design in UML Testing Profile

But this is not the goal of the class!!

4. Paper 8. Model Based Testing Using Software
Architecture

 Software Architecture
- use UML
- or use ADL (architecture description language)

 Syntaxes and semantics

 The paper talk about a technique called acme develop by ABLE/CMU

 Example with a client / server system

 The idea connect model and test

 UML Test Profile feed into Tools, like JUnit

Now let put the theory in practice!

5. State Machines

 We are going to illustrate how to do Model-Based testing with State
Machines

 Example: ATM Machine

1.
Invalid
card
/

Wait
for
a
error

card

2.
Stolen
card

A
3.
Valid
card
/

prompt
for
a
PIN

Capture

the
card
Waiting
for

a
pin

F

5.
Bad
PIN
/
B

Retry
4.
Valid

PIN

5.
Bad
PIN

4.
Valid

PIN

Waiting
for
Waiting
for
a

a
pin
/
2nd
transaction

try

5.
Bad
PIN

C
D

Waiting
for

4.
Valid

PIN

a
pin
/
3nd

try

E

 So if this is the way the system works Test Each Path. For each state
try each input or stimulates

 State machine Coverage: to reach each state and test each stimuli
or event from the state.

 We need to be sure that each state is tested and covered.

 Try to test each event/state/event combination

(3, 1)(4,1)
(3,2)(4,2)
(3,7)(4,7)

 Let us see a Simple Way to attempt to achieve this coverage

Testing cover creation

 3 step process:

1. develop a state testing tree from the final state machine
2. identify test sequences from the tree (paths)
3. develop tests that the contain the sequences
(start  end/terminal state)

 STEP 1. Develop State Testing Tree

1. Start state = ROOT. From the example it would be A.
2. 1st. level. Identify the events and new states reached from the root.

From the example it would includes:

A ->1 -> A (A-1)
A-> 2 -> F (F-2)

A-> 3 -> B (B-3)

where the first letter is the state reached and the number indicates the
event.

3. 2nd and remaining levels,
expand each state not previously expanded in the tree

B -> 4 -> D (D-4)
B -> 5 -> C (C-5)

C -> 4 -> D (D-4)
C -> 5 -> E (E-5)

E -> 4 -> D (D-4)
E -> 5 -> F (E-5)

 STEP 2. Identify Test Sequences (Path)

The terminal states in this example are D and F.

1
2
3,4
3,5,4
3,5,5,4
3,5,5,5

 Step 3. Develop Test Cases That Contains The Sequences

We need to include the sequences in the exact order!!

T1 could be 1, 3,4 this include (1) (3,4)
T2 could be 2
T3 could be 3,4,5
T4 could be 3,5,5,4

T5 could be 3,5,5,5

The goal is not the minimum but if we can it is ok.
In this case we have 5 test cases to cover 6 paths.

3,4 is different from 3,5,4 (remember same order)


1. Continue with Model Based Testing Techniques.

1

2
A
B

4

5

3

C
6

D

7

9

E

8

 STEP 1. Build the tree

A -> 1 -> B
A -> 2 -> A
A -> 3 -> C

B-> 4 -> A
B-> 5 -> D

C ->9 -> E

D -> 6 -> C
D -> 7 ->D
D -> 8 -> E

Do not extend previous visited nodes!

Take care of LEVELS

 STEP 2. Sequences

1,4
1,5,6
1,5,7
1,5,8
2,
3,9

 STEP 3. Test Cases

All of them should start with A and finish on E

1,5,8
3,9
1,4,3,9
2,1,4,3,9
1,5,7,7,8
1,5,6,9

2,3,9


1.4. Design of Experiments (DOE)

 So far we have covered this:
EP
BVA
Decision Tree and Decision Tables
Model-Based Testing
State Based Testing (state machine)

 Now we will review a new technique: Design of experiments (DoE)
 Sometimes called Pair-wise Interaction testing
 We have to be careful when we use this tech
 Is sound cool but some times could be difficult and cause more
problems than solutions

 This is for decision tables that are Too large (a lot
combinations)

 Example One:
Let see the following problem:
Performance testing for a car:

Engine (3 options): 3.0, 3.8, 5.0
Transmission (2 options): manual, auto
Body style (2 options): 2Doors, 4Doors
Tires (2 options): normal, high performance

 Having all this inputs we will have 24 possible configurations (3*2*2*2
options = 24 configurations)
But It could be exhaustive

 DOE ( it is a sampling technique). Instead of test all, use small sample.

 DOE – Pair Wise Combination Testing

 Steps:

1. Identify the inputs parameters to the SUT (systematic under Test)
2. Partition the inputs/ create samples (remember EP)
3. Specify constraints prohibit combinations.

In some systems something can not happen together, for instance
having two buttons to be pressed at the same time. This would a
restriction.

4. Develop test wich satisfy:

For any two parameters/inputs P1 and P2 and for any partition value
V1 for P1 and V2 for P2 there is a test where P1 has the value V1 and
P2 has the value V2.

 Example Two:
P1 (3 values v1, v2, v3) and P2 (v4, v5) , P3 ( 4 partitions), P4 (1
partition), P5 (2 partition) in SUT (system under test) so 48 different
combinations.

 Example One (continuation):

Take the column with more values P1 combine with a second column.
Verify that combinations are satisfied.

Engine (3) Transition(2) Body(2) Tires(2)
3.0 A 2D N
3.0 M 4D HP
3.8 A 2D HP
3.8 M 4D N
5.0 A 4D HP
5.0 M 2D N

You can do 24 cases (all) but DOE warranty that this 6 are enough.

 Example Three

CPU - A B C
OS – D E H
DB – F, G

All combinations are 18

 With Pair wise combination testing at least 9.
 The parameters with the most number of values (3 each) that would
generate 9 combination, this would be the minimum number of
combinations.
 To create the list, first create all possible combinations of the two
partitions with the greater number of values. And then complete the
third column assuring that we have a combination that covers all
options. NOTE: The table below is not complete, it is needed to
complete the third column.

 CREATE AN ORTOGONAL ARRAY 
 Software for this: AETG, Allpairs, tvguwm.

A D F
A E G
A H F
B D F
B E G
B H G
C D G
C E F
C H F

2. Paper 9. The AETG System: An Approach to Testing
Based on Combinational Design

 Describe a tool to do DOE
 Basic combinatorial design paradigm

 See Table 9.

3. Paper 10. Automated Combinatorial Test Methods –
Beyond Pair wise Testing

 They review pair wise as an introductions
 Unlike pair wise technique instead of testing in pair, we can take
groups of inputs, for instance P1-P2-P3 or P1-P2-P4. This would help,
but it is kind of difficult. So here is were automation comes =)!.

4. Problem about Database

 8 inputs with 2 o 3 partitions each one.

 Applying pair wise will be 9 test (because we have 2 cases with 3
partitions (take the 2 with more partitions and multiply).

 If we use pair-wise combination, we will have as minimum as 9 test
cases, but does this test cases would be enough?... No due the
dependency we have in the inputs. This happen even if we use a 3x or
4x flavor of pair-wise.

Here maybe it is do not going to work …
Why? Because dependences

This technique must be a very careful!!!
Pair wise combinations cold cause problems.

THIS is good for configuration testing!
Something that make sense I try this and that!

This software interact with this device driver by example or for
performance or stress testing

This is also used for Interaction Testing between Black boxes.


1. Paper 11. An Innovative Approach for Testing
Bioinformatics Programs Using Metamorphic Testing.

 Bioinformatics programs: organize and analyze large & complex
biological dataset.

 Invoke complex algorithms to extract useful information.

 Test case: input -> expected results

 Testing “oracle” problem
1- N-version programming
Input to several version of the program (v1, v2, v3) and compare.
(version means working implementations)

 Metamorphic Testing:
 Been able to look at the input domain and search “REASONABLE”
outputs.
 Define properties about the output.

 Another example: graph analysis: search for shortest path (G,a,b)

 Another example: graphics (huge data sets)
Moving a Light source to cause a shadow

 In summary, the idea that testing need inputs and predefined outputs
works very well in theory, but there are a lot of fields where this do not
work in the same way. So the methods reviewed above could help for
these fields.

2. Structure-Based Testing (White Box Testing)
 A white box testing methods gets its name due the fact it see the
code. Unlike black box testing methods that are all based only in
requirements.

 Tend to be techniques to be applied into lower level (small units of
code) unit, services, components.

Not for entire systems
Not fot Integration or Qualification Level of Testing

Two categories

 Static Techniques

Analysis of the code. Example: symbolic execution.

 Dynamic Techniques
Execute the code: run for test.
• Structure - control flow.
o Statement coverage
o Decision coverage
o Decision-Condition Coverage
o Multiple decision Coverage
• Data flow

3. Statement Coverage

 Develop test cases such that every statement is execute at least once

 Example

if a <10 or b>5
(1) then x<-50
(2) else x <-0;

if w = 5 or y>0
(3) then z<-48
(4) else z<-5;

 Flow Chart = Control Flow Diagram.
Test predicate (diamond)
Statement (oval)

 4 statement in the previous code

 Path Testing to execute every Path into the code!
Testing strategy execute all path into the code

 Loops are going to make the numbers of paths bigger. So path
coverage is complicate.

 Values of (a, b, w, y) to execute all statements (with 2 cases we do it)!

a = 0, b=0, w=0, y=0 (execute 1 and 4)
a=10, b=4, w=5, y = don’t care (execute 2 and 3)

Whit these two tests cases we are covering all the statements.

 However, this kind of testing is the minimum need.
FAA D0178B standard / best practices:
1. statements 100% statement coverage!

 Compilers help us to do that. By example the PROFILER measure the
time of each statement.

4. Decision (branch) Coverage

 Develop test cases such that each branch (decision point) is
executed at least once.

 Decision point -> conditional statement = test predicates = diamonds

 EJEMPLO UN MAPA CUADRICULAR CON CALLES Y CADA INTERSECION
ES DECIDIR UNA DIRECCION

 Even If I use structured programming (no GOTO) then if I have NOT
100% statement coverage then I have 100% test predicated.

s1
if x<10
then s2
s3

x =5
I have 100% statement coverage
But NOT 100% decision coverage, false case is MISSING!

 Statement Coverage Do Not Imply Decision Coverage!

 Decision Coverage satisfies Statement but Statement not necessarily
satisfies Decision!
So Software must cover 100% Decision Coverage

3. Decision-Condition Coverage

 Develop test such that each condition in a decision takes on all
possible outcomes and each decision takes on all possible outcomes

 This is like Decision++ (the second part of the definition, is Decision
Coverage)

if x<10 or y>50 and z!=0 or flag= T and status = NIL
then S1
else S2

2 test to statement coverage
2 test to decision coverage

But that not enough for Decision-Condition Coverage.
We need to open the test predicate and look inside: there are 5
conditions!

For each condition we need to test their TRUE and FALSE cases

We need 2 cases. All false and All True.

4. Multiple Condition

 Develop test to execute all combinations of conditions within a
decision.
This is the most powefull.

5. Example, Test Binary Search Algorithm)

(1) start <-1;
(2) end <-num;
(3) found <- false;
(4) while starts <= end and not found //(C1) and (C2)
(5) middle <-(start = end)/2
(6) if key > table[middle] //(C3)
(7) then start <- middle +1
(8) else if key = table[middle] //(c4)
(9) then found <-T
(10) LOC <- middle;
(11) else end <- middle -1

We have 4 conditions, and we have 11 statements (each row). We have
three decision points and we have one case with multiple (2)
conditions.

if4
T
F

if6
T
F

if8
T
F

For multiple-condition coverage C1 y C2 make combinations of T y F

For conditions C3 y C4 simply try one T and one F of each one

Example of Execution

10 20 30 40 50 60 70 Key=55

1
2
3
4 with T, T
5
6T
7

4 done before T, T
5
6F

8 with F

11

4 con T, T
5
6T
7

4 con F, T

 Multiple-condition Coverage: 100% NO
 Decision-condition Coverage: 100% NO
 Decision Coverage: NO
 Statement coverage: NO

Now execute this:

10 20 30 40 50 60 70 Key = 40

1
2
3
4 with T, T
5
6 with F

8 with T

9
10

4 with T,F

 Coverage is cumulative

4. Multiple-condition Coverage NOT 100% (75%)
3. Decision-condition Coverage 100%
2. Decision Coverage 100%
1. Statement Coverage 100%


0. Last Class

 We review different types of Coverage in particular Control-Flow
related.

 We review the Binary Search Algorithm

 We do not have 100% Multiple-Condition Coverage. What can we do?

 Test Case 3. Search for the missing case that give me 100% Multiple-
condition coverage.

Do not EXIST!

Loop Entry
1 2
T T

circle middle E
diamante T y F
if T start E
if False diamante

if if true found E
if false end E

So I have 3 path to test

T {impact condition 1}
FT {impact condition 2}
FF {impact condition 1}

But any single path affect both conditions!

TT
FT

TF

So F F can not occurs!

1. Problem

• Code coverage for midterm exam

if x <10 or y>50
(1) then S1
(2) else S2
if w=50 or z>10
(3) then S3
(4) else S4

• Whit this four test cases

x y w Z
T1 0 0 0 0
T2 11 75 50 12
T3 5 15 10 5
T4 15 75 50 3

The question is, do we have 100% coverage with this four test cases?

Statement Coverage
We have 4 statements (each s)

Statement Test Case
1 T1, T2, T3,T4
2
3 T2, T4
4 T1, T3

Decision Coverage
We have 2 decisions points (ifs)

Decision Test Case
1T T1(T o F), T2(F o T),T3(T o F),T4(F o T)
1F
2T T2(T o T), T4(F o T),
2F T1 (F o F), T3(F o F),

Decision-Condition Coverage
We have 4 conditions (underlined)

Decision Test Case
1T T1, T3
1F T2, T4
2T T2, T4
2F T1, T3
3T T2, T4
3F T1, T3
4T T2
4F T1, T3, T4

Multiple-Condition Coverage
We have 2 multiple conditions (line 1 and 4)

Decision Test Case
1 TT
1 TF T1, T3
1 FT T2, T4
1 FF
2 TT T2
2 TF T4
2 FT
2 FF T1,T3

So, we have:

• Multiple-Condition Coverage: 5/8 = 62.5%
• Decision-Condition Coverage and Decision Coverage: 11/12 = 91.6%
• Sentence Coverage: 3/4 = 75%

2. Path Expressions

• Convert Control Flow Graph to path expression!

a

a
a

1
2
3
4
5

a

a

 We can label the edges

 We have 4 paths for 1 to 5

Approach to convert a Control Flow Graph (CFG) into a Path
Expression:

1. Combine serial links by “multiplying” path expressions

2. combine parallel links by “adding”, “+” their path expressions

3. Remove self loop by replacing them with a link of the form x*

• Lets make it more meaningful

Example:

( 1) read (,y)
( 2) while x!=0
( 3) y <- x + y
( 4) if y>10
( 5) then put (y)
( 6) else put (x) 1

a

( 7) if x>10 m

( 8) then x<-x-5
11
2

( 9) else x<-x-1
(10) read (x,y) b

(11) end; 3

Convert the code into graph c

d
4
e

5
6

g

l

f

7

h
i

8
9

j
k

10

The nodes on the CFG are not diamonds and ovals, but circles that
represent the nodes. The labeling of the edges is totally arbitrary.

And then into Path Expression

a ( bc (df + eg) (hj + ik) l ) * m

So, whit this notation we “somehow” can define how many paths we have
from 1 to 11. In this case it is a infinite situation. So if we define how

many times we will run the loop we could mathematically define how
many paths we have.

3. Anomaly Detection

 Static analysis to find potential errors in code!

 It is all about to find anomalies in the code, such as open a file that is
already open, or close a file that is already close. Or assign a value to
a variable in the statement 10 e.g. x=3 and the next line (11) we
assign another different value e.g. x=5.

 There is a theorem that helps to define these situations.

Huans Theorem
Let A,B,C be nonempty sets of character sequences whose smallest
string is at least 1 character long. Let T be a 2-character string. Then if
T is a substring of AB^NC, then T will appear in AB^2C.

 We are going to get sophisticated on the CFG, where edges’ labels are
not going to be arbitrary but they will represent what is going on in
that path. With this kind of labels it would be easier to find any
anomaly.

Hvans theorem works with the path expressions to find anomalies.


0. Last Class

 Continue with Testing Approaches
 We looking at control flow graph , path expression, data anomalies,
continue with that and introduce symbolic execution.

1. Symbolic Execution

 Definition from Paper 3 (homework list):
Symbolic execution represents values of program inputs with symbolic
values instead of concrete (initialized) data and executes the program
by manipulating program expressions involving the symbolic values.

(like algebra)

 Uses
• Probe of correctness
• Generate test data
• Anomalies Detection

Example One:

(0) input A, B
(1) A <- A + B
(2) B <- A - B
(3) A <- 2 * A + B
(4) C <- A + 4

 (0)
A0 – value of A after execute step 0 (defined)
B0 (defined)
C0 (undefined)

 (1)
A1 = A0 + B0
B1 = B0
C1 = C0 = (undefined)

 (2)
A2 = A1 = A0 + B0
B2 = A1 - B1 = A0 + B0 – B0 = A0
C2 = C1 = C2 = (undefined)

 (3)
A3 = 2 *A2 + B2 = 3A0 + 2B0
B3 = A0
C3 = (undefined)

 (4)
A4 = 3A0 + 2B0
B4 = A0
C4 = 3A0 + 2B0 + 4

Example Two:

F

T

if (x<=0) or (y<=0)
then x<-x^2; y <- y^2
1
2

else x<-x+1; y <-y+1;
endif;

if (x<1) or (y<1) T
F

then x<-x+1; y <- y+1
else x<-x-1; y <-y-1; 3
4

endif;

Then, create a Control Flow Diagram (flowchart diagram)

Now for symbolic execution we have 4 paths into the diagram

In these cases we need to execute the code and do the symbolic
execution for each of the paths.

So if we do the True-True case we will have:

TT path:

x1 = X0^2
y1 = y0^2
x3 = x1+1 = X0^2 + 1
y3 = y1+1 = y0^2 + 1

Example Three:
read

read(x, y) F

if(x>7) T

(1) then y<-x+10
(2) else y<-x-10;
1
2

if y > 0
(3) then x<-y+5 T
F

(4) else x<-y-5
3
4

if x + y > 10
(5) then x<-x+10
(6) else x<-x-10; T
F

5
6

TTT Path:

x1 = x0
y1 = x0 + 10

x3 = x0 + 15
y3 = x0 + 10

x5 = x3 + 10 = x0+25
y5 = x0 + 10

The numbers correspond to the sentence number

 The main use of this is to create test data, in particular path conditions

Things gets ugly when we add loops, references inside data structures.

 Assuming things simple this is useful to create test data, in particular
Path Conditions: Necessary Values for the path to be covered.

 Use in conjunction with the branch condition for all the branches
along the path with symbolic variables substituted in.

3. Path Condition

 What values do we need in a path to be transverse.

 Conjunction of the branch conditions for all the branches along the
path with symbolic variables substituted in.

 Example Two (TT Path):

 We identify 4 paths.
For TT Path
( (x0<=0) or (y0<=0) )  ( (x1 < 1) or (y1<1) )

Then, What test data we need to drive down the path?

( (x0<=0) or (y0<=0))  ( (x0^2<1) or (y0^2<1) )

x = -0.5

y = -0.5

This are the values to test the path

 Example Two (FT Path):

Calculate the symbolic variables.

x2 = x0 + 1
y2 = y0 + 1

x3 = x0 + 2
y3 = y0 + 2

Then define the path condition

( (x0>0) ^ (y0>0) ) ^ ( (x0<0) or (y0<0) )

Then, create test data for that!

This show that, this is a infeasible path, it is a path that can not to be
executed (FT is a path invisible THAT CAN NOT BE EXECUTED)

4. Problem in class: Path Condition

 Develop the path condition for TTT path in code from example 3

x1 = x0
y1 = x0 + 10

x3 = x0 + 15
y3 = x0 + 10

x5 = x3 + 10 = x0+25
y5 = x0 + 10

And the conditions are:

initial:
(x0 > 7) and (y1 > 0) and (x3 + y3 > 10)

then,
(x0 > 7) and (x0 > -10) and (x0 > -15/2)

Easy, just make substitutions!


1. Complexity Measure: McCabe Number.

 Now we are going to talk about Complexity measure this is a tool
that take a piece a of code and generate a complexity number for this
code. This is know as McCabe number. This is equal to the complexity
of a graph.

 Code -> # (calculate mcCabe complexity number)
 Based of [graph]

 The formal definition is
V(G) = edges – nodes + 2(connected components) = e –n + 2p

 The informal definition is
V(G) = test predicates + 1,
where test predicates (points of decisions)

 For the example 3 is 4 (3if plus one)

 If a piece of code has a high complexity number it means that it has a
lot of test predicates which means it has a lot of points of decision. On
companies that develop software they define a threshold for this
complexity and if a code gets a number that is over this threshold the
code is flag. It does not means that the code is wrong but it has to be
reviewed more careful.

McCabe complexity testing
Strategy
Structured testing

V(G) = #of basis paths that must be tested = minimum # of test

Let see an example

S1
If x<10

Then s2
Else if y>0
1

Then s3
Else s4
T
F

If z=5
Then s6
Else s7 2

T
F

3
4

T
F

We have 6 paths (3 x 2).

Complexity = 4 (diamantes + 1)

5
6

So, the recommendation us to test 4 cases

 So for the code above we will choose the basis paths, that we call
them basis paths.
 This paths are the ones which can be used to create all other paths.

Approach:

1. Select an arbitrary path
2. Flip the first condition, attempt to return to 1st path
3. Reset first condition, flip the second condition, attempt to return to 1st
path
4. Repeat

(no audio since last exercise, audio continue here)

Example One:

 This are the 4 basis path in the example:

(1) F T T (arbitrary)
(2) T _T
(3) F FT
(4) F TF

This would be the four basis paths according with McCabe technique. We
have 4 basis paths due the fact the complexity of the code is 4.

read

Example Two (with 3 consecutive ifs)

 Complexity : 4 (diamonds + 1) F

T

 For basics Paths
1
2

(1) T T T
(2) F T T
(3) T F T T
F

(4) T T F
3
4

T
F

5
6

 In the past classes we talk about Code coverage (white box).

 We mention: Control and Data flow coverage.

 Data Flow Coverage, look definitions an uses of data and create paths.
(next class)


0. Last class

 Control Flow Coverage: McCabe Testing, Decision coverage, Multiple
Condition, etc.

 Today Data flow coverage.

1. Data Flow Coverage

 Even 100% Control Flow coverage, that means even 100% multiple
condition coverage do not warranty no errors.

Try to do all path coverage it still do not warranty no errors)

 Data flow coverage extend the capability of detect errors.

Data is going to search data problems like point with cero division

Definition:
Variable Occurrence Classifications

 def: (variable defined) x:=0

 c- use: (computational use) variable used in a computation. y = x +1

 p-use: (predicate use) variable is used in a predicate. If x<1

Approach:

1 Classify Variables.
2. Construct Control Flow Graph (CFG) with annotations about (def/use)

 For each node:
o def(i) : set of variables for node i which are globally defined
o c-use(i): set of variables for which node i contains a c-use
o p-use(TP) : set of variable for which the TP contains a p-use (TP
= test predicate)

Example One:

if x<10
then y<-0; z<-0
T
F

else y<-5; y<-10
endif

1
2

 In this example we can see that we have nodes in our graph that have
multiple statements. Node (1) has 2 statements Y=0 and z = 0.

1

Example Two:

T
F

(1) get x,z
y <- 0 TP1

if x > 10
2

(2) then y <- 15;

if z > 0
(3) then w <- y + 1
T
F

(4) else w <- y - 1

TP2

def(1) = {x,y,z}
3
4

def(2) = {y}
def(3) = {w}
def(4)= {w}

c-use(1) = 0 (nothing)
c-use(3) = {y}
c-use(4) = {y}

p-use(TP1) = {x}
p-use(TP2) = {z}

 Then, a new definition, CLEAR PATH w. r. t (whit respect to) "some
variable"

Definition: Clear Path (def-clr)
It is a path between two nodes i and j with intermediate nodes (i, N1, N2,
… Nm, j) which contains no definitions of x in (N1,… , Nm)

Definition: Definition Computation Use (dcu)
dcu(x, i)
Where i is any node, and x is defined in i
dcu(x, i) is the set of all nodes j such that x is in c-use(j) and for which
there is a definition-clear path w.r.t x from i to j

Definition: Definition Predicate Use Path (dpu)
dpu (x, i) [definition /predicate use set]
is the set of all TP such that x is in p-use(TP) and for which there is a def-
clr path w.r.t. x from i to TP

Definition: Use Path (du)
It is a path (n,…. Nj, Nk) is a du path w.r.t. x if N1 has a global definition
of X and either:
1. Nk has a c-use of x and (n1, … nj) is a def-clr path w.r.t. x
2. Mk is a TP and has a p-use of X and (ni, ..nj) is a def-clr path w.r.t. x

(the path from definition to use )

Go back to Example Two:

def(1) = {x, y, z}
def(2) = {y}
def(3) = {w}
def(4)= {w}
1

T
F

c-use(3) = {y} TP1

c-use(4) = {y}
2

p-use(Tp1) = {x}
p-use(Tp2) = {z}

T
F

 dcu(var, node)
nodes that use the “var” defined in “node”
TP2

dcu (x,1) = empty
3
4

dcu (z,1) = empty
dcu (y,1) = {3,4}

dcu (y,2) = {3,4}

dcu (w,3) = vacio

dcu (w,4) = vacio

 dpu(var, node)

dpu (x,1) = {TP1}
dpu(z,1) = {Tp2}

 Test Coverage Criteria for data flow means "all-uses coverage"
We want 100%

3. All-Uses Coverage

 Goal: develop test cases such that for each definition, there is a
definition use path (du) to every use of the definition.

1

Problem One

get x, y; T
F

a <- 0;
b <- 0; TP1

2
3

if x>10
then w <-a + 1; b <- 4
else w <-b + 1; a <- 4;
T
F

if y>10
then z <- a + w
else z <- b + w; TP2

4
5

def(1) = {x,y,a,b}
def(2) = {w,b}
def(3) = {w,a}
def(4) = {z}
def(5) = {z}

c-use(1) = empty
c-use(2) = {a}
c-use(3) = {b}
c-use(4) = {a, w}
c-use(5) = {b, w}

p-use(TP1) = {x}
p-use(TP2) = {y}

 Create a dcu for each pair def(N) = {i1, i2….}

 Search {i1. I2…} en c-use(M) = {i1, i2…} and add the numbers M to
the set

dcu(x,1) = {}
dcu(y,1) = {}
dcu(a,1) = {2, 4}
dcu(b,1) = {3, 5}
dcu(w,2) = {4}
dcu(b,2) = {5}
dcu(w,3) = {4}
dcu(a,3) = {5}
dcu(z,4) = {}
dcu(z,5) = {}

dpu(x,1) = TP1
dpu(y,1) = TP2

X Y Coverage
Test 1. TT 15 15 Def(a1) & c-use(a2) & c-use(a4)
Test 2. FT 5 15 Def(a1) & c-use(a4)

 For the 4 paths. Test EVERY definition

 Give me all-uses coverage with respect to a but not respect to b
You need to do this for all-uses.

The same use cases give me all te missing ones 

X Y
Test 1 15 15
Test 2 5 15
Test 3 5 5
Test 4 15 5

What control – Flow Coverage do I have?
100% statement coverage

100% condition coverage

All definitions criteria is satisfied if for each definition in the program
some use of this definition is executed by at least one input in the test
suite.

201008 Software Testing Notes (part 1/2)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to 201008 Software Testing Notes (part 1/2)

Similar to 201008 Software Testing Notes (part 1/2) (20)

More from Javier Gonzalez-Sanchez

More from Javier Gonzalez-Sanchez (20)

Recently uploaded

Recently uploaded (20)

201008 Software Testing Notes (part 1/2)