2. Software Testing
Testing is the process of exercising a program with the
specific intent of finding errors prior to delivery
o the end user.
errors
requirements conformance
performance
an indication
of quality
2
3. Who Tests the Software?
developer
Understands the system
but, will test "gently"
and, is driven by "delivery "
3
independent tester
Must learn about the system,
but, will attempt to break it
and, is driven by quality
5. Goals
Verification: Have we built the software right?
Bug-free, meets specs
Validation: Have we built the right software?
Meets customers’ needs
Are Verification and Validation different or variation of the
same idea
My argument: “meeting specs” should equal “meeting
customer needs”...
not generally true (my kitchen, satellite systems)
6. Software Testing
Goals
Types of tests
Levels of tests
Test measures
Test plan
most of this discussion focuses
on verification
(more specifically bug testing)
8. Black box tests
input
output
interfa
ce
1. Does it perform the specified
functions?
2.Does it handle obvious errors in
input?
3.Ariane5 – lousy error handling
4.Classic ints vs floats, yards vs
meters
Black box should catch these if
there is adequate “test coverage”
10. Black box tests
Advantage: black box tester≠developer is unbiased by
implementation details. e.g., Use Case testing, just work through
all the Use Cases
Disadvantage: black box tester is uninformed about
implementation details
unnecessary tests – test same thing in different way
insufficient tests – can miss the extremes, especially if actual use
follows a different pattern
12. unnecessary tests
Input
Code
large range of input may exercise a small part of code
e.g., operator test of satellite control stations, run through
each
input and output light/key option. Testing same functions,
13. insufficient tests
Input
Code
a small range of input may exercise a large range of code
but can you ‘know’ this without knowing the code? Did we miss
the 20%
16. Example: ordered list of ints
class ordInts {
public: …
private:
int vals[1000];
int maxElements=1000;
…
}
17. Example: ordered list of ints
bool testMax()
{
L=create();
num=maxElements;
for (int i=0; i<=num; i++)
print i
L.insert(i)
print maxElements;
}
18. White box tests
Advantage:
design tests to achieve good code coverage and avoid duplication
can stress complicated, error-prone code
can stress boundary values (fault injection)
Disadvantage:
tester=developer may have bias
if code changes, tests may have to be redesigned (is this bad?)
19. Example: ordered list of ints
L=create()
L.insert(1)
L.insert(2)
L.insert(3)
class OrdInts
create
getFirst
1
2
3
insert
delete
print
…
L.insert(1001)
p=L.getFirst()
print(p)
p=L.getNext(p)
print p
…
…
getNext
20. Types of tests
Black box: test based on interface, through interface
White box: test based on code, through code
Testing strategy can/should include all
approaches!
My experience:
Black Box = non developer, outside testing idiots
in your case who?
White Box = developer, part of development process
in your case who?
Gray Box = hated, non developer within developer
organization
in your case who?
21. Software Testing
White-Box Testing
Black-Box Testing
requirements
output
.
.. our goal is to ensure that all
statements and conditions have
een executed at least once ...
21
input
events
22. White-Box Testing
Basis Path
Testingwe compute the cyclomatic
First,
complexity:
number of simple decisions + 1
or
number of enclosed areas + 1
In this case, V(G) = 4
22
23. White-Box Testing
Basis Path
Next, we derive the
Testing independent paths:
1
Since V(G) = 4,
there are four paths
2
3
4
5
7
8
6
Path 1:
Path 2:
Path 3:
Path 4:
1,2,4,7,8
1,2,3,5,7,8
1,2,3,6,7,8
1,2,4,7,2,4,...7,8
Finally, we derive test
cases to exercise these
paths.
A number of industry studies have indicated
that the higher V(G), the higher the probability
23
or errors.
26. Black-Box Testing
Comparison Testing
Used only in situations in which the reliability of software is
absolutely critical (e.g., human-rated systems)
Separate software engineering teams develop independent
versions of an application using the same specification
Each version can be tested with the same test data to ensure
that all provide identical output
Then all versions are executed in parallel with real-time
comparison of results to ensure consistency
26
28. Testing measures (white box)
Code coverage – individual modules
Path coverage – sequence diagrams
Code coverage based on complexity – test of the risks, tricky
part of code (e.g., Unix “you are not expected to understand
this” code)
29. Levels of Testing
Unit testing
Integration testing
Validation testing
Focus is on software requirements
System testing
Focus is on system integration
Alpha/Beta testing
Focus is on customer usage
Recovery testing
forces the software to fail in a variety of ways and verifies that recovery is properly performed
Security testing
verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration
Stress testing
executes a system in a manner that demands resources in abnormal quantity, frequency, or volume
Performance Testing
test the run-time performance of software within the context of an integrated system
29
30. Unit Testing
Tests the smallest individually executable code units.
Usually done by programmers. Test cases might be
selected based on code, specification, intuition, etc.
Tools:
Test driver/harness
Code coverage analyzer
Automatic test case generator
31. Integration Testing
Tests interactions between two or more units or
components. Usually done by programmers.
Emphasizes interfaces.
Issues:
In what order are units combined?
How do you assure the compatibility and correctness of
externally-supplied components?
32. Integration Testing
How are units integrated? What are the implications of
this order?
Top-down => need stubs; top-level tested repeatedly.
Bottom-up => need drivers; bottom-levels tested repeatedly.
Critical units first => stubs & drivers needed; critical units tested
repeatedly.
34. Stages of Testing
Testing in the Large
System Testing
End-to-End Testing
Operations Readiness Testing
Beta Testing
Load Testing
Stress Testing
Performance Testing
Reliability Testing
Regression Testing
35. System Testing
Test the functionality of the entire system.
Usually done by professional testers.
36. Realities of System Testing
Not all problems will be found no matter how thorough or
systematic the testing.
Testing resources (staff, time, tools, labs) are limited.
Specifications are frequently unclear/ambiguous and changing
(and not necessarily complete and up-to-date).
Systems are almost always too large to permit test cases to be
selected based on code characteristics.
