reference "Automating Regression Testing For Evolving GUI Software"

1. Automating Regression Testing
For Evolving GUI Software
Zheng-Wen Shen
2005/10/19
1

2. References
Automating regression testing for evolving
GUI software
– Atif Memon, Adithya Nagarajan and Qing Xie
– JOURNAL OF SOFTWARE MAINTENANCE
AND EVOLUTION: RESEARCH AND
PRACTICE 2005; 17:27–64.
2

3. Outline
1. Introduction
2. The DART process
3. Design of DART
– 3.1. GUI Representation
– 3.2 Modules of DART
4. Experiments
5. Conclusions
3

4. 1. Introduction
Today’s competitive software development
market demands that several developers,
who are perhaps geographically
distributed, work simultaneously on large
parts of the code during maintenance.
Rapid quality assurance feedback
– nightly/daily building and smoke testing.
– To build the product and test it every day
4

5. DART
(Daily Automated Regression Tester)
A new framework that address the needs of re-
testing frequent builds of GUI software.
Automation!!
– structural GUI analysis
– test-case generation
– test-oracle creation
– code instrumentation to
test execution
coverage evaluation,
regeneration of test cases
Test cases re-execution.
5

6. 2. The DART Process
6

7. 7

8. Roles of the developer/tester and DART
Maintenance
cycle
8

9. M
M’
9

10. 3. Design of DART
 Driving philosophy behind the design
1. Automated
2. Efficient
3. Robust
4. Portable
5. General
10

11. Runs all
test cases
Expected
results
How much
testing
“glue”
Coverage
information Reverse
Engineering
Test Cases
11

12. 3.1 GUI Representation
A formal model of the AUT’s GUI
GUI Representation
1 Objects & Properties
2 Events 3 Components
4 Event-flow Graphs 5 Integration Tree
12

13. 3.1.1 Objects and properties
objects O = {o1, o2, . . . ,om}
properties P = {p1, p2, . . . ,pl }
A GUI = O + P
The state of a GUI = a set P at time t.
valid initial state set = iff the GUI may be in
any state Si ∈ SI when it is first invoked.
13

14. 3.1.2 Events
The events E = {e1, e2, . . . , en} associated
with a GUI are functions from one state to
another state of the GUI.
legal event sequence = e1;e2;e3; . . . ; en
14

15. 3.1.3 Component
A GUI component C is an ordered pair (RF, UF)
– A modal window + a set of modeless windows
RF represents a modal window in terms of its
events.
UF is a set whose elements represent modeless
windows also in terms of their events.
Each element of UF is invoked either by an
event in UF or RF.
15

16. 3.1.4 Event-flow graphs
All possible interactions among the
events in a component.
An event-flow graph for a component C is
a 4-tuple <V, E, B, I> where:
– V : all the events in the component
– E ⊆ V × V : directed edges
– B ⊆ V : available to the user when the
component is first invoked
– I ⊆ V : invoke other components.
16

17. EFG for part of MS WordPad.
“main” component
17

18. 3.1.5 Integration Tree (IT)
To identify interactions among
components.
Invocations
– Component Cx invokes component Cy iff
Cx contains an event ex that invokes Cy .
18

19. Definition of IT
An IT is a triple <N, R, B>
– N : components in the GUI.
– R ∈ N: a designated component called the
Main component.
– B : directed edges , the invokes relation
between components
19

20. IT for part of MS WordPad.
20

21. 3.1.6 Event Classification
1. Restricted-focus events
2. Unrestricted-focus events
3. Termination events
4. Menu-open events
5. System-interaction events
21

22. 1 Restricted-focus event
open modal windows
22

23. 2 Unrestricted-focus events
open modeless windows
23

24. 3 Termination events
close modal windows
24

25. 4 Menu-open events
Open menus
25

26. 5 System-interaction events
Interact with the underlying software
to perform some action
26

27. 3.2 Modules of DART
2
1 Test Case Generator
3
GUI Ripper Test Oracle Generator
GUI Representation
4 6
5 Test Executor
Coverage Evaluator
Code/Event Instrumenter
27

28. 3.2.1 GUI Ripper
Step1: GUI traversal and extraction (DFS)
Step2: Manual inspection by Tester
Step3: Generating the event-flow graph and
Integration Tree
28

29. 3.2.2 Test-case generator
A GUI test case T is a pair <S0, e1; e2; . . . ;
e n>
Generate test case by traversing the
event-flow graphs
29

30. 3.2.3 Test-oracle generator
Test oracles are used to determine
whether or not the software executed
correctly during testing.
<S0, e1; e2; . . . ; en>  S1;S2;…;Sn
expected state or not
30

31. Approach: execution extraction
A test case is executed on an existing,
presumably correct version of the software
and its state is extracted and stored as
oracle information.
31

32. Level of Oracle Information
(LOI)
Complete (LO1)
– {(w, p, o), ∀w ∈ Windows, ∀o = objects ∈ w, ∀p =
properties ∈ o}
Complete visible (LO2)
– {(w, p, o), ∀(w ∈ Windows)&(w is visible), ∀o = objects
∈ w, ∀p = properties ∈ o}
Active window (LO3)
– {(w, p, o), (w = active Window), ∀o = objects ∈ w, ∀p =
properties ∈ o},
Widget (LO4)
– {(w, p, o), (w = active Window), o = current object, ∀p
= properties ∈ o}
32

33. 3.2.4 Coverage evaluator
Code coverage and event coverage
Event-based coverage criteria
– Event coverage (length-1)
– Event-interaction coverage (length-2)
– Length-n event-sequence coverage
33

34. 3.2.5 Event instrumenter
All event sequences that are executed on
the GUI be collected.
The key Idea: To detect the existing
listeners and attach our own listeners.
34

35. 35

36. 3.2.6 Test executor
Executing an entire test suite
automatically on the AUT.
– perform all the events in each test case
What properties should be compared?
– level of testing (LOT1 - LOT4)
– The test designer may choose to employ
partial oracle information.
36

37. Possibilities available to the test designer for level
of detail of oracle information.
10 different combinations
37

38. 4. Experiments
The subject applications: TerpOffice
http://www.cs.umd.edu/~atif/newsite/terpoffice.htm
38

39. 39

40. 40

41. 41

42. 42

43. 43

44. 44

45. 4.1 Results
Space Requirements
Time requirements
Code coverage
45

46. Space requirements
500mb
46

47. Time requirements
All times are reported for a 2.2 GHz
Pentium 4 machine with 256 MB of RAM.
The results show that the smoke testing
process is practical, in that it can be
performed in one night.
47

48. TerpPaint
51.9 Hours
48

49. TerpPad
4.7 Hours
49

50. TerpCalc
11 Hours
50

51. TerpSpreadSheet
18.6 Hours
51

52. TerpDraw
8 Hours
52

53. TerpManager
2.7 Hours
53

54. Code coverage
We instrumented the application before
running all of the smoke test cases.
We recorded the statements that were
executed for each user-implemented class
during test-case execution.
54

55. TerpPaint
55

56. TerpPad
56

57. TerpCalc
57

58. TerpSpreadSheet
58

59. TerpDraw
59

60. TerpManager
60

61. 5. Conclusions
We define a formal model of a GUI
derived from specifications that is
useful for smoke testing.
We develop a new process for re-
testing nightly builds of GUI software.
Our regression testing process
cannot only be used for nightly builds,
but also for general GUI re-testing.
61

62. In the future
We will study the effectiveness of the DART
process by analyzing the number of faults
detected.
We will also integrate DART in a higher-level
process that involves executing other types non-
GUI) of smoke tests on the software.
We will also investigate the application of DART
to other software systems that take events as
input. (e.g. Web applications)
62