Доклад Ильина Александра на SQA Days 7

1. <Insert Picture Here>
UI Test Automation Effectiveness
Alexandre (Shura) Iline
Java SE and JavaFX Quality architect.

2. The following is intended to outline our general product
direction. It is intended for information purposes, and
may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions.
The development, release, and timing of any features
or functionality described for Oracle's products remains
at the sole discretion of Oracle.

3. This presentation shares experience gotten
from testing such products as
JavaFX Authoring Tool

4. What's effectiveness
• Regular functional testing
• Saving human time
• Special groups of tests
(Things that could only be done through automated testing)
• Doing it with less human time

5. UI testing … by Wikipedia
«GUI software testing is the
process of testing a product
that uses a graphical user
interface, to ensure it meets its
written specifications.»

6. UI testing … most often ...
«Checking whether usage of a
product UI leads to results
expected by the the person
who performs testing»

7. Sample test scenario
●Start text editor
●Push «File/Open»
●Verify file chooser directory
●Select some file
●Verify editor area content
●Verify application title
●Verify buttons availabilities
●
....

8. Test automation ... for me
«Building a process which
exercises and reports certain
product characteristics while
run unattended.»

9. Test automation is like development
• Putting logic in a code
• Same lifecycle:
• Requirements
• Design
• Implementation
• Same set of problems
• Bugs
• Instabilities
• Scope
• defined through test specification/plan

10. Test automation is not like
development
• Big fat dependency – the tested product
• vs many libraries and platform
• Many small programs
• vs one big program
• Does one thing good – reports status
• vs does many thing ... good
• Perfectness is not the goal
• other than maintenance cost, ease of use

11. UI Test automation
• Avoidable
• Testing could be done with no automation
• Less obvious
• Many misconceptions
• More expensive
• Requires specific tools
• Such as the test drivers

12. Misconceptions
• Automated testing replaces manual testing
• it will find all the bugs
• it will find more bugs
• it does the same amount of work
• Create once use forever
• This is easy
• This is too hard
• No need to create test plan
• “Will just buy the XXX tool and it will solve all out
ptoblems”

13. UI changes
• “Improvements”
• Specification/requirement changes
• Bug fixes
If UI is not changing ...
Why bother testing?

14. Test base characteristics
• Coverage
• How well the test base tests the product?
• Initial creation cost
• How expensive it is to initially implement it?
• Sustainability
• How expensive it is to support it?
• Reliability
• Does it have intermittent failures?

15. Coverage
• Implementation coverage
• Line, block, condition, sequence, ...
• Specification coverage
• How well the tests covering the functional specification
• Public API coverage
• Whether the public API is tested fully
• UI coverage
• Whether the tests cover UI fully
• Combine with bug density
• First cover the areas with more bugs
• Combine with code complexity
• First cover the more complicated code.

16. Initial creation
• Record && replay
• Inexpensive in creation
• Lacks functionality
• (More) expensive to support
• Disables effective UI test automation
• Coding
• Expensive to create
• Require experienced engineers
• Encourages effective UI testing
• (Much) Less expensive to support

17. Tests code needed to be changed
• Test code needs to be updated every time UI
changes
colorComboBox.selectColor(“Grey”)
turns into
colorButton.push();
...
colorChooser.selectColor(Color.GREY);

18. Application UI
Product UI
Product UI
18
18

19. Coordinates
• click(134,32) //selects some record
• click(215,122) //hits “Properties”
• sleep(5) //sleeps to let dialog be painted
• click(64,182) //expands color combo
• click(235,182) //selects Gray
• click(235,212) //hit OK
19
19

20. Widgets
• Find “Car records” frame
• Find table
• Select “1abc234” cell
• Push “Properties” button
• Wait for “1abc234” dialog
• Select “Gray” color in combo box
• Push “OK”
20
20

21. Widgets or coordinates
Car record
Domain
Domain
model
model
Model
Model Make VIN
VIN Color Year License plate
Make Color Year License plate
Product UI
Product UI
Test
21
21

22. UI Primitives
• Find car list frame
CarListFrame list = new CarListFrame()
• Open properties dialog for car “1abc234”
CarDialog propDialog =
list.carProperties(“1abc234”);
• Set color to gray
propDialog.setColor(Color.GRAY);
• Apply changes
propDialog.ok();
22
22

23. Library
Domain
Car record
Domain
Model Make VIN Color Year License plate
model
Model Make VIN Color Year License plate
model
Product UI
Product UI
CarListFrame Test library
Test library CarDialog
Test
23
23

24. Domain model
• Set color to gray for a car “1abc234”
new CarRecord(“1abc234”).
setColor(Color.GRAY);
Underneath the cover, CarRecord class does all
described earlier
24
24

25. Domain library
Domain
Car record
Domain
model
Model Make VIN Color Year License plate
model
Model Make VIN Color Year License plate
Product
Product
UI
UI
CarList UI test library
UI test library CarDialog
CarRecord Domain test library
Domain test library
Test
25
25

26. The formula?
TM * NR * NC
EA =
TD + TS * NR * NC
EA – automation effectiveness
To be used for every particular product.
NR and NC are unique for a product.
TM is a characteristic of a test suite.
Smaller TD and TS - higher the EA.
Coefficient depend on the way you write your t

27. Td and Ts together
8
7.5
7 Td/Tm
6
Ts/Tm
5
5
4
3
3
2
1.1 1
1
0.5
0.1 0.05
0
Coordinates Widgets UI Library Domain library

28. TD and TS for NC=3, NR=8, TM=1
30
25.1 Td
25 24
Ts
Td+(Ts*Nc*Nr)
20
15
15
12
10 8.7
7.4 7.5
5
5
3 2.4
1.1 1.2
0
Coordinates Widgets UI Library Domain library

29. 3.5
EA for NC=3, NR=8 3.24
3
2.76
2.5
2
1.6
1.5
0.96
1
0.5
0
Coordinates Widgets UI Library Domain library
Ea

30. Other Ts improvements
• Store resources in a property file
• Take from the application itself

31. Testbase reliability
• Select right tool
• No sleeps – only waitings
• Use event queue
• Right lookup criteria
• By ID
• By text
• By location
• Not by index

32. Special test types
• Continuous build tests
• aka sanity aka acceptance aka smoke
• Pre-integration
• Performance
32
32

33. Continuous build
Build
Executed
Code automatically Success
Commit
changes after commit
Yes.
No
Analysis.
Rollback!!!
Development
Promote
Test further Code is compilable

34. Continuous build with testing
Code Build Success
changes Commit
No
Rollback!!! Yes.
= Compilation
Test Analysis. successful
changes No
Testing
Test further Passed
Build is good Yes Is it working?
Code line is healthy
Go on ...

35. Pre-integration
Code
changes
Yes
Testing Passed Commit
Test
changes No

36. How to benefit from UI automation?
Use it!
• Sanity
• Pre-integration
• Attach to bug reports
• Make is a quality criteria
• Run it for every build
• Show it to the boss :)
• Don't forget to show it to the dev. team

37. <Insert Picture Here>
SQE reporting infrastructure improvements
Alexandre (Shura) Iline
Java and JavaFX Quality architect.
shura@sun.com