This's a Sikuli tutorial I gave to the Megason Lab at Harvard Medical School.

1. PRACTICAL SIKULI
USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING
Tsung-Hsiang (Sean) Chang
MIT CSAIL
Collaborators: Tom Yeh, Rob Miller
1

2. WHAT IS SIKULI?
2

3. WHAT IS SIKULI?
2

4. 2
3
1
3

5. Low-level API?
Mac’s system design?
4

6. EMPTY THE TRASH CAN
5

7. OUTLINE
• What Sikuli is
• How Sikuli works
• Sikuli for GUI automation
• Sikuli for GUI testing
• Conclusions
6

8. How Sikuli Works
7

9. TEMPLATE MATCHING
click(
)
8

10. TEMPLATE MATCHING
click(
)
8

11. SYSTEM DESIGN
9

12. DEMO 1:
EMPTY TRASH CAN
2
3
1
10

13. REGION AND MATCH
Region
match = region.find( or “text pattern”)
11

14. REGION
• basic attributes: x, y, width, height
• Match is a Region
• Screen is also a Region
• chained methods: Screen(0).find(W).click(X)
12

15. KEY METHODS OF REGION
• find, findAll, exists, wait, waitVanish
• Mouse Actions
click, doubleClick, rightClick, hover, dragDrop
mouseDown, mouseUp
• Keyboard Actions
type(text), paste(text)
keyDown, keyUp
13

16. EX2: FACEBOOK STALKER
14

17. EX2: FACEBOOK STALKER
14

18. RESTRICT SEARCH REGION
15

19. But the Region constant is bad
16

20. APP CLASS
• firefox = App.open(“Firefox”)
• firefox.focus(), firefox.close()
• firefox.window([n])
• returns the Region of Firefox’s n-th window
17

21. FIND IN CERTAIN APP
18

22. EX3: MUTE SPEAKERS
19

23. EX3: MUTE SPEAKERS
19

24. SPATIAL OPERATORS
20

25. SPATIAL OPERATORS
20

26. SPATIAL OPERATORS (2)
21

27. TEXT MATCHING AND
RECOGNITION
• click(“OK”), find(“Cancel”)
• region.text() - read the text in a region
22

28. VISUAL-EVENT-DRIVEN
PROGRAMMING
• observes visual events in any Region
• Region.onAppear(Pattern, handler)
• Region.onVanish(Pattern, handler)
• Region.onChange(Pattern, handler)
23

29. VISUAL-EVENT-DRIVEN
PROGRAMMING
24

30. VISUAL-EVENT-DRIVEN
PROGRAMMING
24

31. VISUAL DICTIONARY
25

32. GUI
TESTING
26

33. VISUAL ASSERTION
• make sure the appearance of visual feedback
27

34. VISUAL ASSERTION
• make sure the appearance of visual feedback
27

35. JUNIT INTEGRATION
28

36. SIKULI IN COMMAND LINE
usage:
Sikuli-­‐IDE
[-­‐-­‐args
<arguments>]
[-­‐h]
[-­‐r
<sikuli-­‐file>]
[-­‐s]
[-­‐t
<sikuli-­‐test-­‐case>]
-­‐-­‐args
<arguments>
specify
the
arguments
passed
to
Jython's
sys.argv
-­‐h,-­‐-­‐help
print
this
help
message
-­‐r,-­‐-­‐run
<sikuli-­‐file>
run
.sikuli
or
.skl
file
-­‐s,-­‐-­‐stderr
print
runtime
errors
to
stderr
instead
of
popping
up
a
message
box
-­‐t,-­‐-­‐test
<sikuli-­‐test-­‐case>
run
.sikuli
as
a
unit
test
case
with
junit's
text
UI
runner
29

37. CONCLUSIONS
30

38. PLATFORM INDEPENDENCE
• Works
on ANY GUI that can be displayed on
Windows/Linux/Mac
• Virtual machines
• Remote desktop
• Mobile simulators: Android, iPhone
• Web: Flash, HTML+Javascript
31

39. PROGRAM AGAINST UI
Sikuli programs are written against the user interface
instead of an API
UI: visible, familiar, always exists
API: faster, probably more stable
32

40. FUTURE WORK
• fast and accurate OCR on screen
• Accessibility API integration
33

41. Thanks!
http://sikuli.org
or, just google “sikuli”
New Sikuli X 1.0-rc2 released.
Follow us on twitter @sikuli
Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)
34