Доклад Andrejs Kalnacs на SQA Days-22. 17-18 ноября 2017. Санкт-Петербург, Россия www.sqadays.com

1. Loading time testing and results visualisation
of web games
Andrejs Kalnačs

2. who am i
2
Lead Software Developer in Test @ Evolution Gaming
10 years of industry experience in QA
Many different roles
Wide business domains experience
Last 6 years – testing automation
Java, Groovy, Bash, Perl, Tcl
Evolution Gaming, Nokia, Accenture, Nokia Siemens
Networks
Riga Testing Automation Club meet-ups co-organiser
English, Russian and Latvian language proficiency @TesterFromRiga

3. Evolution Gaming
3
The World leader in live dealer gaming
3 development locations – Riga, Tallinn, Amsterdam
B2B product company

4. Agenda
4
Web based
loading time
testing
Practical guide
in test data
points
visualisation

5. Use case of Evolution Gaming
5

6. Use case of Evolution Gaming
6
16 different games
3 test environments
2 browser cache states

7. Creation and execution of web based loading
time testing

8. Browser automation using WebDriver API
8

9. Custom metric collection from client
9
Loading time
Intermediate loading metrics
Component/service loading metrics
Proprietary API

10. Interception & archiving of HTTP transactions
10
Proxy
System under test
HTTP, HTTPS, WebSocket

11. HTTP archive format
11
JSON
Libraries
Many viewers
Storage solutions
Diff solutions
http://har.tech

12. Chrome 62
12
https://developers.google.com/web/updates/2017/08/devtools-release-notes#har-imports

13. OWASP ZAP proxy
13
HTTPS support
WebSocket support
API
Open Source
Cross platform (Linux, Mac OS X, Windows)

14. Bandwidth throttling
14
Virtual or physical machine with two network interfaces
Linux
tc
tcconfig
tcset --device eth1 --rate 800k --direction outgoing
tcset --device eth0 --rate 800k --direction incoming

15. Cloud?
15
Virtual hardware for desktop browsers
Traffic throating only at Sauce Labs & AWS
No HAR produced

16. Local desktop & device farm
16
Support, maintenance & update of infrastructure, hardware & software
Stability of devices
Complexity of iOS testing automation

17. Execution from CI server
17
Selenium Hub
Appium iOS node
Appium Android node
Proxy Bandwidth
throttler
Docker host
Docker host

18. Tests results
18
+
Information provided by API
Calculated & aggregated metrics from HAR

19. Test data storage
19
Jenkins (job artefacts)
Attachments to report
File storage (FTP, SAMBA, WebDAV, or anything else)
Dedicated .har storage (harstorage)
SQL or NoSQL database

20. Results visualization

21. Why?
21
Show stack holders how good or bad user experience is
Compare your products
Visualise regression
Understand how new feature is affecting performance

22. Ops & finance tools
22

23. Use-case of performance tests results
23

24. 24
Prometheus
Prometheus Push gateway Test job
pushget
(interval based)

25. 25

26. 26

27. 27
Templating
SCREENSHOT TO BE ADDED

28. 28
Annotations

29. 29
Alerts

30. simplest-test-dashboard
30
https://github.com/andrejska/simplest-test-dashboard
Saves web page with content to disc and calculates it size

31. Grafana dashboard version control (git)
31
Edit dashboard Copy in IDE
Diff & throw
out unneeded
things
Push to branch Merge request
Merge to
master
Grafana
dashboard
update via API

32. Key takeaways
32
Selenium can be
used for
performance tests
Application it self
can provide a lot of
insides for testers
Visualisation is
excellent for displaying
results in interactive,
understandable and
easily comparable way
Mastering data
visualisation
tools is not a
rocket science

33. Thank you!
Any questions?
andrejska
Andrejs Kalnačs
@TesterFromRiga