Slides from my talk at Selenium Conference 2016.
In this talk you will get ideas about how you can instrument test result information to provide actionable data, paving the way for more robust, reliable and timely test results.
By capturing this information over time, and when combined with visualization tools, we can answer different questions than with existing solutions (Allure / CI tool build history). Some examples of these are:
Which tests are consistently flaky
What are the common causes of failure across tests
Which tests consistently take a long time to run
Using this information we can move away from the ‘re-run’ culture and better support continuous integration goals of having quick, reliable, deterministic tests
Video of the talk is here: https://youtu.be/29fPYx7OJnE?list=PL_7kBU2XBlbKuRNVHeqjXUygXtToqMHsn
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Test Trend Analysis : Towards robust, reliable and timely tests
1. @hughleo01 Liberty Information Technology
Liberty IT, Belfast and Dublin
Test trend analysis:Towards robust,
reliable and timely tests
Hugh McCamphill
2. @hughleo01 Liberty Information Technology
2Liberty IT @hughleo01
Indianapolis Dover Belfast GS LIU DCs Europe RDC Russia DRDC
Redmond DC
Global DRDC
& Utilities
Kansas City DC
Latin America RDC
GS LIU DRDC
Portsmouth DC
GS LIU DC
Turkey DCs India DCs Asia RDC China DCs
Wausau
Fairfield
Boston
Seattle Blanchardstown
11 major IT offices
7 country data centers (DCs)
DRDC = Disaster Recovery DC
3 US domestic data centers (DCs)
4 international regional data centers
LIU = Liberty International Underwriters
More than 5,000+ IT
employees around the globe
Australia
About Us
3. @hughleo01 Liberty Information Technology
3Liberty IT @hughleo01
Principal Software Engineer in Test
Belfast Selenium Meetup Organiser
Writing Automation Since 2004
WebDriver user since 2011
4. @hughleo01 Liberty Information Technology
4Liberty IT @hughleo01
Which tests are slow
Which steps are slow
By Martin Lewison from Forest Hills, NY, U.S.A. - Cedar Point and Oberlin's Commencement Uploaded by
Astros4477, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=27339553
5. @hughleo01 Liberty Information Technology
5Liberty IT @hughleo01
Tumbling Dice : Decorated Anti-tank Blocks near Harkess Rocks, Bamburgh, Northumberland
Image Copyright Richard West. This work is licensed under the Creative Commons Attribution-Share Alike
2.0 Generic Licence
Which tests are intermittently passing
What are common failures across tests
6. @hughleo01 Liberty Information Technology
6Liberty IT @hughleo01
“Create commons cute robot mural, Toronto"
by https://www.flickr.com/photos/margonaut/ is licensed under CC BY 2.0
Where are the issues?
13. @hughleo01 Liberty Information Technology
13Liberty IT @hughleo01
A time series database is a
software system that is optimized
for handling time series data,
arrays of numbers indexed
by time
Elasticsearch is a tool for storing, searching, and
analyzing structured and unstructured data
14. @hughleo01 Liberty Information Technology
14Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
15. @hughleo01 Liberty Information Technology
15Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
16. @hughleo01 Liberty Information Technology
16Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
17. @hughleo01 Liberty Information Technology
17Liberty IT @hughleo01
Setting up a mapping
{"mappings":{"result":{
"properties":{
"project":{"type":"string"},
"environment":{"type":"string"},
"duration":{"type":"double"},
"name":{"type":"string","index":"not_analyzed"},
"classname":{"type":"string","index":"not_analyzed"},
"message":{"type":"string","index":"not_analyzed"},
"status":{"type":"string"},
"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}}
curl -XPUT “http://localhost:9200/test-result” –
d @mapping_schema_framework.json
18. @hughleo01 Liberty Information Technology
18Liberty IT @hughleo01
{"index":{"_index":"test-result","_type":"functional" }}
{"project":“TestProject",
"environment":"QA",
"classname":“TestProject.Foo.Bar",
"name":"ShouldBeAbleToAddItem",
"duration":"47.12",
"created":"2016-09-14 21:35:53",
"status":"Passed"}
curl -XPOST “http://localhost:9200/test-
result/_bulk?pretty” --data-binary @testResults.json
Posting of results
19. @hughleo01 Liberty Information Technology
19Liberty IT @hughleo01
A Simple Process Flow
RunTests Results
Post
Results Visualize
20. @hughleo01 Liberty Information Technology
20Liberty IT @hughleo01
Kibana is an open source data visualization plugin for ElasticSearch
25. @hughleo01 Liberty Information Technology
25Liberty IT @hughleo01
Splitting error messages into buckets of tests affected by that error
26. @hughleo01 Liberty Information Technology
26Liberty IT @hughleo01
Error Messages
Individual Tests
Splitting error messages into buckets of tests affected by that error
34. @hughleo01 Liberty Information Technology
34Liberty IT @hughleo01
References
http://martinfowler.com/articles/nonDeterminism.htm
https://watirmelon.blog/2015/11/11/your-tests-arent-flaky/
35. @hughleo01 Liberty Information Technology
35Liberty IT @hughleo01
Thank you
Hugh McCamphill
@hughleo01
Run Tests Results
Post
Results Visualize
In Summary