James Farrier founded Appsurify to address test automation challenges, particularly unreliable or "flaky" tests that fail unexpectedly. The document discusses various causes of flaky tests and strategies for handling them, including manually or automatically removing flaky tests from runs, fixing the tests, rerunning tests, using bots to quarantine flaky results, prioritizing relevant tests for changes, and employing classification algorithms to detect flaky tests. The key challenges are scaling the efforts while still allowing tests to run and find defects.

1. 1
James Farrier
From - Auckland New Zealand
Studied - Software Engineering
Been working in Test Automation for 12 years
Recently founded Appsurify
About Me

2. With less time
(thanks Agile!)
we need to
improve our
testing efficiency

3. 3
Test Automation Challenges
Tests Are
Unreliable
Failures
Are
Expected
Unreliable
Tests Are
Hard To Fix
Tests Stop
Being Run
Tests Take
Too Long
To Run

4. What causes
unreliable
automation?
● Async waits: Thread.sleep
● Concurrency: data races, atomicity violations, or
deadlocks
● Test order dependency
● Resource leak
● Network
● Time
● I/O
● Randomness
● Floating point operations
● Unordered collections
● Device losing connectivity or power
● Browser crash
● PC or phone downloading update
● 3rd party systems
● Caching
● Setup/clean-up/teardown issues
● Elements in motion

5. 74% of test
failures at Google
are caused by
Flaky Tests

6. Almost 16% of our tests have some level of flakiness
associated with them!
John Micco, Google
“

7. How can we
handle flaky
tests?

8. How can we
handle flaky
tests?
● Use ID’s
● https://github.com/testdotai
● Mocks
● Hermetic, can run in isolation
● WaitUntil, Fluent or Explicit Waits,
not Thread.Sleep
● Use frameworks
● But process is more important
because you can’t fix them all

9. Manually remove
flaky tests from
test runs

10. Manually remove
flaky tests from
test runs
Advantages Disadvantages
● Easy to get started
● Tests can continue to
be run, potentially
finding bugs
● Stops flaky tests
breaking the build
● Difficult to scale
● Can lose track of tests
● Requires setting up
multiple test runs

11. Delete flaky tests

12. Delete flaky tests
Advantages Disadvantages
● Easy
● Scales
● Stops flaky tests
breaking the build
● Misses defects
● Eventually recreate
the same tests
● Disheartening

13. Fix flaky tests

14. Fix flaky tests
Advantages Disadvantages
● Tests continue to run
and find bugs
● Leads to better
practices creating
tests
● Stops flaky tests
breaking the build
● Difficult to scale
● Difficult to reproduce
● Difficult to fix
● May not be caused by
tests themselves (i.e.
third party)
● May not be possible
● Tests need to be
removed while being
fixed, bugs maybe
missed
● Should be combined
with another process

15. 99.9% isn’t reliable enough! If your build contains
1000 tests which are 99.9% reliable your build will
break 73% of the time
“

16. Bots for flaky tests

17. Bots for flaky tests

18. Bots for flaky tests
Advantages Disadvantages
● No manual
intervention required
● Scales
● Stops flaky tests
breaking the build
● Reasonably
technical/difficult
setup
● May miss defects
because of tests not
being run

19. Results from - Bots
for flaky tests

20. Results from - Bots
for flaky tests

21. What if all the
tests are flaky?

22. Flaky tests are still
valuable and can
still find new
defects

23. Rerun flaky tests

24. Rerun flaky tests
Advantages Disadvantages
● No manual
intervention required
● Scales
● Tests continue to run
and find bugs
● Stops flaky tests
breaking the build
● Reasonably
technical/difficult
setup
● How many times
should you rerun a
test?
● Can greatly increase
test execution time,
when you need this to
be as short as possible

25. Quarantine failure
reasons for flaky
tests

26. Results from -
Quarantine failure
reasons for flaky
tests

27. Quarantine failure
reasons for flaky
tests
Advantages Disadvantages
● Tests continue to run
and find bugs
● Scales
● Stops flaky tests
breaking the build
● Reasonably
technical/difficult
setup
● Failures need to be
marked as flaky
● Flaky tests still
initially fail the build

28. Using a
classification
algorithm to
detect flaky tests
and then
quarantine the
flaky results

29. Which is likely to
be Flaky? Failure
early in the test or
at late?

30. Some of the most
common failures
with Selenium
occur early i.e.
Browser setup

31. Which is likely to
be Flaky? Error or
Assert Failure?

32. Which is likely to
be Flaky? If we
change Transfers
section a
Transfers test that
fails or a Payment
test that fails?

33. Which is likely to
be Flaky? New
Failure or an error
that’s caused
flaky failures
before?

34. Which is likely to
be Flaky? A test
that has never
flaked before or a
test that looks like
Xmas lights?

35. Risky change is
more likely to
have real defects
than flaky failures

36. How do we know
if it’s a risky
change? What is
the change and
who completed
it?

37. Sometimes devs
even tell you
when something
is likely to have a
bug

38. Classification
algorithm for flaky
tests
Advantages Disadvantages
● Tests continue to run
and find bugs
● No manual
intervention required
● Scales
● Stops flaky tests
breaking the build
● Reasonably
technical/difficult
setup, unless you use
Appsurify!
● To learn failures need
to be initially marked
as flaky

39. Run a prioritized
set of tests
targeted at the
change

40. 99.9% is reliable enough!?!? If your build contains
1000 tests which are 99.9% reliable, if you only run 50
for a change then your build will break only 5% of the
time, compared to 74% if you run all the tests.
“

41. Run a prioritized
set of tests
targeted at the
change
Advantages Disadvantages
● Tests continue to run
and find bugs
● No manual
intervention required
● Scales
● Faster run time
● Stops flaky tests
breaking the build
● Reasonably
technical/difficult
setup, unless you use
Appsurify!
● Needs to be combined
with another
technique for
handling flaky failures

42. james@appsurify.com
Thanks for Listening