Mobile developers conference MBLTDev15 November 17th, Moscow Organisers: e-Legion, RAEC

1. Mobile Testing at
SoundCloud
Brigit Lyons
brigit.lyons@soundcloud.com
MBLTDev 2015

2. title, date, 01 of 10
● Product engineers write unit tests
● Product engineers write automated UI tests for story’s
acceptance criteria
● Team performs manual QA check for every story before
it’s “Done”
● Automated tests run after every push to master/release
● Manual regression testing done before every production
release
General Approach

3. title, date, 01 of 10
● Provide tools and services that enable product engineers
to build and release high quality software efficiently
● Understand team’s pain points and provide solutions
Software Engineer in Test

4. title, date, 01 of 10
Continuous Integration

5. title, date, 01 of 10
UI Tests

6. title, date, 01 of 10
Models a page in application that tests interact with in order to:
● Reduce duplicated code
● Limit code refactoring after UI changes
● Ensure tests only care about the “services” they interact with and not
implementation details
The only thing that has a deep knowledge of the structure of the page (or parts of
a page)
PageObject

7. title, date, 01 of 10
Example

8. title, date, 01 of 10
● Parallelization vs Fragmentation
● 34 Nexus 5 + 2 emulators
● 22 Mac Minis 44 iOS simulators
UI Tests Are Run on Devices

9. title, date, 01 of 10
Why Parallelize?

10. title, date, 01 of 10
1 hour 42 minutes 19 seconds

11. title, date, 01 of 10
2 hours 1 minute 23 seconds

12. title, date, 01 of 10

13. title, date, 01 of 10
● Ruby gems
● Android: Rubotium https://github.com/ssmiech/rubotium
● iOS: Distributed Cucumber and Cmdr
Custom Test Runners

14. title, date, 01 of 10
● Given application and test packages, create a list of tests
● Determine all available devices
● Run a test on an available device, repeat
Custom Test Runners

15. title, date, 01 of 10
10 minutes 4 seconds

16. title, date, 01 of 10
8 minutes 11 seconds

17. title, date, 01 of 10
● Configuration options:
○ run on a specific type of device
○ run tests with a set of specified annotations
○ run tests without a set of specified annotations
● Better debugging:
○ HTML reports
○ For all failed tests:
■ Stack trace
■ Full logs
■ Screenshots -> videos
Custom Test Runner Benefits

18. title, date, 01 of 10
● No built-in scheduling or resource management
○ Must rely on Jenkins
● Only CI jobs can run tests in this environment
● We have a few configurations that run on CI (master,
release, development branches, etc)
● CI jobs that use same resources for UI tests must block
one another, leading to inefficient wait time
Custom Test Runner Problems

19. title, date, 01 of 10
● Run configurations are set within Jenkins job
● New configurations or one-off configurations require
setting up a new Jenkins job
● Pre-commit testing is not feasible
Custom Test Runner Problems

20. title, date, 01 of 10
● Test running order cannot be specified
● Test results are only collected from devices after all tests
have been completed
● Failed tests are retried a second time, but this is not
visible in the reports
Custom Test Runner Problems

21. title, date, 01 of 10
Mobile Test Runner Service!

22. title, date, 01 of 10
● Android only for now, but iOS support coming soon
● Mostly written in Scala
● Communicates with devices through ADB using ddmlib
● REST API for clients
Mobile Test Runner Service

23. title, date, 01 of 10
Mobile Test Runner Service

24. title, date, 01 of 10
● Resource management is controlled by the service itself
○ Don’t have to rely on Jenkins
● Test runs can be executed in parallel
● Resources are used efficiently; device workers keep
executing tests as long as there are tests in the queue
● Developers can test pre-commit changes on same
environment used by CI
Results

25. title, date, 01 of 10
● Test queue ordering is also controlled by the service
itself, so we can run the most important tests first (test
age, flakiness, etc)
Results

26. title, date, 01 of 10
● Test results (stack trace, full log, and videos for failed
tests) are available as soon as each individual test
finishes
● Retries for failed tests clearly indicated in HTML report
● Data about devices, tests, and results are stored in a DB,
making statistical analysis easier
Results

27. title, date, 01 of 10

28. title, date, 01 of 10
● Test runs are easily configured via REST API
Results

29. title, date, 01 of 10
● Built a Java client library
● Integrate into Gradle tasks so test configuration for CI is
part of the Android codebase
But API Endpoints can be annoying to
remember...

30. title, date, 01 of 10
● Not easily parameterized
● Names can be annoying to to remember
● Command line is not the most user friendly interface
And Gradle tasks have problems, too...

31. title, date, 01 of 10
● Developers can run tests for local changes on CI
environment right from Android Studio
● Right click to run specific test method, test class, test
package
● Results are reported natively, including links to videos
and logs for failed tests
Solution: IntelliJ Plugin

32. title, date, 01 of 10

33. title, date, 01 of 10

34. Thanks!
Brigit Lyons
brigit.lyons@soundcloud.com