With more and more applications competing for users attention it is getting increasingly important to ensure that your app works on all Android devices as it should. If you launch your app without proper testing you risk wasting your marketing and development efforts by getting bad ratings and having your users disappointed just because your app does not work on their device. In his presentation Jouko Kaasila from Bitbar goes through the most common approaches to tackle the testing challenge from application developer's point of view and explain why testing for device compatibility is a must for any serious Android application developer. He will also give examples of most common urban myths about testing and go through some best practices to minimize the cost and effort related to setting up your test infrastructure. Jouko Kaasila is a co-founder of Bitbar, the company behind Testdroid Android testing tools. He has more than 14 years of experience in mobile software from early mobile services through Smartphone operating systems to Android platform. He is a serial entrepreneur with strong passion to build and help others to build businesses around mobile technologies and he is always keen to exchange views and experiences in this field.

1. PRESENTATION OVERVIEW
− Why to test at all?
− Richness of Android devices – a real opportunity
− Android success story in numbers
− Urban myths about mobile testing
− Options for professional grade testing
− Typical application implementations pitfalls
− Some automated testing frameworks for Android
1
info@bitbar.com

2. WHY TO TEST AT ALL?
 Beta test approach  Systematic testing
Pre launch activities: Light manual testing for Testing criteria: Some level of basic
catching most obvious bugs. Usually done on acceptance criteria needs to be created: What
one or max 2-3 devices. devices are supported, what use cases are
important, what is the expected performance
Beta phase: App is distributed to 10-100 beta
users who will report bugs and any other issues Setting up test automation: Test cases are
converted to test scripts, that are executed
Launch: Based on the feedback from beta every time when the app changes
testing a new version is created and launched to
the app stores. Testing throughout the development cycle:
Testing starts when first features are
implemented and continues even when the app
is released
2

3. WHY TO TEST AT ALL?
 Some quotes:
Summary: Developers need to avoid treating
end-users as beta testers and conduct pre-
launch app testing to avoid critical reviews due
to poor usability once released to market
“If there is a major bug that is not caught before
the launch, then a lot of expensive development
and marketing work is quickly wasted”
"The first wave of users gives the app bad
ratings, which then undermine [the number of]
downloads and the app would [eventually] sink
into oblivion."
3 Source: http://www.zdnet.com/mobile-app-testing-imperative-but-not-foolproof-
7000000787/

4. WHY TO TEST AT ALL?
 2012 Cap Gemini Quality
Report (Mobile Testing)
“Mobile design and development fits nicely
with iterative development. Looping around
the cycle of requirements, design, build and
test in smaller chucks means that testing
need not simply be a cost “tacked-on” to the
end of a project. There are some types of
such as penetration and security testing that
should only be done towards the end of the
project, but for functionality, performance and
ease-of-use testing we recommend that
this be done iteratively during the mobile
project.”
4 Source: http://www.capgemini.com/technology-blog/2012/09/quality-report-2012-
mobile-testing/

5. WHY TO TEST AT ALL?
 2012 Cap Gemini Quality
Report (Mobile Testing)
“Notable from the survey is that
18% of responding companies
say that they do not have enough
time to test mobile apps, and
65% do not have the right tools.
Also 52% cite lack of devices as
a reason not to do mobile testing”
5 Source: http://www.capgemini.com/technology-blog/2012/09/quality-report-2012-
mobile-testing/

6. THE RICHNESS OF ANDROID ECOSYSTEM
 Fragmentation is innovation
Samsung
Galaxy S2
< 10% market share
Graphical representation of fragmentation in the Android ecosystem. Each rectangle represents the market share of one
6
Android device. Samsung Galaxy S2, the most successful Android to date, has less than 10% market share.
Source: Open Signal Maps, May 2012

7. ANDROID SUCCESS STORY IN NUMBERS
− Over 500m Android
Q2/2012 Q2/2012 Q2/2011 Q2/2011 devices shipped since
OS shipments market shipments market YoY change
share share Android 1.0
− About 220m iOS devices
Android 104.8 68.1% 50.8 46.9% 106.5% have been shipped since
2007
iOS 26 16.9% 20.4 18.8% 27.5%
Blackberry 7.4 4.8% 12.5 11.5% -40.9%
Symbian 6.8 4.4% 18.3 16.9% -62.9%
Windows 5.4 3.5% 2.5 2.3% 115.3%
7
info@bitbar.com Source: IDC, 8.8.2012

8. URBAN MYTHS ABOUT ANDROID TESTING
Myth #1: Testing on emulators is enough
This app works perfectly on
Android 2.3 and 4.0 emulators
but on some of the most
popular devices looks like
this:
8
info@bitbar.com

9. URBAN MYTHS ABOUT ANDROID TESTING
Myth #2: Testing on 5-8 most common devices is
enough
Yes, if reaching every every
fourth Android user is ok for
your CEO:
9
info@bitbar.com

10. URBAN MYTHS ABOUT ANDROID TESTING
Myth #3: Exploratory testing just before launch is
enough
 Exploratory testing is not very good for catching any regressions
 Exploratory testing is not very repeatable
 Exploratory testing (any manual testing) does not scale very well
 However, exploratory testing has its own merits – it is not enough just by itself
10
info@bitbar.com

11. URBAN MYTHS ABOUT ANDROID TESTING
Myth #4: Users will understand if there are some
issues
 Application not working and having bugs is #1 reason for uninstalling the app
 Users don’t update their apps very frequently
 Quality issues are #1 reason for bad reviews in Google Play (about 80%),
Application content related issues are only (20%)
11
info@bitbar.com

12. TYPICAL APPLICATION IMPLEMENTATION
PITFALLS
 Mix different technologies and frameworks
 Extensive use of web views
− Webkits differ from one device to another
− All sorts of timeout due to most content coming from server
− Issues in http headers across devices
− CSS&Javascript cause their own problems
 Use cross platform tools
− Especially the ones that generate the code on server side are very difficult to debug
− Wrapping HTML5 to an app is quite ok
− The time you save on development, you loose on debugging
12
info@bitbar.com

13. HOW TO SET UP A PROFESSIONAL GRADE
TESTING
 Set up Continuous Integration and version control
 Automate most common use cases
− Address the basics first
− Establish a repeatable baseline
− Keep adding new tests as your functionality grows
− Do not try to automate everything – there is still time and place for manual checks as well
− Maintain your test library as your code evolves (store both in version control)
 Define your target devices:
− Depending your ambition level anything from 20 to 50 will do.
− Test automatically and often with these devices
− To get a reference point run occasionally tests with all possible devices (especially before any
release)
13
info@bitbar.com

14. TESTDROID PRODUCTS
 Testdroid Cloud Testdroid Cloud
− Device cloud for developers to run their tests on 100+ real
Android devices, in parallel
− Provides detailed test results and analytics
 Testdroid Private Cloud
− Private on-premise device cloud for Enterprises to run their
tests on Android devices in their own development lab
− Provides detailed test results and analytics
 Testdroid Recorder
− Developer tool to simplify and accelerate the creation of
test scripts
14
info@bitbar.com

15. INTRODUCING TESTDROID APP CRAWLER
 Intelligent systematic app See video: http://youtu.be/eYvhyjf7HoY
exerciser
− Device cloud for developers to run their tests on
100+ real Android devices, in parallel
− Provides detailed test results and analytics
 Visual easy to read results
− Visual browser for inspecting the crawler's path
through the app
− Screenshots, logs and performance info
 Android testing will never be the
same!
15
info@bitbar.com

16. TEST AUTOMATION FRAMEWORKS FOR
ANDROID - OVERVIEW
This is an overview of OPEN SOURCE Android testing frameworks – there are tens of
closed proprietary testing tools/frameworks that are not covered here
Selected frameworks:
- Monkey exerciser
- Monkey Runner
- Robotium
- Roboelectric
- Calabash
- Monkey Talk
16
info@bitbar.com

17. AUTOMATION FRAMEWORKS FOR ANDROID -
OVERVIEW
Two main approaches:
Android Instrumentation Framework Instrumentation wrapped to the app itself
17
info@bitbar.com

18. AUTOMATION FRAMEWORKS FOR ANDROID –
MONKEY EXERCISER
 What:
- A UI/Application exerciser monkey for creating stream of random user events such as clicks,
touches, gestures and some system level events
 Usage:
$ adb shell monkey -p your.package.name -v 500
 Pros:
- Useful way to stress test your application to find non-obvious usage patterns and corner cases
 Cons:
- No way to create any other inputs than clicks which make for instance logging in to an app
impossible
18
info@bitbar.com

19. AUTOMATION FRAMEWORKS FOR ANDROID –
MONKEY RUNNER
 What:
- An API toolkit for writing programs that control an Android device or emulator from outside of
Android code
 Usage:
device.press('KEYCODE_MENU','DOWN_AND_UP')
 Pros:
- Only way to create tests that span across application boundaries
 Cons:
- UI elements can only be clicked with x,y clicks that do not scale => you have to create one script
for each resolution. There is no validation mechanism for any events created by clicks
19
info@bitbar.com

20. AUTOMATION FRAMEWORKS FOR ANDROID –
ROBOTIUM
 What:
- It's like Selenium for Android. A JUnit based helper library/framework
 Usage:
solo.clickOnText("Configuration");
 Pros:
- Powerful UI, functional and system testing capabilities (because it's based on JUnit)
 Cons:
- Cannot cross the application boundaries in test scripts (because it is using Android
Instrumentation Framework)
20
info@bitbar.com

21. AUTOMATION FRAMEWORKS FOR ANDROID –
CALABASH
 What:
- Cucumber based test automation framework for Android and iOS
 Usage:
And I select ”Options" from the menu
 Pros:
- Creating scripts does not require any programming skills, Same scripts work across platforms
Cons:
- Wraps the application under test with an http server so you will be testing a different binary that
the one you are going to ship.
21
info@bitbar.com

22. AUTOMATION FRAMEWORKS FOR ANDROID –
MONKEY TALK
 What:
- Cross platform testing tool/framework for Android and iOS
 Usage:
Input address Verify “123 Main St.” %timeout=500
 Pros:
- Same script works for your Android and iOS app (if the apps are identical)
Cons:
- Uses proprietary scripting language. Requires you to wrap your app with a testing library so you
will be testing a different binary that the one you are going to ship.
22
info@bitbar.com

23. AUTOMATION FRAMEWORKS FOR ANDROID –
ROBOELECTRIC
 What:
- Running unit tests on virtual machine – instead of an emulator or real devices
 Usage:
activity.findViewById(R.id.press_me_button);
 Pros:
- Ultra fast! Can be run in every code commit
 Cons:
- It is just for unit testing – it doesn't give you any info on how your app runs on target devices
23
info@bitbar.com

24. FURTHER INFORMATION
 Additional information on web:
− Company and Products: http://www.bitbar.com
– Testdroid Cloud: http://cloud.testdroid.com
– Testdroid App Crawler: http://youtu.be/eYvhyjf7HoY
− Demo video (3min): http://youtu.be/zj11lzUmScg?hd=1
24
info@bitbar.com