For mobile applications, automated testing is a wise choice due to the many OS platforms and devices they run on; we offer a strategy for setting mobile test automation goals and selecting the right tool sets.

1. • Cognizant 20-20 Insights
Selecting the Right Mobile Test Automation
Strategy: Challenges and Principles
Understanding the various options and choosing the tools and
automation processes best suited to testing both mobile Web and
native mobile apps can reduce complexity and speed time to value.
Executive Summary through test automation. As with conven-
tional applications, there is an increased need
The mobile phone has transitioned from its tra-
to automate mobile applications testing to
ditional role as a mere communication medium
improve scale and efficiency. A well-orchestrated
to that of a multipurpose personal gadget. Tech-
automation strategy enables hardware manufac-
nological advancements and the proliferation
turers and software developers to reduce effort
of devices across operating systems (Apple iOS,
and accelerate time to market.
Android and Windows Mobile) and platforms have
made it more challenging for hardware manufac-
Mobile Technology’s Growth Promise
turers and application developers to develop and
roll out new products. The growth of mobile devices is evidenced by the
following:
For vendor peace of mind, mobile applica-
tions must be tested to ensure they run on key
• Globally, the user base of location-based
services has grown from 96 million in 2009
platforms and across a multitude of networks. to more than 526 million in 2012, according to
Despite the pressures of short mobile develop- Gartner Research.1
ment cycles, quality testing of applications across
operating systems, device platforms and networks • Worldwide mobile advertising and content
is a necessary but daunting task to ensure revenue is forecast to reach $67 billion this
long-term success in what is a highly fragmented year.2
and competitive global market. Moreover, non- • IDC predicts that smartphone shipments will
functional testing — including usability, security reach 978 million in 2014.3
and adaptability — is as critical as functional
testing. Effective testing can help device makers • By 2016, smartphones and tablets will
put power in the hands of a billion global
and application developers collect appropriate
consumers, according to Forrester
metrics that improve product quality.
Research, Inc.4
This white paper explores industry best practices • Mobile device makers are providing
on mobile testing and demonstrates effective customized platforms to increase their share
ways of managing mobile application quality of the customer’s wallet.
cognizant 20-20 insights | september 2012

2. Two Key Challenges in Mobile consider a test case that is created and needs to
Application Test Automation be executed on three different platforms — iOS,
Android and Blackberry. In this example, the test
Mobile Platforms and Technology Coverage
case is expected to be executed on 13 different
The primary factor that determines an automation devices and in 10 different combinations, which
tool’s success is its ability to work across platforms leads to 130 rounds of testing. Designing a test
and technology stacks. The following challenges execution matrix of this scope is both difficult and
influence automation success: time-consuming (see Figures 1 and 2).
• Device Diversity: Mobile Testing Industry Opportunities
>> Multiple platforms and browsers. In the growing mobile market, the need for spe-
>> Rendering differences. cialized mobile testing is growing. The following
>> Mobile devices with varied application run three opportunities contribute almost 80% of
times. total market demand.
• Network Challenges: Mobile Automation Testing
>> Multiple network types (e.g., GSM/GPRS/ There are only a handful of tools available to suc-
Wi-Fi/Wi-Max). cessfully automate testing. The ROI promised by
>> Different speeds of connectivity across automation testing highlights the need for more
options.
geographies.
>> Multiple network operators with custom- Functional Testing
ized network features.
Mobile functional testing always involves a non-
• Hardware Challenges: heuristic component that cannot be automated.
>> Limitations in processing speed. Mobile Performance Testing
>> Limitations of mobile memory size. Compared with a desktop/notebook environment,
>> Differences in device communication mobile device system resources (e.g., processing
protocols (e.g., WAP/HTTP). power, memory, etc.) are limited. Hence, perfor-
mance testing of mobile applications is crucial.
Test Execution Challenge Therefore, this domain is growing and has great
Test execution across different devices and brows- potential.
ers is a huge, challenging effort. For example,
Real-time Scenario for a Smartphone-based App: Sample Test Execution
Mobile Applications
Use Case 1 iPhone
Mobile Test Lab
Use Case 2
Wi-Fi Blackberry
Use Case n Android Emulators Real Device Device Anywhere
iOS Native Android Native Blackberry Native Opera
Devices/Browsers Opera Mini Sky Fire Firefox 4 Firefox 5 Firefox 6 Dolphin
3,4&5 (2.1,2.2,2.3, 3.0,4.0) 5,6 ,7,8 & 9 Mobile 11.1
iphone3
iphone4
iphone5
Nexus One
Moto Droid
Samsung
Captivate
HTC Incredible
Nexus S
Moto Droid X
Curve 8330
Tour 9630
Bold 9650
Bold 9000 Touch
Figure 1
cognizant 20-20 insights 2

3. Smartphone Platforms and Browser Combinations
iOS Android Windows Phone Blackberry OS Symbian
Current Version 6.0 Current Version 4.2 Current Version 7.5 Current Version 7.0 Symbian 4
Key Versions Key Version 6.5 Key Version
Key Version 2.3, 3.0, 4.0
(iPhone 4.0, iPad 2) 6.x & 7.x series Open Source
Closed Source Open Source Closed Source Closed Source
Symbian
Apple Google Microsoft RIM
Multitasking
Multitasking Multitasking Multitasking Multitasking
Safari Web Browser Chrome/Webkit Browser Internet Explorer Webkit Browser Webkit Browser
Figure 2
Mobile Automation: What and Why? • User agent-based automation: This approach
leverages user agent add-ons that come with
Mobile applications are becoming increasingly
popular browsers such as Mozilla Firefox
sophisticated, significantly increasing the require-
and Google Chrome. The user agents help
ment for functional testing (see Figures 3 and
render the specific Web content that would be
4). To tackle this, test organizations are increas-
displayed on the device onto a regular desktop
ingly exploring alternatives to traditional manual
browser. The functionality can be automated
testing. Automated testing is a highly effective
by using popular tools such as QTP, Selenium
approach to mobile app QA that can provide
or RFT — each of which supports all desktop
significant business returns, provided it is imple-
browsers.
mented by using the right tools and architecture,
factoring in cross-platform challenges. • Cloud-based automation: Cloud-based test
automation vendors support Web-based QA
The following specific types of testing also need across platforms. Cloud-based automated
to be automated: solutions are highly effective as they are
offered as a unified package.
• Cross-platform compatibility testing: This
is necessitated by the growing number of • Lab-based automation (devices/simulators):
handsets and platforms. This method involves setting up tools in a
test environment and leveraging simulators
• User experience testing: Most testing orga-
or physical devices to automate testing using
nizations have limited experience with design
popular open-source tools such as Selenium.
and execution of usability tests.
Web driver–supported device browsers and
• Field or network testing: Testing must be simulators are best suited here.
performed in a geographically distributed envi-
ronment to account for a variety of network Approaches for Native and Hybrid
types. Mobile Applications
• Structural challenges: Integrating mobile Cloud-based Approach
testing toolsets into existing IT systems is a Cloud-based test automation is one popular
crucial hurdle. means for achieving native and hybrid types of
test automation. The method is useful when
Test Automation Mechanisms
long-term automation is required across a set
for Mobile Apps
of predefined devices. Commercial tool add-on
A mobile Web site can be automated in one of the mechanisms such as HP QTP and IBM RFT are also
following three manners: available for test engineers who are familiar with
cognizant 20-20 insights 3

4. Three-tier Mobile Application Testing Framework
Backend Middleware Device UI
Application Layers
Admin Services Mobile Client
Enterprise
Data
Communication
Mobile Web Framework/
Services Mobile Website
Application Server
Data
Integration Testing Device Management Installation Functional
Types of Testing
SOA Testing
Testing Testing Testing
Integration
Testing Security Testing Compatibility Device
Testing Profiling
Authentication Testing
UI Testing Interruption
Fully Automatable Partially Automatable Not Automatable
Figure 3
industry-wide products. The ability to automate in enabling automation in a closed environment,
from anywhere is a great advantage: All that for continuous automation or to achieve test
is needed is connectivity to the cloud lab. The regression functionality. They also cover nonfunc-
“automation-from-anywhere” feature is a distinct tional scenarios that handle mechanisms largely
advantage when using tools such as Perfecto relevant to mobile apps.
Mobile and Device Anywhere.
Open Source/Platform-specific Tools
Lab-based Approach Device platform-specific tools reliability is a high
There are multiple tool sets with different capa- priority. Platform vendors supply tools such as
bilities available for organizations interested in Robotium for Android and iOS instruments for
a lab-based approach. Tools such as EggPlant, iOS to provide this facility. However, to leverage
Jamo, See Test, ZAP, SilkTest and Test Quest this effectively, it requires additional program-
provide industry tool add-ons. They are useful ming skills.
Potential Areas of Automation in End-to-End Mobile Application Testing
100
90
80
Percentage of tests performed
70
using automated tools
60
50
40
30
20
10
0
Mobile Functional Mobile Mobile Cross- Security Testing SOA/ Package Enterprise
Automation Testing Performance Browser Middleware Application Application
Testing Testing Testing Testing Testing (e.g., ERP) Testing
Source: Cognizant; based on 350-plus projects performed across all vertical markets and our
analysis of industry trends.
Figure 4
cognizant 20-20 insights 4

5. Cognizant’s Tools Web. Consider different connectivity aspects for
automation execution runs.
We have our own platform to easily automate
testing of mobile native and Web apps automation In addition, the following recommendations will
across platforms. be helpful in informing your test automation
journey:
Our platform ensures single type scripting that
works cross-platform and thus guarantees greater
reusability of scripts. The platform also offers
• Perform tool feasibility tests, because mobile
technologies and platforms vary.
an easy interface for testers to auto-generate
scripts. For instance, scripts can be automated • Select tools that support both platform
without any programming or scripting language simulators and devices, because you can mix
knowledge. It also comes with a facility to and match devices and simulators to optimize
automate as applications are created to support runs on different platforms.
the Agile development methodology. • Aim for automation in nonfunctional areas
— interruption and hardware scenarios that
Mobile Application Testing Tools include battery state changes.
Selection Criteria
There are many successful techniques that can
• Always optimize on platform support: In some
cases, there may be a need for one or more
make testing applications on mobile devices
tools to perform automation.
more effective and efficient. It is essential to
ensure functional and nonfunctional compliance • Identify multiple device support and versions
of mobile applications on devices and mobile support.
automation. It is also key to establish labs for • Determine the use of utilities and reusable
testing mobile applications. functions that add value to automation.
Some basic criteria for mobile application testing • Ensure ROI per each mobile platform and
include the use of emulators, multiple browsers, duration of execution as part of the tool con-
different operating systems and the unique types sideration process.
of testing required for GUI, field and interrup- • Look for avenues for early automation, because
tion testing. There are roughly 10 key principles most mobile projects are Agile.
that need to be considered while selecting the
right tool.
• Integrated execution with the test manage-
ment tool and application lifecycle manage-
ment is an important success factor for a tool.
11 Key Principles in Selecting the Right Tools for
Mobile Testing • Look for data-driven automation support:
It is good practice to start with a proof of concept Iterations in the execution will increase
(PoC) on critical platforms before narrowing coverage and ROI.
down to a tool or tool set. One tool may not • Good candidates for automation are test
achieve cross-platform automation goals. When cases that have repeated business logic,
test automation tools are evaluated, organiza- functional flows or data-driven tests: Smoke
tions must not overlook how they integrate with test and build acceptance testing, regression
test management environments. One starting testing, synchronization testing, compatibility
point is to identify reusable scenarios across testing, GUI testing and testing non-dynam-
platforms and group them accordingly. This facili- ic UI components, as well as cross-platform
tates the creation of an appropriate, time-saving functional testing.
framework.
Moving Forward
For native apps, choose critical test cases that can
The principles and success factors of test
be run across releases as automation candidates
automation for mobile applications are much
and ensure that there is a cross-platform reus-
different than those traditionally applied to con-
ability and multi-browser compatibility. Always
ventional applications. The attributes of devices,
keep in mind different form factors while deter-
environment, tool sets and test coverage add
mining test coverage, especially for the mobile
cognizant 20-20 insights 5

6. multiple layers of complexity. It requires keen and choosing appropriate coverage, the right
insight and a customized approach for a well- tool sets and a comprehensive approach. Making
orchestrated automation strategy. automation effective and efficient ultimately
yields cost and time reduction in the app develop-
The best practices covered in this paper will ment lifecycle.
guide you toward setting test automation goals
Footnotes
1
http://www.gartner.com/it/page.jsp?id=1230413
2
http://www.mediapost.com/publications/search-marketing-daily/edition/2012/07/13/?print
3
IDC Worldwide Smartphone 2012-2016 Forecast Update, June 2012.
4
Ted Schadler, John C. McCarthy, “Mobile Is the New Face of Engagement,”
Forrester Research, February 2012.
About the Authors
Pradeep Kumar Govindasamy leads Cognizant’s Automation and Mobile Testing Center of Excellence,
a practice comprising more than 1,500 experts servicing global customers. He brings over 12 years of
experience in the information technology industry and is a valued speaker on topics such as automation,
test data management and tools at events worldwide including Starwest 2009 and 2011, Stareast 2011,
Swiss Testing Day, HP Universe and EuroStar. Pradeep can be reached at PradeepKumar.Govindasamy@
cognizant.com.
Ramakrishnan Venkatasubramanian heads Cognizant’s Automation R&D and Offshore Mobile Testing
teams. He has over 12 years of experience in the information technology industry. His experience spans
software development, product engineering, testing, test automation, mobile and technology consulting.
Ramakrishnan can be reached at Ramakrishnan.Venkatasubramanian@cognizant.com.
About Cognizant’s Mobile Testing CoE
Cognizant‘s Mobile Testing Center of Excellence (CoE) provides state-of-the-art solutions for mobile testing require-
ments with best-of-the-breed testing practices and processes. It specializes in offerings such as unit testing, functional
testing, multi-language compatibility testing, security testing, automation testing, performance testing, device testing
and application testing. Our Mobile Testing CoE features a strong pool of certified mobile testing experts, built through
long-term competency development programs. In addition, it houses smart products and frameworks powered by a
focused R&D team, which takes mobile application testing to a whole new dimension.
About Cognizant
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process out-
sourcing services, dedicated to helping the world’s leading companies build stronger businesses. Headquartered in
Teaneck, New Jersey (U.S.), Cognizant combines a passion for client satisfaction, technology innovation, deep industry
and business process expertise, and a global, collaborative workforce that embodies the future of work. With over 50
delivery centers worldwide and approximately 145,200 employees as of June 30, 2012, Cognizant is a member of the
NASDAQ-100, the S&P 500, the Forbes Global 2000, and the Fortune 500 and is ranked among the top performing
and fastest growing companies in the world. Visit us online at www.cognizant.com or follow us on Twitter: Cognizant.
World Headquarters European Headquarters India Operations Headquarters
500 Frank W. Burr Blvd. 1 Kingdom Street #5/535, Old Mahabalipuram Road
Teaneck, NJ 07666 USA Paddington Central Okkiyam Pettai, Thoraipakkam
Phone: +1 201 801 0233 London W2 6BD Chennai, 600 096 India
Fax: +1 201 801 0243 Phone: +44 (0) 20 7297 7600 Phone: +91 (0) 44 4209 6000
Toll Free: +1 888 937 3277 Fax: +44 (0) 20 7121 0102 Fax: +91 (0) 44 4209 6060
Email: inquiry@cognizant.com Email: infouk@cognizant.com Email: inquiryindia@cognizant.com
©
­­ Copyright 2012, Cognizant. All rights reserved. No part of this document may be reproduced, stored in a retrieval system, transmitted in any form or by any
means, electronic, mechanical, photocopying, recording, or otherwise, without the express written permission from Cognizant. The information contained herein is
subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.