Designing a Mobile Device Lab - XBOSoft Webinar

with Costa Avradopoulos
Designing a Mobile Device Lab
#MobileDeviceLab

XBOSoft
Dedicated to software quality improvement
Founded in 2006
We speed products to market with our expert:
• Software QA consulting
• Software testing
Global team with offices in San Francisco,Beijing
and Amsterdam
“Thorough, accurate and fast”
XBOSoft, Inc. All Rights Reserved. 2

House Rules
 Participants other than the speakers are muted
 Questions via the GoToWebinar control on the
right side of your screen or through Twitter
@XBOSoft
 Questions may be asked throughout the webinar -
we’ll try to answer them at the end
 You will receive info on recording after the webinar
Webinar Hashtag: #MobileDeviceLab

Meet Our Speakers
Philip Lew
CEO and Founder, XBOSoft
• Relevant specialties and
passions
o Software quality process,
evaluation, measurement
and improvement
o Software quality in use /
UX design
o Mobile User Experience
and usability
o Cycling and travel
Karla Smith
Marketing Manager, XBOSoft
• Emails you about our
industry hot topic
webinars
• Invites you to download
our latest trends in
software testing
whitepapers
XBOSoft Team

Guest Presenter
Costa Avradopoulos
• Over 25 years’ experience in development, from
requirements, to delivery, and process engineering
• Roles in Development, QA, Automation, Product
Management, VP, and CTO. Management of
systems up to 15 million users in verticals such as
Telecommunications, Financial, Wireless,
Transportation, and Retail.
• Much of career dedicated to the mobility space, culminating with a breakthrough
patented mobile technology
• Frequent speaker. Writing book due to be published soon: Winning Mobile
Strategies – Bridging the Quality Gap
• Principal for Avracom. Advises organizations in the creation of winning mobile
strategies to develop best of breed applications.
• Certified Six Sigma and TPI
costa@avracom.com
www.avracom.com

Why We’re Here
• Mobile applications – disruption in the market
and in the SDLC
• Mobile testing requires a different approach
• A good strategy starts with designing a good
mobile device lab
• With thousands of variations, traditional
approaches are insufficient

Agenda
• Mobile Testing Challenges
• Defining Lab Requirements
• Building a World Class Device Lab
Out of Scope for Today:
• Discussion on specific tools
• Automation

How Do You Define Mobile?
1 ABLE TO MOVE OR BE MOVED
FREELY OR EASILY. 2
OF OR RELATING TO CELLULAR PHONES,
HANDHELD COMPUTERS, AND SIMILAR
TECHNOLOGY.
Smartphones, Feature phones, tablets, laptops, ruggedized devices,
medical devices, wearables, kiosks, ATMs…

World Quality Report 2014
KEY FINDINGS:
The QA function is adapting to business demands by streamlining and
centralizing its structure, to achieve efficiency and cost optimization
Significant investment in Test Environments may also be undermined
by a lack of specialist expertise
Testing’s late engagement in the application delivery lifecycle and its
reliance on basic IT metrics is possibly applying a negative brake on
this increasing maturity
Mobile testing has increased in importance as a key discipline within
the function, but lacks specialized methods, expertise and
environment

World Quality Report 2014
KEY FINDINGS (Cont.):
Cloud adoption and cloud-based testing has slowed as organizations
continue to manage concerns regarding data security and
performance
A higher proportion of overall IT budget is being invested in testing
and focused on transformation projects
Agile development is now widely adopted but still gives rise to
problems for testing, particularly in relation to specific methodologies
and expertise
NOTE: Mobile Testing is a dedicated section in this report
To obtain a copy of this report, email
costa@avracom.com

Mobile Testing Areas of Focus - 2014
Source: World Quality Report, 6th edition
11
XBOSoft, Inc. All Rights Reserved.

The Cost of Poor Quality Escalates
65%OVER
of all apps have a
1 star rating or less
Top quality issues in last 12 months exceed $2
Billion in losses:
• Security: breach at top 10 major retailer ($570M+ loss)
• Performance/Outages: Top 3 social media company – 5 major
outages in 2014, and 2 outages in 2015
• Security: breach at top 10 major retailer ($500M+ loss)
• Functional: Top 3 bank, double payments issue
12

Impact of Failure
Cost of Poor Quality is Accelerated with Mobile
Visibility is far greater, exposing your brand & reputation:
• Social Media makes failures exponentially more visible, with potential
to go viral
• App store is a public report card - you cannot hide or cover up poor
ratings
A plethora of choices for customers = competitive risks
• Near zero switching costs for customers
• 60% of users will abandon your app/site if does not load in 3 seconds
• Less than half will return to give a slow app/site a 2nd chance
13

Mission
Critical
Low Criticality
Mobile Testing Challenges
Security
Usability
Performance
Networks
Fragmentation
Due to involvement of GSM/ CDMA carriers identification of threats,
risk and determining its business impact is crucial
Thorough understanding of Visual, Accessibility and certification
testing is a must
Due to the limited hardware architecture of mobile devices, success
and failure of application in market depends on performance of the
app
Testing to be performed on different networks like 3G, 4G, Wi-Fi and
Edge, etc
When you factor in different OS versions, unique hardware models,
and carrier versions, the permutations are in the thousands
Tools Specialized knowledge of tools specific to mobile testing
Lack of Awareness
There are many mobile tools and test approaches available in market,
but all are not mature
14

Internal Cloud
Definitions of Cloud
What do we mean by:
PUBLIC CLOUD
A mobile lab hosted by the testing tool
provider. All cloud clients have access to
the devices in the lab. A specific tester
reserves the device he needs and uses it
when its available. Accessible through
the internet.
PRIVATE CLOUD
A mobile lab hosted by the testing tool
provider dedicated to a specific client.
The tester can reserve a device without
having to share the devices with other
clients. Accessible through the internet.
ON-PREMISE CLOUD
A secure mobile lab set up by the
client itself. All the devices are
connected to a server inside the
company network. The company has
complete control over what devices
are procured and who uses them.
External Cloud
15

Shared Advantages
• Enables better collaboration between Dev/QA
• Both provide very robust lab/device
management, supporting provisioning and
maintenance
• Resource management. Both provide a
mechanism for scheduling/reserving devices
• Improve efficiency of test execution
• Enable continuous integration for DevOps
16

Using an External Cloud
Pros:
• Scalability - easy access to hundreds of devices on demand
• No need to maintain devices (provider does it)
• Newer devices added to the pool at no extra cost
• Flexible for more advanced solutions (ex. GPS)
Cons:
• Tester session performance tends to be slower, especially with higher
bandwidth apps like video
• Some security concerns –Testing done outside the firewall, while data
is transferred back and forth through the internet
• Different companies use the same devices with
your app and potentially your private test data
17

Using an On-premise Cloud
Types:
• Commercial administration via cart/hub
• DIY lab
Pros:
• Higher performance real-time user sessions
• Increased security over devices and test data
• Allows users to pull devices from lab for hands-on tests
• DIY is least expensive
Cons:
• Limited scalability
• Maintaining devices requires a lab manager
• Procuring/retiring devices can be challenging
• DIY is most difficult to maintain and physically secure
18

Emulators
Benefits:
• Most cost effective
• Least infrastructure setup
• Flexible and Scalable
• Support for most test plans
Constraints:
• Cannot execute Usability,
Performance, & Security test
cases on emulators
• Network and hardware
interrupt testing cannot be
done on emulators
Mobile Testing Comparison
Physical Devices in Lab
Benefits:
 Covers all test types
 Safest option
 Should be used for most
common device types
 Best for UAT / Usability
Constraints:
 Very costly to support across
all OS, device, and version
combinations
 Procurement and setup of new
devices can delay testing
Physical Devices in Cloud
Benefits:
 Cost effective
 Fastest start up
 Flexible and Scalable
 Can cover approx 90% of test
plan
Constraints:
 Latency during usage and
longer test execution time
 Lower level of security
excludes this as an option for
some clients

Defining Lab Requirements

Use Real Mobile Devices
the degree to which a test system matches a production
system, including hardware, firmware, devices, software,
appliances, networks, etc.
TEST
INTEGRITY
Emulators/Simulators are tempting due to ease of use and low cost, however:
Proven to cause significant
defect leakage (stats & peers)
Lowest level of test integrity Certain functions cannot be
replicated
21

Other Device Lab Considerations
LAB SETUP WILL DEPEND ON:
Test
objectives
on coverage
Manual vs.
automation
Testing types Security Budget Scale
Location of
Teams
22

POLL
Approximately how many unique models of mobile
devices exist today?
A. 5,000
B. 10,000
C. 15,000
D. 20,000
E. 35,000
Assumptions:
• We are actually looking for approximate total # of permutations
• Therefore a device with the same model number, but on a different carrier,
counts as more than 1 “model” or instance
• Total active devices in use today globally, including feature phones, tablets,
wearables, but excluding IoT/embedded (cars, kiosks, appliances)

Test Coverage
Device type (handheld, tablet, kiosk)
Screen size & resolution – dozens just in the top 80%
Device models – 20,000+ unique variations exist!
Operating system – 27 variations in the top 85%
Connection type (USB, WiFi, Bluetooth, Cellular)
Mobile Web vs. App
Variance in carrier installations – 750 carriers globally
Even using 80/20 rule, you are left with thousands of permutations! So what do we do?
24

Use Analytics to Understand Your Users
Implicit Analytics: Platform usage and user behavior
Omniture
Google
Analytics
Flurry
TestFlight
(monitor
behavior)
Mixpanel
(native vs.
mobile web)
Localytics
25

Use Analytics to Understand Your Users
Xyo
Applause
(external,
sentiment) Appsfire
MeltwaterHootSuite
Explicit Analytics: What your users say
26

Customer Feedback: Great place to find those hidden test
scenarios!
1. Social Media (Facebook, LinkedIn Groups, Twitter, etc.)
2. App reviews
3. App ratings / comments
4. Comments, forum posts and articles on the web
5. Complaints and support requests
6. Trending analytics (ex via Applause)
Ensuring Proper Test Coverage – Rounding it Out

Lab Definition Exercise
First step – review analytics
The data says our customers are:
• 55% on iOS
– 62% iPhone 6, 15% iPhone 6+, 23% iPad
• 43% on Android
– 45% on Samsung S5, 20% on S6, 20% Note 5, 15%
Tab S (tablet)
• 2% on Other (Windows/Blackberry)

Additional considerations:
• 8 screen sizes/resolutions in top 80%
• 12 OS versions in top 80%
• 4 major U.S. carriers
• 1,000 functional test cases

Tier iOS Android Other Carriers Screens Tests Execution
Total 6 6 2 4 8 1,000 N/A
Tier 1 4 4 2 3 5 1,000All tests
Tier 2 3 3 0 2 3 750
Mission
Critical
Tier 3 2 2 0 1 2 400
Light
regression
Tier 4 1 1 1 1 1 100
Exploratory/
Smoke

Test Coverage Sample By Phase
Source: Perfecto Mobile

Test Coverage Sample By Index

Test Coverage Sample By OS

Building a World Class Device Lab

Overview of Device Lab Components
Desktop Users U.S. or India
Device Management Interface (View device inventory, reserve, etc.)
Real-time Device Viewer (launch sessions)
Remote Control protocol (access to launch apps, perform gestures, text, etc.)
Device interface (USB hub, Desktop/Server, Proprietary Appliance)
Real Devices,
On-premise or Cloud
36

U.S. Test
Team
Firewall
Example On-premise Cloud setup
India Test
Team
Mobile Devices
on-premise Cloud
Device access
via desktop tool
Wi-Fi Access Point
Carrier Network
Device access
via desktop tool
Mobile Devices
on-premise Cloud
Wi-Fi Access Point
Carrier Network
Firewall

U.S. Test
Team
Firewall
Example Public Cloud setup
India Test
Team
Mobile Devices
Off-site Cloud B
Device access
via desktop tool
Wi-Fi Access Point
Carrier Network
Device access
via desktop tool
Mobile Devices
Off-site Cloud A
Wi-Fi Access Point
Carrier Network
Firewall

Mobile Device Lab Recap
• Define mobile for your organization
• Create thorough test objectives
• Weigh all factors when defining lab
requirements
• Decide on Cloud vs. On-premise
• Define test coverage by leveraging analytics
• Monitor channels for continuous feedback
• Use tier-based approach to optimize coverage

Post your questions on Twitter and we'll answer them @XBOSoft
Join us to keep updated on all our webinars, reports and whitepapers:
facebook.com/xbosoft
+xbosoft
linkedin.com/company/xbosoft
We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/
Download our free Whitepapers, available here:
http://xbosoft.com/knowledge-center/
You will receive an email with information on today’s webinar slides and recording.
Please email us with any further queries regarding our services or ideas for future
webinars! Services@xbosoft.com
Q+A
www.xbosoft.com

Designing a Mobile Device Lab - XBOSoft Webinar

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (7)

Semelhante a Designing a Mobile Device Lab - XBOSoft Webinar

Semelhante a Designing a Mobile Device Lab - XBOSoft Webinar (20)

Mais de XBOSoft

Mais de XBOSoft (20)

Último

Último (20)

Designing a Mobile Device Lab - XBOSoft Webinar