Watch a live presentation at http://offer.bitbar.com/which-one-works-you-the-best-in-house-or-cloud-based-development-environment The best and the most suitable development environment whether it is a full in-house solution with your own devices, servers and other infrastructure – or development infrastructure environment fully hosted for you, depends on various aspects. Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/

1. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
1
8 January 2015
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com
Which One Works You The Best:
In-House or Cloud-Based
Development Environment
W E B I N A R

2. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
2
Agenda
• Comparison of In-house and Cloud-based Setups
• Investments in Mobile App Development and
Testing Environments
• The Characteristics to Use for Comparison of Both
Development Environments
• Calculation Examples
• Testdroid Update
• Q&A

3. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
3
Introduction to Creation of
In-House Development Labs

4. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
4
Picture from #GTAC2014

5. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
5
Why Automation?
Smaller
coverage,
More
money
burnt
&
?me
wasted,
Error-­‐prone
Large
coverage,
quickly
completed,
Less
money
&
?me
wasted,
Exact
results.
Manual
Automa@on

6. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
6
• Why Do You Need an In-House
Development and Test Lab?
– Pre-market devices (OEMs, ODMs,
integrators ...)
– Prototypes / parts of hardware
– Corporate security policies
– Availability and high usage level
– Special setups (integration w/other
equipments)
– Large set of devices available
In-house Dev&Test Labs

7. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
7
• What is Required for an In-House
Development & Test Lab?
– Server hardware / infrastructure
– Own mobile devices
– Device control servers
• Hard Disks, RAM, Energy management
– USB hubs
– WiFi / Network infrastructure
– Continuous Integration/Delivery/Testing
– Build systems / other software tools
– POWER!
In-house Dev&Test Labs

8. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
8
Continuous Integration

9. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
9
• Version control and build revision
• Build automation, instant execution&testing
• All types of testing & integrations enabled
• Frequent commits, Code consolidation ->
Fast builds
• CI integrated with development and testing
provides the best fit for mobile agile process
Core Benefits of CI

10. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
10
• How to Reach Enterprise-Grade Reliability
for In-House Dev&Test Lab
– Device management
• Always run on clean devices
• Create intelligent retry mechanisms
• Create fall-back mechanisms
– System management
• Automatic reconnect of network and USB
• Automatic configuration changes
• Professional-level monitoring system
– Diskspace, network/connections, Power
In-house Dev&Test Labs

11. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
11
Issues?
29%
of problems are tightly
related to devops of an
in-house system

12. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
12
Introduction to Adoption of
Cloud-based Development Labs

13. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
13
• Someone else is taking care of heavy-lifting
• Cloud-based device cloud includes ALL
relevant devices (phones, tablets, cams...)
• You get everything automatically delivered
– Data, Results, Logs, Screenshots, Pass/Fail
Reports, Additional data-sets
• Diversity and Keeping Up-to-Date with SW
– New OS versions and OEM updates
• Easiness of Adoption and Integration
Why Cloud?

14. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
14
Access via Comprehensive API

15. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
15
• Authentication
• Project Management
– Create, Delete, Configure, Managing Users
• Configuring Access and Assets
– Configs, Upload/Download Files (App & Tests)
– Device Access & Device Group Configuration
• Running Tests
– Upload App + Tests <-> Download Results/Data
– Notifications, Sharing, Pre-processing of Data
Access via Comprehensive API

16. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
16
Investments in Mobile App
Development Environment

17. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
17
Investments Aspects
• What is the Most Cost-
Efficient Approach?
• What is the Total Cost of
Ownership?
• Capital Expenses
(CapEx)
• Operation Expenses
(OpEx)

18. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
18
• CapEx is a business
expense incurred to
create future benefits or
to add to the value of an
existing asset
• OpEx refers to the
expenses that the mobile
app developer spends in
order to get their day-to-
day activities going on
and to get their
infrastructure fully
working
CapEx vs. OpEx

19. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
19
• CapEx:
Devices,
Servers,
Infrastructure
• OpEx:
Running
&
Maintaining
the
System
Costs: In-House Dev&Test Lab

20. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
20
• Cost
of
Service
includes
all
major
expenses
(both
CapEx
and
OpEx)
Costs: Cloud-based Dev&Test Lab

21. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
21
10 Different Characteristics for
Comparing Environments

22. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
22
• A
Different Characteristics & Benefits

23. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
23
• Corporate Policies
• Regulation
• Minimizing security
breaches
• Security/Protection
mechanisms
• Monitoring
• Authentication &
User Access
Security1

24. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
24
• Needs to be fully
configurable
• Back-ups & Retention
• Preventing non-
authorized use and access
• Needs to available 24/7
• Transferrable nature of
data, source code, results
Data Ownership and Retention2
In-house
vs.
Cloud

25. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
25
• API access – no delays
• Critical/Medium/Low
• DevOps in crucial role
• Various data center
locations
• Hardware failures?
Monitoring? Scale up?
• Average Cloud Up-time:
99.5%
Performance and Availability3

26. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
26
• Support: Email, Phone,
Online service systems
• Intuitive online help:
Manuals, How-Tos,
Instructions (self-service)
• Integration with bug-
tracking systems (e.g. JIRA)
• Frequent product updates
• Needs to be available 24/7
Support4

27. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
27
• Clear policies on updates,
maintenance, monitoring
• Terms of Service + EULAs
• Manual vs. Automated
enhancements/updates
• Customized features
• Product roadmaps
Enhancements and Update Policy5

28. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
28
• Agile process enables agile
integrations
• Tools, Frameworks, Other
complementary software
• API
• Avoid lock-ins to proprietary
frameworks and other SW!
• Can define cost-efficiency of
using either approach
Integrations6

29. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
29
• Intuitiveness, Easy-to-
Use, Good Help
Materials
• Directly related to TCO
• Live demos and free
trials with the system
• Frequent webinars,
Q&A, "How-To"
materials
Usability7

30. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
30
• Lock-ins with in-house
environments
• Term or Perpetual
licensing model
• Return-on-
Investment?
• Subscription models –
optimal use of service
Contractual Flexibility8

31. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
31
• Own devices vs. devices
from cloud farm?
• Scalability with server
infrastructure
• Device control servers
– Linux with Android = 15
– iOS with Mac = 3
• Easiest scalability comes
w/privately hosted cloud
Equipment Scalability9

32. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
32
• Case Studies
• Customer References
• Reaching out to user
to ask any details
Recommendations & References10

33. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
33
Calculation Examples

34. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
34
Diminishing Value Curve

35. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
35
• Different levels
of control and
abstraction
• Definitions
aren't always
clear – so make
sure you
understand
what you are
buying!
SaaS-PaaS-IaaS vs. In-house

36. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
36
• Assumption
– You are building several apps
• No. of apps is not limited here, as all example
infrastructures can handle large number of apps
– You use 50 different devices for testing
– You need everything up and running and
someone maintaining the system 24/5 – OR –
just business hours
Basic Example

37. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
37
• In-house Setup
– Devices: ($200 x 50 = $10,000)
– Server infrastructure: ($1,000 per 5 devices -> $10,000)
– Software: ($99/month x 50 = $4,950/mo -> $59,400/year)
– Setup/deployment: 3 biz days ($50/hour x 8 x 3 = $1,200)
– DevOps to manage the system 24/5: ($50/hour x 24 x 5 =
$6,000/week x 50 weeks = $300,000)
OR ALTERNATIVELY
– DevOps to manage the system on business hours: ($50 x 8 x
5 = $2,000/week x 50 weeks = $100,000)
• Majority of costs comes from OpEX
In-House Dev & Test Lab

38. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
38
• Public Cloud Setup
– Deployment: 1 business day ($50 x 8 = $400)
– Integration: 2 business days ($50 x 8 x 2 = $800)
– License for 50 devices: ($1499/mo -> $17,988/year)
• Private Cloud Setup
– Deployment: 1 business day ($50 x 8 = $400)
– Integration: 2 business days ($50 x 8 x 2 = $800)
– License for 50 devices: ($199 x 50 -> $9,950/month ->
$119,400/year)
• Majority of costs comes from CapEx
Cloud-Based Services

39. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
39
• Head-to-Head:
– In-house (24/5)
costs $380KUSD
vs. Private Cloud
$120KUSD
• Privately hosted
cloud is 31.5%
of in-house
system costs
Cost Comparison
Comparable
environments:
216%
Significant difference in costs between in-house and cloud

40. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
40
Testdroid Update

41. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
41
Testdroid
PrivateCloud -
Running user-selected
devices at any of our
hosting facilities,
more cost-efficiently
than your own
devices!

42. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
42
Option 1:
Public Device Cloud
on-demand devices
(multitenant)
Mobile app testing on
over 400+ real
Android and iOS
devices hosted by
Bitbar
Option 2:
Private Device Cloud
reserved devices
Hosted by Bitbar in the
US and/or Europe
Devices chosen by and
reserved only for the
Customer
Option 3:
On-premise Device
Cloud
Automated mobile
app testing on real
Android and iOS de-
vices hosted by the
customer, usually
30-500 devices
Testdroid Deployment Options

43. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
43
• Get to know the essential
facts in terms of selecting
cloud-based or in-house
solution for your needs:
– Characteristics
– Costs vs. Benefits
– Calculation examples
• Available at
www.testdroid.com/library
New EBOOK Available!

44. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
44

45. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
45
Video recording & slides of this webinar will
be soon available at Vimeo and Slideshare!
Visit at www.testdroid.com for more information.
THANK YOU!
W E B I N A R

46. ©
Copyrights
by
Bitbar
Technologies
Ltd.
2015
All
rights
reserved.
46
We operate the largest global device
cloud with over 400 devices
instantly available for developers
Devices from the US, Europe, China, Japan and Korea
– global devices for global customers