An Essential Guide to Effective Test Automation Leveraging Open Source

An Essential Guide to Effective Test
Automation Leveraging Open Source
1
A Whitepaper by RapidValue Solutions
©RapidValue Solutions
0 1 0 0 1
0 0 1 1 0
1 1 0 0 1

2
Index
02 The Road that Led to Automation
03 What Makes Automation Important?
01 Introduction
04 Open Source Tools and its Impact on QA Automation
06 The ABC’s of Automation Test Coverage
07 Risks Involved in Test Automation
15 Benefits of QA Automation Testing
08 Optimization of Automation using Automation Framework
17 Conclusion
11 Use Cases of Different Open Source Automation Tools
17 Author
18 About RapidValue

1
Introduction
Mankind is stepping into a new era completely owned by automation and the advent of mobility and digital
technologies has simplified life by providing easy access to any individual to access technology without any
bias. In the business environment, this pushes companies to deliver high-quality applications in less time with
limited resources. This has ultimately led to a competition to deliver secure and advanced digital solutions
without breaking the requirements and guidelines of the clients. Projects developed under such pressure
have failed to meet objectives and suffer from vital schedule and budget slippage because of a delay in
discovering the defects. This results in an increased percentage of ‘Defect Leakage’ and eventually customer
dissatisfaction due to sub-par products. Organizations have begun to adopt automation to attain better
product quality and develop secure, faster and useful mobile applications. This whitepaper endeavors to
provide insights from our experiences on working with automation and elaborates on its importance with
reference to certain case studies.

2
The Road that Led to Automation
As the saying goes, “Necessity is the mother of invention”, automation too rose from the need to simplify
particular processes that were time-consuming. Testing mobile applications are more complex than testing
desktop and web applications. This is because, mobile applications need to be tested on a variety of software
platforms and versions, on varied hardware and custom factors and also under different network connectivity
conditions. In addition to this, the frequent OS updates, the introduction of new devices and the customer
expectation of experiencing quick upgrades, would require additional test cycles. This gave rise to test
automation where special tools are used to control the execution of tests and actual results are compared
with expected results. It can automate certain repetitive yet essential tests in a formalized testing process
already in place, or add additional testing that would have been otherwise difficult to perform manually.
Organizations have been opting for automation due to the following reasons:
• Automated testing simplifies low-level interface regression testing which is difficult to perform
manually and also aids in finding errors in the testing. This saves a lot of time that is spent on
performing repetitive and boring manual testing.
• Manual testing can be performed only with a limited number of physical devices, automation could
employ any number of physical devices as it supports cloud integration.
• While in case of manual testing, if there is a change in software, one has to run all the tests from the
beginning while in case of automated tests, this is not required as the tests are already set up. It also
offers a standard and structured framework to incorporate test executions.

3
What Makes Automation Important?
During the initial stage of project development, one needs to cover only a limited number of test cases and
hence manual testing would be of help in such a scenario. However, as the project progresses, automation
would become an inevitable component. As mentioned earlier, test automation saves the time that would
have been otherwise used for running regression tests, re-entering the same test data and checking against
coding standards. It fuels better consistency in results which could be tracked through the continuous
execution of scripts. It also supports static measures and coverage and also helps one in understanding
application behavior. The detailed reports generated by it provides statistics and graphs about test progress,
incident rates and performance and also allows real-time report collaborations with others in the project. The
automated test cases are more reliable and the scripts are scalable anytime in case of new enhancements in
the project. When the test cases pass, it builds up the confidence of the developer as it suggests that the new
code meets the test requirements and does not interrupt or degrade any existing features.

4
1. Selenium – Initially developed in 2004 by Jason Higgins as an internal tool at ThoughtWorks, it was later
open sourced in the same year. It is undoubtedly one of the best open source tools that supports quite a lot
of programming languages, browsers, and multiple system environments. With the flexibility that it provides
to the testers, Selenium acts as the core framework for other open-source automation tools.
2. Appium – The Appium philosophy is that testing native apps shouldn’t require including an SDK or
recompiling an app. With its primary focus on mobile applications, it provides full access to back-end APIs
and DBs from test code while automating mobile apps from any language and test framework. It is easy to
install and user-friendly, making it a much-preferred testing tool.
3. Katalon Studio – Built on top of open-source automation frameworks like Selenium and Appium, it is an
automation testing solution developed by Katalon LLC. Advanced testers use it for saving time otherwise used
on building new libraries and maintaining their scripts, non-programmers use it to easily start an automation
testing project. Katalon Analytics provides comprehensive views of test execution reports that include metrics,
charts, and graphs.
4. Watir – An acronym that is derived from, ‘Web Application Testing in Ruby’, Watir is an open-source testing
tool that supports cross-browser testing including Firefox, Opera, headless browser and IE. It consists of several
smaller projects with the most important ones being watir-classic, watir-webdriver and watirspec. Watir also
integrates with BBD tools like RSpec, Cucumber and Test/Unit and also supports data-driven testing.
5. Protractor – It is a NodeJS program written in JavaScript and that identifies the web elements in AngularJS
applications which is otherwise not possible by using JUnit or Selenium WebDriver. It combines powerful
technologies like Selenium, Jasmine and the like to work as a solution integrator. Protractor is reserved not
only for AngularJS applications but also for writing automated regression tests for normal web applications.
Open Source Tools and its Impact on QA Automation
Identifying the right tool is a crucial step in guaranteeing success in test automation. Enabling intelligent
automation, these tools are pivotal in the DevOps toolchain. Choosing the right tool might be tricky owing
to the wide range of choices available in the market. Also, there is no ‘one size fits all’ feature and hence it is
important to choose a tool that has the potential to address any testing challenge and can take advantage of
the upcoming trends in the digital world. Some of the top open source tools are the following:

5
Comparison of Automation Tools
Difference Selenium Appium Protractor
Test Development Platform
LicenseType
Scripting Language
Programming Skill
ImageBasedTesting
Continuous Integrations
Framework Support
Community
Ease of Installation
and Use
Object Storage and
Maintenance
Applications Under Test
Cost
Learning Curves
Script Creation Time
Cross Platform
Open Source
Java, C#, Perl, Python,
JavaScript, Ruby, PHP
Advance skill required to
integrate various tools
Requires additional
library support
Requires additional
configuration for CI
Open Source community
Requires installation and
integration of various tools
Need to create separate
object repository to store
the objects
Java, C#, Perl, Python,
JavaScript, Ruby, PHP
Advance skill required to
integrate various tools
Requires additional
library support
Requires additional
Open Source community
Requires installation,
integration of various
tools & additional setup
Need to create separate
object repository to store
the objects
Advance skill required
for automation script
development
Requires additional library
support like protractor
image
Requires additional
Community available
Requires installation,
integration of various
tools & additional setup
Need to store the objects in
the script itself
Web
Free
High
Medium
Cross Platform
Open Source
Mobile
Free
High
Medium
Cross Platform
Open Source
JavaScript
Web (AngularJS & Angular)
Free
High
Slow

6
The ABC’s of Automation Test Coverage
While analyzing the existing testing tools, we observed that most of the tools provide maximum test execution
coverage for applications. However by using an automation tool one cannot automate 100% functionality
of an application, especially if the application has any third party integrations. Automation test coverage is
calculated using the following formula:
Automation Test Coverage =
(Total No. of test cases automated / Total No. of test cases identified for automation) * 100
Here are the percentage of execution that were tracked in Appium and Selenium tools based on our
experience with multiple projects.
A coverage of 80% in execution was observed using Selenium.
Total No. of test cases automated = 960
Total No. of test cases identified for automation = 1200
Automation Test Coverage = (960 / 1200) * 100 = 80%
A coverage of 75% in execution was observed using Appium.
Total No. of test cases automated = 600
Total No. of test cases identified for automation = 800
Automation Test Coverage = (600 / 800) * 100 = 75%
Automation test coverage often refers to a set of automation scripts that are written against the requirement
specifications. The software quality could be measured through a specific analysis employing automation
test coverage. Automation tools easily execute thousands of different complex test cases during every test run
providing coverage that is otherwise impossible with manual tests. Testers thus find themselves with more
time that could be utilized in creating new automated tests and deal with complex features. Automation test
coverage also evaluates the ability of the tool that can automate complex scenarios thus helping us to select
a better tool. In some cases, paid tools might have more test coverage on the complex features of application
because they provide methods or commands to handle the flows. Whereas in case of trial version tools, we
may fail to reach the expected test execution coverage as it does not provide any methods or commands as
in the case of paid tools.

7
Risks Involved in Test Automation
Risks in automation can be divided into three major points:
• Appium instructor does not start: Faced while executing the Appium tool for iOS or Android
mobile applications. This was due to issues related to the WebDriver Agent.
• Failed to start instruments: Issue faced in Appium tool
• Failed to start an Appium session: Faced usually while launching an iOS application using Appium
tools when instruments crashed on setup.
1. Risk while selecting a tool – As a trial version does not offer full coverage, companies blindly go for paid
versions. When some companies purchase paid tools without understanding their maturity level, it could
result in a loss in profit. Thus before purchasing and implementing automation tools, one must possess
the ability to operate them with appropriate coding knowledge and skills. In fact, the success of the testing
process relies on the efficiency with which it is handled. Unrealistic ideas surrounding the tools ultimately
leads to failure. Likewise underestimating the time and effort needed to achieve significant and continuing
benefits from the tool could also be a risk. With the world moving at a rapid pace, one must anticipate
changesandpossessaproperestimationtoincorporatethechanges.Alsoonemustbecarefulwhileselecting
an automation framework as an unsuitable framework will lead to loss of time and effort. Therefore before
selecting a tool, one must have a clear idea about the organization’s maturity, areas within the organization
where tool support will help improve testing processes.and the requirements and objectives that a tool can
meet.
2. Risk while configuring the tools in different environments – It is possible to face risks while configuring
the Appium, Selenium tools in Windows as well as Mac OS. However, configuration issues do not pose a
major risk as these can be resolved immediately. Some issues that one could face include incorrect details or
missing details in environment variables, device OS version conflicts or Xcode version conflict.
3. Risk while test execution using tools, especially in the case of mobile applications – Here are some
of the issues that we have had to face while executing scripts in mobile applications, processes thereby,
accelerating continuous delivery.

8
A test automation framework is a set of expectations, ideas, and practices which provides you support for
automation testers to run the test in a flexible manner and also helps in establishing a work platform for
automated testing. Frameworks that are application independent are preferred and testers expect the
automation tools to be compatible with the selected automation framework. Thus automation framework
is the system in which the tests are automated and the tools follow the standards and strategies that are
designed in the framework.
Previously, the environment around the test automation development did not have a structural approach to
illustrate function libraries for reporting, error handling and driver scripts. The framework plays an important
role in automation test development by providing an outline of the overall test structure, ensuring consistency
of testing and minimizing the amount of code for development and thereby enabling less maintenance.
Some companies use ‘a single standard’ test automation framework that includes a test library. The test
library is created following a standard design and development process with proper documentation, standard
scripting, and team consistency. The scripting standard should be maintained across the whole framework
behind the library creation and the project team should follow the defined scripting standards. This would
help in avoiding duplication of code and in providing consistent coding across the team. In this type of
framework scripts are separated from the data stored in different files and modifications are not made in
the script. However, the data is manipulated for testing with multiple input values. Implementation and
establishment of libraries and utility components across the teams will help in reducing the effort needed for
repeated regression suites and thus enabling maximum re-usability.
Optimization of Automation using Automation Framework
• Elements not found: This issue is faced in both Appium and Selenium where the locator fails
to find the elements. An element could not be located on the page using the given search
parameters. It could have been a locator issue or an incorrectly passed parameter.
• Issues with configuration files: This might happen in all types of tools.

9
Some of the automation frameworks that are available in the automation testing world are:
Linear Scripting – Also called as ‘Record and Playback’, model it is a simple framework model. In this model,
testers can manually record the actions and scripts are automatically generated when the recording stops.
Automation expertise is not required for this framework model.
Test Library Architecture Framework – This framework is also called as ‘Structured Scripting’ model and
testers employing this model initially perform record and playback. Later the common tasks are grouped into
functions and these functions are called by the main test script named ‘driver’. An advantage of this model is
the ease with which the scripts could be managed.
Data-driven Testing framework – Test cases reside in test scripts and test data is kept separately in an external
file (Excel files, text files, CSV Files, ODBC Sources). The variables could be loaded into the scripts from external
files when required and could be used as input and verification values. The advantages of this framework are
that the test scripts do not affect the test data and that the test scenarios can be executed by updating test
data in the external data file.
Keyword-Driven or Table-Driven Testing Framework – This framework requires one to develop data tables and
keywords. It works independently of a test automation tool used to automate them. The three components
of this framework are keyword, application map and component function. Keyword is an action that can
be performed on a GUI component. An application map provides named references for GUI components.
Component functions actively manipulate or interrogate GUI component. Data table contains the action to
be performed on GUI objects and corresponding arguments. Advantages of this framework include the re-
usability of code and test tool independency.
Hybrid Test Automation Framework – As the name suggests, it is a combination of one or more frameworks.
The main intention is to mitigate weaknesses of automation flows and other frameworks by adding more
frameworks as adding framework will improve the strength of the framework as well as the automation
flows. It is the most often implemented framework by companies.

10
Companies with an automation framework and optimized automation testing reap the following benefits:
• Establishing a single standard across the organization helps in following a standard process as
compared to pre-empted ad-hoc processes that do not yield results.
• Independence from individual coding standards and the utilities that have been created. The effort
by new members could be reduced with the help of complete documentation.
• The companies collect complete requirements for automating and providing complete test
coverage with the help of the framework. This minimizes the testing efforts during the later stages
of the releases.
• The framework would have the capacity to adopt future enhancements and ensure consistency of
testing.

11
Use Case of Different Open Source Automation Tools
Using Selenium to automate – Selenium supports multiple desktop browsers as well as mobile web. The latest
version of Selenium, 3.141.59 is observed to work great with the latest version of Firefox, however, will result
in random failure if used with Firefox35 or below. An area that is considered to be risky is the configuration
of Webdrivers for multiple browsers. However, this could be tackled through proper scripting including a
configuration file. Companies employ selenium features such as ‘WebDriverWait’, ‘ExpectedConditions’ and
‘Explicit Wait’ to ensure that the scripts do not break due to fast execution or execution of the next line
before finishing the first one. Other important features are Action Classes and JavaScript Executors that
support Scroll, Mouse Over, Ajax Handling and Alert Handling activities. Users are supposedly 90% to 95%
satisfied with the tool. Selenium provides options to locate UI elements and to compare expected test results
against actual application behavior. One of its key features is the support it provides in executing one’s tests
on multiple browser platforms. Selenium is composed of different tools with their own roles:
Selenium RC: An older concept that is not widely used in the current automating world, Selenium RC merged
with a powerful tool called Selenium Webdriver.
Selenium Webdriver: Most of the testers are using this tool for automating their web applications.
Selenium IDE: A Firefox plugin that provides an easy-to-use interface for developing automated tests. It has
a recording feature that records user actions as they are performed and exports them as usable script in one
of the many programming languages that could be later executed.
Selenium-Grid: It allows to run tests in parallel, that is, different tests could be run at the same time on
different remote machines. (Hub-Node Model)

12
The benefits of Selenium are that it is open source and supports all browsers like IE, Firefox, Mozilla, and Safari.
It also supports all operating systems and programming languages such as Java, Ruby, C#, and Python. It
enables one to run multiple tests at a time and runs test in parallel at remote machines using Selenium-Grid.
The streamline working flow of Selenium tool is as follows:
WIRE (JSON / HTTP) XPCOM
COM
webdriver.xpi
IEDriverServer.exeWIRE (JSON / HTTP)

13
The internal streamlined working of Appium tool is as follows:
Using Appium to automate – Appium helps companies to establish an automation foundation for both
iOS and Android and helps build strong relationships with big clients. It provides features like inspector
support that helps in determining the elements that are implemented in the application (say buttons, text
fields, labels, etc). It supports the features provided by Selenium and also enables ‘Record and Playback’. It
is an open source, cross-platform test automation tool for native and hybrid mobile apps that supports iOS,
Android and Firefox OS platforms. It takes selenium commands from the app and converts those into a
readable format for UI automation. It does not require recompiling or modifications to app and allows writing
tests in one’s favourite programming languages and testing frameworks (Java, Python, Ruby, Node, PHP, C #,
and Perl) using the Selenium API. It could run locally or remotely and also provides detailed reports.
Appium Instructor generates scripts and these will be sent to the Appium server. The server will generate the
corresponding script in UI automation and the UI automation script will be executed in the simulator/devices.
About 75% of satisfaction could be achieved through the usage of this tool for automating applications.
WIRE (JSON / HTTP)
WIRE (JSON / HTTP)
UIAutomator
Instruments
Test Code

14
protractor <name of the configuration file>
The important features of the tool are as follows:
Using Protractor to automate – This tool provides automation support for different AngularJS applications.
The process of protractor installation is relatively easy. Initially one could face a challenge in finding a good
IDE for protractor. Webstorm is a good IDE to configure AngularJS and Protractor concepts. This tool has
similar features as Selenium such as element locators and the like. To install Protractor first we need to install
NodeJS and configure its path properly. Later we need to open a command prompt and enter the following
command to install the protractor globally:
npm install -g protractor
We can install the protractor locally in our project directory. For that we need to type the following command
in the command prompt of the project directory:
npm install protractor
All the tests are created in protractor in the form of JavaScript files and run using the following command in
the command prompt:
• An end-to-end testing framework for AngularJS applications and works as a solution integrator by
combining powerful tools and technologies such as NodeJS, Selenium, WebDriver, Jasmine, and
Cucumber.
• Protractor speeds up testing by avoiding the need for lots of ‘sleeps’ and ‘waits’ in tests as it optimizes
sleep and wait times.
• Built on the top of Webdriver JS and Selenium server, it provides new locator strategies and functions
that are helpful to automate the AngularJS application.
• Introduced new and simple syntax to write tests and allows to run tests targeting remote addresses.
• It has the ability to take advantage of Selenium-Grid to run multiple browsers at once.

15
The streamline working flow of Protractor is as follows:
Benefits of QA Automation Testing
How profitable is switching to ‘Automation Testing’ is one of the most frequently asked questions. Here are
some of the benefits rendered by automation testing that helps in the growth of businesses from all over
the world.
1. Cost Optimization – Handling customer issues, escaped bugs and poor performance have been reported to
be some of the most time-consuming tasks. Switching to automation reduces the time taken to run tests and
fix glitches, thereby reducing the cost incurred. Though the initial investment is high, test automation helps
in saving the project and support costs in the longer run.
2. Improved Efficiency – As mentioned earlier, manual testing is a dreary task that proves to be vexing for
the testers. The much accurate test automation is a one-stop solution that saves time and also allows the
testers to focus on other important tasks. Since testing consumes a major portion of the time of the overall
development cycle, reducing the time spent in it would ultimately result in improved efficiency.
WebDriver
Browser DriverProtractor
Node.JS
* Selenium WebDriver API
JavaScript Bindings
* Run locally (standard)
Run remotely (standard)
* Selenium WebDriver Drivers
Chrome, other implementations
IEDriverServer.exe
WebDriver.JS *
Selenium Server *
Test Server Browser

16
3. Reduction of Defects – Unlike manual testing, test automation efficiently detects escaped bugs and
run routine checks to improve performance. This fuels the development speed and improves quality while
ensuring that there is a reduction in the number of glitches the next time. The early detection of the bugs
make them easier to fix and also reduces the money spent on it.
4. Faster Time-to-Market – Organizations that employ manual testing take longer time-to-market as they
spend a lot of their time on repetitive testing. This time is saved in test automation where there are certain
tools and scripts to ensure that the quality of standard is maintained. With faster release cycles and faster
feedback on quality, test automation reduces the time-to-market by a considerable amount.
5. Increased Test Coverage – Another feature that sets apart test automation from manual testing is the fact
that the former has the ability to examine an application to ensure that it is working as expected. Also, the
facility to execute several complex tests increases the overall test coverage which is otherwise impossible with
manual testing.

17
Author
Conclusion
If you’d like to know more about Automation Testing, please reach out to us at
contactus@rapidvaluesolutions.com
Sanoj Swaminathan
Test Architect,
RapidValue Solutions
With the reliability and efficiency that is promised through test automation, business value is
expected to skyrocket with its implementation. DevOps and Cloud Computing have become
familiar names in the IT household as companies compete to outrun each other by adopting the
latest technologies. The process of development, operation and quality assurance has undergone
a complete makeover with automation and it has been so widely adopted that quality assurance
cannot be imagined without test automation. In addition to that, the use of open source testing
tools has made software to be accessible to all. They provide the freedom to work without
expensive licenses and are regularly modified by testers. Since choosing a tool is a very crucial part
to understand requirements, design test cases and much more, it is advised to choose an open
source tool that is user-friendly. While some developers are still apprehensive of switching to test
automation, the initial costs are quite negligible when compared to the ease and convenience
that comes with it.

18
Disclaimer:
This document contains information that is confidential and proprietary to RapidValue Solutions Inc. No part of it may be used,
circulated, quoted, or reproduced for distribution outside RapidValue. If you are not the intended recipient of this report, you are
hereby notified that the use, circulation, quoting, or reproducing of this report is strictly prohibited and may be unlawful.
A global leader in digital transformation for enterprise providing end-to-end
mobility, omni-channel, IoT and cloud solutions. Armed with a large team of
expertsinconsulting,UXdesign,applicationdevelopment,integrationandtesting,
along with experience delivering projects worldwide, in mobility and cloud, we
offer a wide range of services across industry verticals. We deliver services to
the world’s top brands, fortune 1000 companies, Multinational companies and
emerging startups. We have offices in the United States, the United Kingdom,
Germany and India.
www.rapidvaluesolutions.com
+1 877.643.1850
www.rapidvaluesolutions.com/blog
contactus@rapidvaluesolutions.com
©RapidValue Solutions July, 2019

An Essential Guide to Effective Test Automation Leveraging Open Source

An Essential Guide to Effective Test Automation Leveraging Open Source

An Essential Guide to Effective Test Automation Leveraging Open Source

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados(20)

Similar a An Essential Guide to Effective Test Automation Leveraging Open Source

Similar a An Essential Guide to Effective Test Automation Leveraging Open Source(20)

Mais de RapidValue

Mais de RapidValue(20)

Último

Último(20)

An Essential Guide to Effective Test Automation Leveraging Open Source