Automation doesn't automatically make software testing faster, more reliable and less expensive. Up-front costs in automation tools and set-up can be high. In addition, not all automation tools and methodologies have the same features, and each project may have different requirements. To help you get the maximum benefit from automated testing, Cognizant offers eight tips based on our experience with more than 50 enterprise-level automated testing projects, worldwide. Following these tips can help increase the ROI of test automation and improve software quality.

1. Cognizant White Paper
Eight Steps to Better
Test Automation
Automation doesn’t automatically make software testing faster, more reliable
and less expensive. Because the up-front costs in automation tools and set-up
can be high, automated testing only pays off if the long-term cost savings offsets
those initial expenses. In addition, not all automation tools and methodologies
have the same features, functions and capabilities, and each project may have
different requirements that affect its costs and benefits.
To help you get the maximum benefit from automated testing, we offer eight
tips based on our experience with more than 50 enterprise-level automated
testing projects, worldwide. Following these tips can help increase the ROI of
test automation and improve software quality.
Top Eight ways to achieve efficient test automation.
whitepaper

2. One: Choose What to Automate such as Windows and Linux, but often
cannot do the same for controls
Not all software projects lend themselves created by development tools
equally to testing. Understanding which vendors such as Infragistics, and
factors increase the difficulty of automating application vendors such as Oracle
a specific test project is essential to and Siebel.
balancing the costs against the benefits. • Code or test cases for which the
More appropriate candidates for automated production or infrastructure teams
have not supplied the right type of
testing include code that:
data, or the proper amount of data,
to support a full-scale automated
• Plays an important role in an
test program.
application
• Code which changes often, since
• Processes large amounts of data
such changes may also require time-
• Executes common processes
consuming manual revisions to either
• Can be used across applications
the test script or the test data.
We have developed a nine-point decision tree (see
Nine-Point Test Decision Tree
Figure 1) that helps clients select automation
candidates based on the following criteria:
• Technical feasibility
• The frequency of test execution
• The extent to which test components
can be reused
• Total resource requirements
• The complexity of the test cases
• The ability to use the same test cases in
multiple browsers or environments
• The time required to execute the tests
Two: Choose Your Test Tools
Because companies must amortize their
investment in automation, they should select
Figure 1 automated test tools that will meet their needs
for years. Proper evaluation criteria include:
• Support for various types of automated
Organizations should also look for non-
testing, including functional, test
traditional and even unplanned areas, over
management, mobile, SOA and IVR
which they can spread their automation
(interactive voice response) software.
investment. These include automating the
• Support for multiple testing frameworks
testing of installation routines for patches and
(the methodologies that enable the re-
defect fixes, automating test management,
usability and efficiency of automation).
and the creation of test reports.
• The ability to recognize objects created
in a wide variety of programming
Less appropriate factors for automated testing
languages.
include:
• Stable setup and operation on any
environment/platform.
• The extensive use of non-standard
• Ease of debugging the automation
software controls (the components
scripts.
that define the user interface).
• Efficient test execution with a minimum
Automated test tools typically can
of manual effort.
assess the quality of standard
• Automatic recovery from application
controls within operating systems
2 whitepaper

3. failures to prevent test interruptions. the required licenses are purchased.
A powerful scripting language that Among the open-source tools we have
makes it easy to develop scripts (the customized are the Selenium suite for
instructions carried out in a specific automated testing of Web applications and
test) that can be reused across the Bugzilla defect tracking tool. We have
platforms and test types. also created a framework based on the WATIR
• The ability to report results, and be (Web Application Testing in Ruby) toolkit used
controlled by, a wide variety of test to automate browser-based tests during Web
management platforms. application development.
• Ease of use to minimize training costs
and delays. Tool Selection
There are several commercial tools that
provide these capabilities, but they are
expensive. The decision to procure them may
solely depend on the proposed investment,
and as mentioned above, after considering
future requirements. Customizing such open-
source tools to meet specific testing needs
can be an effective way of reducing costs.
Another way to speed ROI from automated
testing is to use inexpensive software tools
that, while not designed for that purpose, can
be useful in automating certain scenarios.
Figure 2
Examples:
• Use of the macro-recording Three: Refine Your Test Process
capabilities that ship with most
operating systems to automate In many organizations, the lack of centralized,
processes such as loading test data, standardized automation processes has led to an
scheduling, and executing tests. elongated testing lifecycle that is prohibitively
• Free or low-cost file comparison expensive and fails to detect the maximum
utilities can be used to determine number of defects. Improving such processes
whether actual test output matches requires the following (see Figure 3):
the expected results at far lower
cost than using commercial off-the- • Describing the risks of current
shelf test automation tools. methods, and showing how testing
can be done at a lower cost, more
We have developed a methodology to help quickly and/or more effectively.
customers choose the test automation software • Management commitment to provide
most appropriate to their needs, and to make the funding and support for changing
the most effective use of both new and existing workflows required to improve
automated test tools (see Figure 2). It begins testing processes.
with defining the objectives for the tools and • Avail support from the test staff for
specifying the tests to be automated, such as meeting improvement goals.
functional testing or back-end validation. The • Training to provide specialized skills
next step is to define the requirements, build to test managers, specialists and
an evaluation scorecard, perform a proof testing engineers in testing
of concept, and finally prepare the tools for methodologies.
deployment. This methodology also helps • Prioritizing process improvements
customers optimize their use of automated based on business goals.
test tools by identifying all testing needs • Ongoing measurement of test
across the organization, creating an inventory processes to achieve higher return
of tools already available, and reviewing on investment.
existing licensing agreements to ensure only
whitepaper 3

4. A Standardized Test Automation Framework A Hybrid Approach to Test Automation
Figure 3
Four: Choose a Framework
Just as with any tool, automation test
solutions must be used correctly to Figure 4
be effective. Choosing an appropriate
framework can help boost long-term It allows multiple test cases to be driven with
reusability and efficiency. multiple sets of data, and parameterizes the
data to ease maintenance and modification.
A framework is not a replacement for It drives efficiency by allowing business
an automation tool, but a roadmap for users to design and change tests without
its effective use. It should also allow for learning complex scripting languages. The
parameterization (separate storage) of test use of keywords makes it easy to measure
scripts and test data to allow the maximum how much of the code required to perform
reuse and easy maintainability of both. In common business functions, such as
this way, when changes are made to a test “validate applicant’s age,” is reused over
case, only the test case file needs to be time.
updated, while the driver and start-up scripts
remain the same. Similarly, the test data can Five: Don’t Underestimate the Manual
be changed as needed without requiring Effort
changes to the test scripts.
The word “automation” implies that machines
Among the popular frameworks are those do the work, not humans. But the amount
that are “data driven,” where the test data of manual effort required in “automated”
is stored separately from the automation testing is one of the least understood aspects
tool. This provides significant ease of use of software testing. Humans must set up the
and customization of reports, eases data test machines, create the scripts, select the
maintainability, and allows multiple test tests, analyze the test results, log the defects
cases to be conducted with multiple sets and clean up the test machines. An accurate
of input data. However, the initial cost and estimate of these costs is important not only
maintenance overhead can be considerable. for budgeting and planning, but also to build
Another approach is “keyword-driven” an accurate return on investment calculation.
in which data tables and keywords are
maintained independently of the automation In our client work, we have identified the
tool and the test scripts that drive the tests. factors IT organizations should take into
But this is somewhat harder to use than the account in estimating the manual effort
data-driven approach, which increases costs required for test automation. These include
and delays. the complexity of the language used to
create test scripts, and the amount of
We have developed a hybrid framework (see work required to plan, generate, execute
Figure 4, above) ,that combines the best and maintain the scripts. Another aid to
elements of both the keyword- and the data- estimating effort is classifying test cases as
driven approaches. It stores the test data simple, medium and complex based on the
apart from the automation tool, (usually in number of transactions and the number of
an Excel spreadsheet), making it very easy to steps within the scripts required by each
maintain and reuse the scripts. case.
4 whitepaper

5. Six: Watch Those Scripts Seven: Identify Who Will Execute the Tests
Areas in which standards are particularly im- Automation raises the expectation among
portant include the following (see Figure 5): managers that testing can be achieved with
• Exception handling, which determines little or no manual effort. Thus, they do not
whether, and how well, the test script allocate the staff required to perform the
can recover from the failure of the manual steps required in automated testing,
application being tested, or unexpected such as analyzing test results and creating
behavior such as the appearance of a and cleaning up test machines. Many of the
pop-up. The use of standards helps likely candidates for such work are either
ensure all scripts continue to run in unavailable or lack the proper skills.
the event of unexpected conditions,
reducing the need for manual For example:
intervention and speeding test execution. • Functional testers already have their
• Error logging, in which standards time allocated to manual testing.
make it easier for multiple developers and • Automation engineers, while qualified
testers to analyze and act on the to run the tests, typically don’t
test results. understand enough of the functional
• Documentation standards in areas requirements to analyze the failed
such as comments and indentation scenarios. They are too busy
help developers create uniform scripts, developing and maintaining tests to
and to understand script code created execute the tests and track defects.
by others. • Without planning for the proper
numbers of skilled staff, an automated
testing program will fall victim to
Scripting for Success unexpected outages, delays and cost
overruns. The organization will
not be able to run enough tests
frequently enough to justify its
investment in test automation.
That’s why it is important to determine
who will own, and execute all the processes
around the actual automated testing at the
beginning of the process. As an experienced
test automation consulting firm, we can help
customers make sure they have the right
amount of skilled staff to cost-effectively en-
sure software quality. This includes accurate-
ly assessing the manual requirements at all
stages in the automatic testing process, and
Figure 5 making better use of existing staff. Functional
testers can be retrained, for example, to both
Along with the development and use of execute and maintain scripts (see Figure 6):
scripting standards, we have developed
consolidated toolsets for script creation that
have saved customers 10% to 15% in script-
ing costs by allowing greater script reuse.
whitepaper 5

6. Resourcing the Test Automation Function
testing.
• Reduced time to market.
• Improvements in customer satisfaction.
• Increased productivity due to improved
application quality.
Figure 6 All these are real benefits that must be
balanced against staffing and tool licensing
Eight: Measure Your Success -- Accurately costs to determine an accurate return on
investment.
Tools such as the Automated Execution
Efficiency Ratio (AEER) can determine how Measuring what Matters
effectively you are executing automated
tests by analyzing the ratio of human effort
as a percentage of the total effort needed to
execute the automated tests.
But don’t just compare the number of
person-hours required for manual vs.
automated tests for a given size of the code.
Be sure to include other benefits such as
the following (see Figure 7):
• A higher percent of defects found.
• Reductions in the time needed for
Figure 7
Real-World Example:
A global financial services company turned to Cognizant to speed its testing of business-critical code, and to improve the return on its
automation investment through the reuse of test components. Cognizant helped the client establish a central function to perform automated
functional and regression testing of key modules such as trading, payments, accounts, safekeeping and wealth management, and to implement
an automation architecture.
Cognizant reduced the overall time required for testing by 72%, and the effort required to design and execute multilingual tests by 70% by
creating a test quality center and developing a framework to maximize component reuse and the ROI of automated testing. The client has
automated 43% of its regression testing, reuses 60% of its automation scripts, and has extended the benefits of this work by implementing the
automation framework on multiple projects. Cognizant also developed a dashboard that provides business and technical managers measurable
performance metrics through dashboards showing weekly progress on key performance indicators and service level agreements.
Conclusion:
Automating software testing is not as simple or as quick as the term implies. Software test tools can be expensive, and setting up, executing
and analyzing test results requires extensive manual work. But by using the proper tools such as open-source software, and the proper
processes and automation frameworks, organizations can realize the cost savings and quality of automated software testing.
About Cognizant:
Cognizant (NASDAQ: CTSH) is a leading provider of information technology, consulting, and business process outsourcing services. Cognizant’s
single-minded passion is to dedicate our global technology and innovation know-how, our industry expertise and worldwide resources to
working together with clients to make their businesses stronger. With over 50 global delivery centers and more than 68,000 employees as of
September 30, 2009, we combine a unique onsite/offshore delivery model infused by a distinct culture of customer satisfaction. A member
of the NASDAQ-100 Index and S&P 500 Index, Cognizant is a Forbes Global 2000 company and a member of the Fortune 1000 and is ranked
among the top information technology companies in BusinessWeek’s Hot Growth and Top 50 Performers listings.
Visit us online at www.cognizant.com.