For Impetus’ White Papers archive, visit- http://www.impetus.com/whitepaper The paper provides insights on the various benefits of using the Cloud for Performance Testing as well as how to address the various challenges associated with this approach.

1. Harnessing the Cloud for
Performance Testing
W H I T E P A P E R
Abstract
In this white paper, Impetus talks about how the infinite
computing power of a Cloud infrastructure can be leveraged for
flexible, scalable and affordable load testing.
The paper provides insights on the various benefits of using the
Cloud for Performance Testing as well as how to address the
various challenges associated with this approach. It discusses the
different models where performance testing tools can be
deployed on the Cloud, in addition to some best practices and
real-world experiences.
Impetus Technologies, Inc.
www.impetus.com

2. Harnessing the Cloud for Performance Testing
2
Table of Contents
Introduction...........................................................................................................3
The challenges and consequences of the growth in data and the user base.......4
Harnessing the power of the Cloud.......................................................................5
The flip side of Cloud and performance testing ....................................................6
The tool deployment models for Cloud-based performance testing ...................7
The Impetus solution...........................................................................................10
Summary..............................................................................................................11

3. Harnessing the Cloud for Performance Testing
3
Introduction
Cloud-based performance testing has suddenly become a major buzzword
today. From startups, to large enterprises, everyone is trying to make the most
of the opportunities offered by the Cloud.
The fact is that that there are several distinct benefits that Cloud infrastructure
offers organizations. It enables faster-time-to-test, is highly scalable and an on-
demand solution. The Cloud also reduces the overall performance testing costs.
It is important however, to understand that why the Cloud is emerging as an
important paradigm in performance testing.
Trends show that the Internet is growing at an unprecedented rate. Globally
there are close to two billion internet users, with Facebook alone crossing the
500 million user mark and Twitter receiving millions of tweets every month. The
use of the Web and mobile applications has also increased phenomenally, with
the availability of devices like smartphones, netbooks and iPads.
This scenario, while beneficial for end users, is creating several challenges for
application architects and developers, who have to ensure that applications are
performing efficiently and effectively under this huge, increasing load. The
concern is not only the two billion users on the Internet, but also the increase in
the size of the data.
Today, data sizes within organizations have gone up from hundreds of millions
of rows, to billions of rows per day.
Internet
Growth
Big Data Challenges and
Consequences
Moving to Cloud

4. Harnessing the Cloud for Performance Testing
4
The Challenges and Consequences of the
Growth in Data and the User Base
Businesses acknowledge that application performance directly affects their
bottom-line. Take the instance of a company operating within the advertising
domain. The amount it earns is directly related to how fast its ads are getting
displayed on the Website.
In this environment, when companies are not sure about the performance
parameters and application scalability, moving to the Cloud ensures effective
management of performance requirements.
Exploiting the power of the Cloud for Performance testing is gaining immense
popularity. Amazon provisioned 18 million EC2 servers last year. Also, the
number of users subscribing to other Cloud vendors such as Microsoft has
increased significantly.
Before discussing the benefits of Cloud-based performance testing, it is
important to examine the performance testing model, currently in use.
The performance testing model involves various components such as the
performance testing tools, the system under test, etc. that are deployed ‘on
premise,’ within a local environment.
One of the first steps in conducting a performance testing assignment within a
local environment is the setting up of a test lab. The lab requires infrastructure,
servers, hardware and software to host the system under test, and the
performance testing tools. Setting up such a lab requires considerable
investment and effort, and the management of overheads such as maintenance
costs. An organization setting up such a lab will need to hire administrators to
maintain and support the lab. This will further involve training costs for the
resources, who will perform the support role.
Also, with the substantial increase in Internet traffic, both in terms of the
number of users and the size of the data, the critical challenge for the
performance testing team is to scale and test immediately to higher
concurrency levels.
This typically calls for additional hardware, tool license extensions, and setting
up the tools and other software on the new hardware, resulting in additional
capital expenditure which may lead to a delay in getting the expected
performance numbers.
It has been observed
that the downtime of a
Website and the latency
of even milliseconds can
result in revenue losses
and customer
dissatisfaction. Thus
application
performance does
matter, and
organizations do
consider it as one of the
key attributes of their
product strategies.

5. Harnessing the Cloud for Performance Testing
5
Another significant aspect of the on-premise model is that it does not measure
up to the geographical distribution. An application will get hits from all corners
of the world and in order to satisfy users, the performance of the application
needs to be measured in peak traffic. The simulated load should mimic different
geographies for a more accurate simulation of real usage. Furthermore, load
generators are an important component of performance testing and in case of
higher volumes, need to be set up in larger numbers.
The good news is that these issues can be addressed by the Cloud
infrastructure.
Harnessing the Power of the Cloud
The Cloud can help organizations scale for higher volume, higher transactions
and higher concurrency, quickly and affordably. It helps in addressing the
limitations indicated above and offers enterprises distinct advantages, including
the following:
Infinite computing power: The Cloud has the capability to simulate millions
and millions of users from thousands of instances. This helps in testing the
applications with production data volumes and concurrent users.
Affordability: The Cloud helps in saving the upfront costs in setting up the
performance testing environment as the Controllers and load generators
can be instantiated, as and when required.
The pay-as-you-go benefit: In the case of the Cloud, organizations need
only pay for the duration the machines are being used. Thus, by planning
the tests, companies can save costs for using the hardware for a longer
period of time. The instances can be created and configured at the time of
use and terminated after running the tests. With the Cloud, companies do
not need to worry about the licensing costs of the software, OS,
performance testing tools, etc.
Easy set up: The Cloud is quick and easy to set up and the test environment
can be up and running in minutes. Many vendors provide automatic
provisioning of the load generators that helps in quickly setting up the
environment and starting the test executions. The Cloud also offers a ready-
to-use environment where no separate installation and configuration of
performance testing tools is required. The instances are pre-configured as
load generators or controllers.
Ability to choose geographies: The Cloud makes it possible for
organizations to choose geographies, so that instances are created and the

6. Harnessing the Cloud for Performance Testing
6
load simulated from different locations. The result is that there is a much
more accurate simulation of production usage of the application under test.
Clearly, Cloud-based performance testing can be beneficial for Internet and
mobile applications that have a large customer base and huge transaction
volumes.
Though the Cloud offers lots of advantages, there are a few challenges that
need to be addressed, before performance testing can become viable.
The Flip Side of Cloud and Performance Testing
Despite its advantages, the Cloud has its share of concern areas. Data Security
is a major issue for any performance testing activity that is performed on the
Cloud, since all the test scripts, test data, server response and test results have
been moved to the Cloud. Today, however, organizations can have a private
Cloud, and set up a separate tunnel for security.
Another problem area is testing intranet applications on the Cloud; where there
is need to work with the client’s IT department in order to receive permission to
access the application from the Cloud. Network latency is another critical factor
that may skew the performance numbers. It poses a challenge for intranet
applications, but can be resolved to an extent by statistically adjusting the
response times. Also, for intranet applications, inclusion of latency in the test
results provides a realistic view of applications performance.
During the performance tests, resource monitoring and diagnostic activity is
data intensive. In case of the Cloud, data transfer that happens IN and OUT of
the Cloud is huge and may pose a challenge.
Since the Cloud instances may not be available all the time, it is important to
have a pertinent test data management strategy, and a proper back-up
mechanism for test scripts, test data and test results.
Test script creation on the Cloud is a tough task as it involves parameterization,
validation and debug runs. Ideally, the script creation should be done in a local
environment, with the scripts being uploaded on the Cloud for full load tests.

7. Harnessing the Cloud for Performance Testing
7
The Tool Deployment Models for
Cloud-Based Performance Testing
The Cloud model
There are some standard components of the performance testing tool
architecture like controllers, load generators, recorders and analyzers. The
variations in the deployment models depend on where all these components
are placed while the performance tests are being conducted. Organizations
have to choose the appropriate deployment model based on their needs.
In the case of the Cloud model, the performance testing tool is moved to the
Cloud and therefore, the controller, load generators and analyzer are also
deployed on the Cloud. The Recorder however, is provided on-premise so that
users can record the test script, debug and validate them and then further push
to the Cloud for load testing.

8. Harnessing the Cloud for Performance Testing
8
There is an option of having the recording component on the Cloud; however
that might result in extra hits to the Cloud environment during script creation. It
is recommended that the scripts are recorded locally and then moved to the
Cloud for scaling.
This model is suitable for performance testing of Cloud-based applications or
Internet applications, since these applications get hits from all parts of the
world. The scenarios can be designed in a way to simulate realistic load.
For instance, in case an Internet application is deployed on the data center,
where 50 percent of the traffic is coming from the east coast, 20 percent from
the west coast and remaining from the Asia Pacific region, it should be possible
to spawn virtual users from the Cloud-based load generators instantiated from
these regions.
This represents realistic simulation as compared to the on-premise performance
testing model. It can also provide as many load generators as required, quickly
and on demand, without the need for hardware procurement, and therefore no
massive expenditure on setting up the infrastructure.
Another benefit is that there is no need to install and configure the performance
testing tool and other software. There are no tool license costs involved. The
process becomes simplified and the organization is only charged for the period
that it uses the Cloud infrastructure.
The Hybrid model
In this case, the components of the performance testing tool are deployed
either on the Cloud or on-premise. There are many combinations possible here,
but it is feasible to take into account the most likely scenario.
Here, the load generators are on the Cloud and all the other components, such
as the controller, recorder and analyzer are on-premise.
This model is more suitable for organizations which have already invested in
performance test labs and have a hardware setup. These companies can use the
hybrid model when they need to scale quickly.
Often regular performance testing can be done on-premise, not only to get the
performance numbers, but also to determine the performance bottlenecks in
the application. However, in case there is a requirement for testing an
application with peak volumes, high concurrencies and more transactions, the
Hybrid model can be used.

9. Harnessing the Cloud for Performance Testing
9
In the Hybrid model, the controller is deployed on-premise and enables the use
of the integrated resource monitoring and diagnostic features of the
performance testing tool. This also addresses any security concerns over crucial
data, as the controller remains in-house. Since the data is available locally, it can
archive and be referred to at any suitable time.
In case of resource monitoring and diagnostics on the Cloud, there will be a
huge data transfer from and into the Cloud as there is pooling of parameters
like memory, CPU, network bytes every 5 or 10 seconds.

10. Harnessing the Cloud for Performance Testing
10
The Impetus Solution
Impetus Technologies has developed SandStorm, an enterprise performance
testing tool, which can be deployed on the Cloud. Using this solution,
organizations can reduce their performance testing costs by 60 percent.
SandStorm is pre-configured on the Cloud instances and offers a ‘Ready-to-use’
test environment. Moreover, the test scripts can also be created locally using
the Recorder and automatically uploaded to the Cloud before running the tests.
SandStorm supports multiple protocols ranging from the Web, mobile and e-
mail like Http, Https, FLEX, Silverlight, Java applets, OMA IMPS, WAP, WV, and
SMTP etc. It offers integrated resource monitoring for most of the popular
applications, Database servers and Operating Systems. It provides the User
Interface wizards for script creation and configuration.
The analysis dashboards provide a summarized view of test results with drill
down capabilities to identify performance bottlenecks (visit
sandstorm.impetus.com).
Case Study: Performance testing using the Cloud
The customer: A leading provider of workforce management solutions ranging
from payroll, leave, schedule, among others. The company offers performance
benchmarking services to its customers wherein the performance testing
activity is conducted at the facilities of its customers.
The need: The customer requested Impetus to come up with a more affordable
and flexible solution for its clients.
The solution: Impetus suggested using SandStorm on the Cloud. The Recorder
component was locally deployed in the test labs to create and verify test scripts.
These scripts were automatically uploaded to the Cloud for test executions. The
client exposed the application to the Cloud by setting appropriate permissions
in the firewall. The performance testing could be easily completed in time for all
the business scenarios. The network latency was adjusted based on the network
correction factor identified with the help of tracert utility.
The benefits: The solution helped the customer to increase customer
benchmarking activities. This in turn helped the company to achieve higher
revenues and improve customer satisfaction.

11. Harnessing the Cloud for Performance Testing
11
Our Recommendation
Summary
In conclusion, it can be said that moving to a Cloud-based performance testing
model offers significant benefits to organizations. At the same time, when
moving to this model, it is important that companies pay attention to some of
the following recommendations:
• Understand the performance goals and objectives. In the scenarios,
where there is no certainty on the peak usage, it is prudent to opt for a
Cloud strategy.
• Check to see whether the organization has uncertain demand. The
elasticity from the Cloud infrastructure provides organizations with a
mechanism to deal with uncertainty. Moving to the Cloud for instance,
saves companies from making capital expenditure and procuring
additional hardware if the existing set-up does match peak usage, or in
case the peak demand does not reach the expected capacity.
• Think from the end user perspective. Cloud-based applications and
Internet applications have to be accessed globally and companies need

12. Harnessing the Cloud for Performance Testing
12
to define proper workload patterns to simulate the realistic load. The
application must be tested at both peak and normal traffic levels from
all the targeted geographical locations.
• Look at the type of the application that can move to the Cloud. This
approach is useful for applications that have high concurrent users, big
data problems and mobile applications accessible from the Internet.
• Choose appropriate tools and models for the Cloud-based
performance testing. The models and tools should be chosen based on
the type of application, user base, non-functional requirements,
scripting requirements, and test data management.
• Analyze performance. The Cloud environment might be shared.
Different vendors have different mechanism of providing the virtual
instances on the Cloud. In case, the underlying hardware is shared,
analyze the performance and resource consumption. Any unexplained
changes in performance may be attributed to either shared resources or
the Cloud provisioning tool.
• Have continuous performance management in place. Performance
testing can be a part of continuous integration and can automatically
execute critical test scenarios with intermediate builds. The process may
use automatic provisioning to generate Controller and load generator
instances based on the peak usage of the application. These can
increase the efficiency of the entire process.
About Impetus
Impetus is a Software Solutions and Services Company with deep
technical maturity that brings you thought leadership, proactive
innovation, and a track record of success. Our Services and Solutions
portfolio includes Carrier grade large systems, Big Data, Cloud,
Enterprise Mobility, and Test and Performance Engineering.
Website: www.impetus.com | Email: inquiry@impetus.com
© 2013 Impetus Technologies, Inc.
All rights reserved. Product and
company names mentioned herein
may be trademarks of their
respective companies. May 2013