Performance Test WCF/WPF app - Selecting right Tool

Software Quality Assurance Plan
For WCF/WPF based Application
Version 0.1
May 15, 2014
Compiled by:
Kamran Shaukat Ali Khan
Chrome Technologies
ChromeIS.com

Performance Test Plan – Ascent –Version 0.1
Table of Contents
Document Information................................................................................................vi
Document Information.................................................................................................vi
Document Review Information.....................................................................................vi
Document Review Information.....................................................................................vi
Document Revision History..........................................................................................vi
Document Revision History...........................................................................................vi
List of Figures..........................................................................................................vii
List of Figures.............................................................................................................vii
List of Tables............................................................................................................vii
List of Tables...............................................................................................................vii
1. Abstract.....................................................................................................................1
1.1 Proposed Solution...................................................................................................1
2. Performance Testing Tools Evaluated.........................................................................2
2.1 HP LoadRunner and Performance Center....................................................................2
2.2 IBM Rational Performance Tester..............................................................................3
2.3 Borland Silk Performer............................................................................................3
2.4 Telerik Test Studio..................................................................................................3
2.5 Apache JMeter.......................................................................................................3
2.6 j-Chav..................................................................................................................4
2.7 Microsoft WCAT – Web Capacity Analysis Tool (formerly WAS – Web Application Stress
Tool)..........................................................................................................................4
2.8 Sun VisualVM.........................................................................................................4
2.9 WAPT...................................................................................................................5
2.10 Web Application Load Simulator (LoadSim)...............................................................5
2.11 Neo Load.............................................................................................................5
2.12 EggPlant Load Tester.............................................................................................5
2.13 OpenSTA – Open System Testing Architecture .........................................................6
2.14 OpenWebLoad......................................................................................................6
2.15 OpenLoad............................................................................................................6
2.16 loadUI.................................................................................................................6
2.17 LoadeaTest..........................................................................................................6
2.18 LoadManager.......................................................................................................7
2.19 Test Automation FX...............................................................................................7
2.20 CodedUI..............................................................................................................7
2.21 Soap UI...............................................................................................................7
2.22 Soap-Stone.........................................................................................................7
2.23 Web Performance Suite.........................................................................................8
2.24 Webserver Stress Tool...........................................................................................8
2.25 WebLOAD............................................................................................................8
2.26 QALoad...............................................................................................................8
2.27 Lisa Load.............................................................................................................8
2.28 Grinder...............................................................................................................9
2.29 nGrinder..............................................................................................................9
Copyrights © NetSol Technologies Ltd. All rights reserved iii
Unauthorized copy or use of this document is strictly prohibited

2.30 GrinderStone.......................................................................................................9
2.31 MStone...............................................................................................................9
2.32 ValGrind..............................................................................................................9
2.33 QEngine..............................................................................................................9
2.34 Qtest................................................................................................................10
2.35 HammerHead 2..................................................................................................10
2.36 Hammerora........................................................................................................10
2.37 Pylot.................................................................................................................10
2.38 OpenWebLoad....................................................................................................10
2.39 SD’s Profiler.......................................................................................................11
2.40 Benerator..........................................................................................................11
2.41 SOATest (by Parasoft).........................................................................................11
2.42 Scapa Test and Performance Platform ...................................................................11
2.43 CLIF Load Injection ............................................................................................12
2.44 ContiPerf...........................................................................................................12
2.45 Coveriety...........................................................................................................12
2.46 D-ITG (Distributed Internet Traffic Generator).........................................................12
2.47 Database Opensource Test Suit (DOTS).................................................................13
2.48 DB Monster........................................................................................................13
2.49 Deluge..............................................................................................................13
2.50 DieselTest..........................................................................................................13
2.51 Faban................................................................................................................13
2.52 FunkLoad...........................................................................................................14
2.53 Curl-Loader........................................................................................................14
2.54 Spirant Avalanche...............................................................................................14
2.55 IXIA IxLoad........................................................................................................14
2.56 Paessler.............................................................................................................14
2.57 AQ TestComplete and AQ Time.............................................................................14
2.58 Test Maker.........................................................................................................15
2.59 TP Test..............................................................................................................15
2.60 Tsung................................................................................................................15
2.61 Tnsgen..............................................................................................................15
2.62 N-Time..............................................................................................................15
2.63 p-unit................................................................................................................16
2.64 Lobo.................................................................................................................16
2.65 MessAdmin........................................................................................................16
2.66 Multi Router Traffic Grapher (MRTR)......................................................................16
2.67 RRRD Tool.........................................................................................................16
2.68 Multi-Mechanize..................................................................................................16
2.69 Stress Driver......................................................................................................17
2.70 Load Impact.......................................................................................................17
2.71 Pure Load..........................................................................................................17
2.72 Pure Agent.........................................................................................................17
2.73 FWPTT...............................................................................................................17
2.74 JBlits Pro...........................................................................................................18
2.75 JCrawler............................................................................................................18
Copyrights © NetSol Technologies Ltd. All rights reserved iv

2.76 PandoraFMS.......................................................................................................18
2.77 Siege................................................................................................................18
2.78 HTTP Load.........................................................................................................18
2.79 httPerf...............................................................................................................19
2.80 iPerf..................................................................................................................19
2.81 Site Blaster........................................................................................................19
2.82 RawLoadTester...................................................................................................19
2.83 Allmon..............................................................................................................20
2.84 AppsWatch.........................................................................................................20
2.85 BurnIn Test........................................................................................................20
2.86 Capacity Calibration (CapCal)...............................................................................20
2.87 Chariot..............................................................................................................21
2.88 CitraTestVU........................................................................................................21
2.89 SSW Performance Pro..........................................................................................21
2.90 e-Load..............................................................................................................21
2.91 MLoad...............................................................................................................21
2.92 Monitor Master...................................................................................................22
2.93 Forecast............................................................................................................22
2.94 GS DataGenerator...............................................................................................22
2.95 vPerformer.........................................................................................................22
2.96 JUnit.................................................................................................................22
2.97 Xceptance LoadTest............................................................................................23
2.98 zapwireless........................................................................................................23
2.99 Seagull..............................................................................................................23
2.100 SIPp................................................................................................................23
2.101 SLAMD............................................................................................................23
2.102 IxoraRMS.........................................................................................................24
2.103 j-hawk.............................................................................................................24
2.104 Wireshark........................................................................................................24
2.105 Ostinato...........................................................................................................24
2.106 Eclipse Test & Performance Tools Platform Project (EPTP)........................................24
2.107 Web Polygraph.................................................................................................24
3. Tools Selection Criteria.............................................................................................25
3.1 Business and Technical Requirements......................................................................25
3.2 Performance Test Objectives..................................................................................25
3.3 Performance Test Execution...................................................................................28
3.4 Load Description...................................................................................................28
4. Conclusion................................................................................................................29
4.1 Selected Tools......................................................................................................30
4.2 SWOT Analysis.....................................................................................................30
4.3 Training Needs.....................................................................................................31
4.4 Lessons Learnt.....................................................................................................31
5. Definition of Terms, Acronyms And Abbreviations.....................................................31
Copyrights © NetSol Technologies Ltd. All rights reserved v

Document & Data Control
Document InformationDocument Information
Category Information
<Title> Performance Test Tool Selection
<Version> 0.1
<Author> Kamran Shaukat Ali Khan
<Identifier>
<Life Cycle Activity> TLC
<Issue Date> Pending
Document Review InformationDocument Review Information
Review Date Reviewer Name Version Reference / Evidence
Document Revision HistoryDocument Revision History
Author Date Version Description
Kamran
Shaukat Ali
Khan
15 May 2014 0.1 Draft
Copyrights © NetSol Technologies Ltd. All rights reserved vi

List of FiguresList of Figures
Figure 1, Performance Acceptance at a glance..............................................................27
List of TablesList of Tables
Table 1. Tools fostering Performance Testing phases.....................................................2
Table 2. Selected tools for Performance Testing phases................................................30
Copyrights © NetSol Technologies Ltd. All rights reserved vii

Performance Testing – Tool Selection
1. Abstract
OUR PRODUCT is engineered for performance and agility.
Performance testing is the process of assessing performance/efficiency of a transaction system
whilst simultaneous users are accessing it; this is to determine if subject application (with all factors
involved) can serve its purpose efficiently. Performance Testing is a fundamental aspect of
software engineering, but it is a practice that too often is easily forgotten in today’s fast-paced
application development culture. Performance testing applications efficiently is essentially an
integral component for meeting end customer expectations.
The aim of this document is to evaluate and compare all available performance testing tools to
determine their usability and effectiveness. Software testing is a crucial part of software
development and the process of automating software testing is vital to its success. OUR PRODUCT
is a complex system and performance testing throughout the software development cycle is
pertinent to the success of the software.
To enable comparison of the automated performance testing tools, a multi-partitioned metric suite
was developed to guide the tool evaluation process. Inferences, implications and results are
presented and discussed in Section 3.
1.1 Proposed Solution
The overall goal of this research is to evaluate and compare several automated performance
testing tools to determine their usability and effectiveness for OUR PRODUCT. To accomplish this
goal, the research:
1. Prepared a comprehensive performance testing tools available on internet.
2. Broadened the scope of tools to include shareware, freeware, Open licenses and
open source tools. The list of tools, along with tools’ assessment is described in Section
2.
3. Defined criteria for tools selection based on business and technical requirements of
OUR PRODUCT; this will help narrow down the scope of research.
4. Compared existing tools in practice at ChromeIS with evolving requirements as
highlighted in above point.
5. Performed assessment of tools based on features and later, experiment with actual
application i.e. OUR PRODUCT (for e.g. CMS)
6. Tested the target applications using the selected automated testing tools and gather
resulting data
7. Evaluated and interpret results to draw inferences and make recommendations
OUR PRODUCT is a smart client being developed in Visual Studio 2010 using.net framework 4.5 – this imposes
certain limitations. An RND activity has been concluded where a number of tools were evaluated. The
assessment concluded that Performance Testing has to be carried out in Visual Studio .net itself. However,
there are certain other tools which can be utilized to foster in Performance Testing.
Below table enlists tools which will be used for Performance Testing as well as for resources monitoring:
Copyrights © NetSol Technologies Ltd. All rights reserved 1

Tool Purpose
Visual Studio 2013 For Performance Testing of areas which can’t be worked out
with HP PC / LR.
HP LoadRunner 12 Automated performance and stress testing
NeoLoad Trace client/server communication
SolarWind Defects logging and tracking
Spotlight Enterprise Resource Monitoring
Windows Profiler Extensive Resource Monitoring
Table 1. Tools fostering Performance Testing phases
2. Performance Testing Tools Evaluated
The tools summarized in this section constitute a concise list of tools available over internet for
automated performing testing (over a hundred tools were looked into). Each tool was evaluated
based on the criteria defined in Section 3.
2.1 HP LoadRunner and Performance Center
HP LoadRunner is a cost-effective and comprehensive tool used for automated performance
testing. This is our first choice because:
• LR is industry leader with over 90% market share
• ChromeIS has most experience in HP LR/PC
• Fully trained resources available
• Uses standard programming techniques and common languages such as C, VB and
Java; thus eliminating need to learn new language.
• Easy to script, parameterize, correlate and close to real test execution
• Extensive reporting, graphs, statistics collection, helping in root cause identification
HP LoadRunner brought support for .net protocol in version 12 which was released last month (15 th
April 2014); however, the support (as in readme file) does not describe component level details
which can be recognized by LoadRunner.
Through experiments, we can conclude that LR does not recognize OUR PRODUCT CMS client. It
fails with the most basic level of scripting (login only – without selecting a branch) and the
LoadRunner component i.e. VUGen stops responding. The script generated fails to compile and
introduces number of errors.
We can keep an eye on new patches of LR and keep experiments alive. However, this must not
stop overall performance test progress.
As of now, the tool is not a match to OUR PRODUCT requirements and cannot facilitate at any
level or phase.

2.2 IBM Rational Performance Tester
IBM RPT was 2nd
choice since they hold nearly 6% out of 10% market share (after 90% being with LR).
IBM RPT is slow in scripting, intelligence, execution and other aspects. However, nowhere on their
website IBM claims that RPT supports .net or WCF or WPF.
A formal conversation was opened with IBM but there had been no response.
We can keep an eye on IBM newsletter to see if they provide any sophisticated mechanism for
performance testing of WCF or SOA based architecture.
As of now, the tool is not a match to OUR PRODUCT requirements and cannot facilitate at any
level or phase.
2.3 Borland Silk Performer
Silk Performer 15 now enables to capture all the raw measure data of a load test. It now
automatically detects asynchronous communication patterns such as polling and long-polling. It
more effectively supports applications with periodic, event-based, or server-triggered content
updates.
Tool is not as rich and introduced support for Ajax just recently with no support for modern needs of
RIA based clients and smart clients.
The tool is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.4 Telerik Test Studio
The tool fails to recognize Telerik “ribbon” components. This was discussed with Telerik support and
they confirmed the problem exists and listed as known issues.
Since the tool fails to recognize the first component which loads after login, the tool cannot be
used.
2.5 Apache JMeter
Through experiments, we can conclude the JMeter is able to script the calls, however, since OUR
PRODUCT uses encryption, JMeter fails to decrypt the client/server communication. This is
necessary since the JMeter cannot process an encrypted call.
Ideally, we require JMeter to decrypt the server response, process it as required, re-crypt the newly
formulated request and then send it to the server. JMeter needs to do it for each response from
server to simulate load.
Another workaround could be to remove encryption from OUR PRODUCT as long as application is
undergoing performance testing. Although this imposes possible concern like:
• The encryption and decryption is costly activity and requires additional layer of effort
• The performance measures may not be realistic (slightly improved than real) since
client and server don’t have to encrypt or decrypt

If the above constraint can be lived with, we can consider JMeter further.
2.6 j-Chav
JChav is a way to see the change in performance of a web application over time, by running a
benchmark test for each build. JChav reads all the JMeter logs from each run (one per build), and
produces a set of charts for each test in each run.
The tool compliments JMeter and hence cannot be right fit for OUR PRODUCT.
2.7 Microsoft WCAT – Web Capacity Analysis Tool
(formerly WAS – Web Application Stress Tool)
WCAT (Web Capacity Analysis Tool) is an excellent tiny tool from Microsoft to perform load test
web application on IIS. This tool enables users to do performance analysis on various scenarios of
web application. All “perfmon” performance counters (like processor time, private bytes usage,
disk queue length, total bytes sent or received in network) can be specified for the performance
testing.
Once scenario in application (i.e. appropriate URLs) is specified, a number of virtual clients from
various client machines will visit the URL with appropriate request data. Like other performance
tools, the following players are involved in a performance testing:
• Server – machines on which your web application components are running
• Client – A virtual client on a machine which acts as end-user to visit the URL
• Controller – This coordinates a test among the virtual clients on various machines. It also
captures and collates the performance counters from appropriate servers.
• Works in Visual FoxPro.
However, with all the excellence this tool may have, it is unable to recognize OUR PRODUCT
application components.
2.8 Sun VisualVM
VisualVM is a free visual tool, originally from Sun, to monitor and troubleshoot Java applications.
Runs on Sun JDK 6, but is able to monitor applications running on JDK 1.4 and higher. Utilizes various
available technologies like jvmstat, JMX, the Serviceability Agent (SA), and the Attach API to get
data and uses minimal overhead on monitored applications. Capabilities include: automatically
detects and lists locally and remotely running Java applications; monitor application performance
and memory consumption; profile application performance or analyze memory allocation; is able
to save application configuration and runtime environment together with all taken thread dumps,
heap dumps and profiler snaphots into a single application snapshot which can be later processed
offline.
Tool is more focused on java based applications. The tool does not address modern needs of RIA
based clients and smart clients.

2.9 WAPT
WAPT can test and analyze the performance characteristics and bottlenecks of web under various
load conditions with dynamic content and secure HTTPS pages. Real User Simulation with Dynamic
Test Data, Recording and Playback of HTTPS/SSL Pages, Flexible Test Volume & Load Definition,
Clear Reports and Graphs, Full Log of Virtual Users Activities Adjustable delays ("think times")
between requests to simulate real user behavior Support of redirects, including redirects to another
server Support of all types of proxies.
The tool is very mature but it lacks support of WPF, WCF and smart clients.
2.10 Web Application Load Simulator (LoadSim)
LoadSim is a web application load simulator. It allows you to create simulations and have those
simulations run against your webserver.
It’s a web application load simulator. It allows you to create simulations and have those simulations
run against your webserver.
2.11 Neo Load
NeoLoad is a professional load testing software for web applications (ASP.Net, J2EE, PHP, etc.)
Works almost without scripting, automatic handling of ASP.Net VIEWSTATE, Automatic support of
user sessions based on cookies or URL rewriting, Realistic VU simulation, cookies handling, plays
simultaneously Virtual Users with different behaviors, modem emulation, IP spoofing, SSL support
(record and playback), HTML/PDF/XML reporting, Compare results of several tests, Controller and
Load Injectors are multi-platform: Windows, Unix.
The tool provides no mechanism to test desktop based applications – this in turns disregards smart
client applications as well. However, it successfully transcribes the sequence and composition of
calls made to web services. This can be used in case the composition is not known to performance
tester.
2.12 EggPlant Load Tester
eggPlant performance enabled HTTP-oriented Java and .NET Web Virtual Users (VUs) to be easily
generated by recording use-cases.
eggPlant Performance is designed for asynchronous requests, push technology, HTTP streaming,
and dynamic data originated by the client by supporting multiple threads of execution and
intelligent script generation.
Some applications, however, simply cannot be automated at the HTTP level and are best
automated at the GUI application level. In these situations eggPlant Performance can be used
with eggPlant Functional or Selenium VUs for application level testing.
eggPlant has in-built support for driving common non-web client-side APIs such as .NET and Java.
This leverages the native support for security, session management, and data serialization making
test creation faster, more effective, and more robust. The tool supports mobile clients, complex
thick clients and propriety devices. More information is available.

The tool needs further RND to reach a conclusion.
2.13 OpenSTA – Open System Testing Architecture
OpenSTA is a distributed software testing architecture based on CORBA. Using OpenSTA (Open
System Testing Architecture) a user can generate realistic heavy loads simulating the activity of
multiple concurrent virtual users. OpenSTA graphs both virtual user response times and resource
utilization information from all Web Servers, Application Servers, Database Servers and Operating
Platforms within environment, so that precise performance measurements can be gathered during
load tests and analysis on these measurements can be performed.
client applications as well.
2.14 OpenWebLoad
OpenWebLoad is a tool for load testing web applications. It aims to be easy to use and providing
near real-time performance measurements of the application under test.
2.15 OpenLoad
OpenLoad provides no mechanism to test desktop based applications – this in turns disregards
smart client applications as well.
2.16 loadUI
loadUI tool focused mainly on Light Directory Access Protocol (LDAP) and web applications
working on HTTP, Web Services, REST, AMF, JMS, JDBC as well. The tool appears to be limited in
functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA) such as flex
or Silverlight.
2.17 LoadeaTest
LoadeaTest works in C# and .net and has various components. Capture module creates new test,
modelize it as a script, enhances, replays and validates it. Control module defines test properties to
launch test on several machines, monitors test in real time and gives a results view. An injection
module executes script and collect data on different machines. Collect module is an http
interface between all modules and database for storage dump. Last, analysis module is to view
test results as tables and graphs, to generate reports, and to activate statistic algorithm on data
(Principle Component Analysis).
It works with HTTP, TCP, SOAP, RSS System resources. Scripts in high level language (C#, VB.NET )
and data in XML format.

The tool provides no mechanism for WCF specific applications in .net.
2.18 LoadManager
LoadManager was developed in Java (Eclipse platform). The result is a testing tool which runs on
all platforms supported by Eclipse and Java such as Linux, Windows, HP Unix, etc. Supported
protocols are: HTTP, HTTPS, Socket (TCP/IP), JMS, RMI, scripts in high level language (C#, VB.NET )
and data in XML format. Color-coding shows which users pass/fail and a script-link is provided*
Testing of individual or groups of users can be paused/restarted at will.
The tool provides no mechanism for WCF specific applications in .net.
2.19 Test Automation FX
TBC
2.20 CodedUI
TBC
2.21 Soap UI
TBC
2.22 Soap-Stone
Soap Stone is a Network benchmark application which can put your network under load and
conduct automatic benchmark and recording activities
The tool is limited to network load. This is very basic to our requirement.

2.23 Web Performance Suite
Web Performance Suite includes a client-side analysis module (Analyzer) a load testing module
(Load Tester) and server analysis modules. It is capable of handling most popular web-design
technologies such as AJAX, Web forms, J2EE, ASP, .NET/C#, PHP, ColdFusion, Java, etc.
2.24 Webserver Stress Tool
Webserver Stress Tool is a HTTP-client/server test application
2.25 WebLOAD
WebLOAD is an Open Source, available for free download and use, WebLOAD is a commercial-
grade open source project with more than 250 engineering years of product development.
Companies that require commercial support, additional productivity features and compatibility
with third-party protocols have the option of purchasing WebLOAD Professional.
2.26 QALoad
QALoad is a performance testing tool from Micro Focus used to perform repeatable load testing
and determine the ultimate performance and potential limits of a system
QALoad has its own language, EasyScript. It has no support for Silverlight or Smart Client.
2.27 Lisa Load
LISA offers direct functional and performance validation of every component of distributed
applications, with or without a UI present. LISA load tests also leverage the end-to-end functional
tests created in LISA, invoking and verifying at every tier of the architecture, with a very low system
footprint required, so capacity can be simulated at the lowest possible effort and cost.
Lisa Load supports only Java/Swing; hence making it as not a choice for OUR PRODUCT.

2.28 Grinder
Grinder tool focused mainly on Light Directory Access Protocol (LDAP) and web applications
working on HTTP protocol. The tool appears to be limited in functionalities it offers. The tool is
incapable of handling Rich Internet Application (RIA) such as flex or Silverlight.
2.29 nGrinder
nGrinder is a stress testing tool, executing script creation, test execution, monitoring, and result
report generator simultaneously.
Since Grinder itself is not applicable to OUR PRODUCT, the GrinderStone is not applicable either.
2.30 GrinderStone
GrinderStone is an Eclipse plug-in for Grinder load testing scripts development including
debugging, modularity and pretty logging.
Since Grinder itself is not applicable to OUR PRODUCT, the GrinderStone is not applicable either.
2.31 MStone
Mstone started as a mail performance measurement system but now can simultaneously test SMTP,
POP, IMAP, and some HTML based systems. It measures transaction latency in multiple stages, and
graphs the combined results from multiple clients.
MStone provider no mechanism for RIA or Smart clients.
2.32 ValGrind
Valgrind is focused on Linux programs. Built in tools automatically detect many memory
management and threading bugs. ValGrind is not robust enough for RIA and smart clients.
2.33 QEngine
AdventNet QEngine WebTest provides accurate way to test the functionality of Web applications
in both Windows and Linux machines. It provides an easy-to-learn user interface to completely
automate the creation of sophisticated test scripts, allows record and playback Web browser
events in IE and Mozilla browsers.
The tool is immature and focused more on Regression automation with no support for Silverlight.

It is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.34 Qtest
Qtest is automated load testing and monitoring solution, offering high capacity load simulation,
extensive monitoring capability. Qtest supports all Web, Web Service, J2EE, .Net, ASP, AJAX, CGI,
and Mainframe Portal environments and, in particular, complex Web applications like Siebel, SAP,
and Epiphany. With its non-protocol based Winload module, Qtest also supports any non-web
application.
The tool offers sophisticated and complex frameworks, but nowhere does it provide Silverlight or
Smart Client based on WCF.
It is not a match to OUR PRODUCT requirements and cannot facilitate at any level or phase.
2.35 HammerHead 2
Hammerhead 2 is a stress testing tool designed to test out your web server by initiating multiple
connections from IP aliases and simulated numerous (256+) users at any given time. The rate at
which Hammerhead 2 attempts to pound web server site is fully configurable.
2.36 Hammerora
Hammerora is a load generation tool for the Oracle Database and Web Applications. Hammerora
includes pre-built schema creation and load tests based on the industry standard TPC-C and TPC-H
benchmarks to deploy against the Oracle database with multiple users. Hammerora also converts
and replays Oracle trace files and enables Web-tier testing to build bespoke load tests for your
entire Oracle application environment.
The tool provides no mechanism to test web services or RIA based or desktop based smart client
applications as well.
2.37 Pylot
Pylot is open performance and scalability testing tool, capable of running HTTP load tests to
compliment capacity planning, benchmarking, analysis, and system tuning. Pylot generates
concurrent load (HTTP Requests), verifies server responses, and produces reports with metrics. Tests
suites are executed and monitored from a GUI.
2.38 OpenWebLoad
preVue-X provides non-intrusive regression and performance testing of X-Window applications.
preVue-X accurately emulates multiple users while requiring no special hooks into your application

or X Libraries, thus guaranteeing that the application tested is the same application that will be
deployed. preVue-X operates at the X protocol level and independently of the GUI. It
automatically collects all user interactions with the application (e.g., mouse movements, button
activity, keystrokes, think time delays) into an easily editable test script for playback. Features
include automated synchronization and time-stamping; geometry independence and script
viewer
The tool does not speak about WCF in specific or smart clients. Need further RND to reach a
conclusion.
2.39 SD’s Profiler
SD's Profiler collects information about execution, frequency of all blocks of code in an application.
A graphical display sorts this information by file or by frequency, and the relative execution costs
are shown graphically superimposed over the source code. Available for C, COBOL, and Java. For
C and Java, easily customized to work in embedded or real-time contexts.
2.40 Benerator
benerator is a framework for creating realistic and valid high-volume test data, used for
(unit/integration/load) testing. Metadata constraints are imported from systems and/or
configuration files. Data can be imported from and exported to files and systems, anonymized or
generated from scratch. Domain packages provide reusable generators for creating domain-
specific data as names and addresses internationalizable in language and region. It is
customizable with plugins and configuration options.
2.41 SOATest (by Parasoft)
Parasoft SOA Quality Solution continuously measure how each service conforms to the often
dynamic expectations defined by both your own organization and your partners, reduces the
complexity of testing in today’s heterogeneous environment – with limited visibility/control of
distributed components or vendor-specific technologies. It validates all critical aspects of complex
transactions, which may extend beyond the message layer through a web interface, ESBs, etc.
The tool provides no clarity in order to test WCF or smart client or web services.
2.42 Scapa Test and Performance Platform
Scapa Test and Performance Platform is testing, diagnosis and monitoring product. The Platform
works by simulating multiple users of a computer system. By measuring the performance seen by
virtual users, it is possible to understand how the system would behave under a corresponding level
of real usage.

2.43 CLIF Load Injection
CLIF is a modular and flexible distributed load testing platform. It may address target systems
reachable from a Java program (HTTP, DNS, TCP/IP...) CLIF provides 3 user interfaces (Swing or
Eclipse GUI, command line) to deploy, control and monitor a set of distributed load injectors and
resource consumption probes (CPU, memory...) An Eclipse wizard helps programming support for
new protocols. Load scenarios are defined through XML-editing, using a GUI, or using a capture
tool. The scenario execution engine allows the execution of up to millions of virtual users per load
injector.
2.44 ContiPerf
ContiPerf is a small testing utility that enables the user to easily leverage JUnit 4 test cases as
performance tests e.g. for continuous performance testing. It is inspired by JUnit 4's easy test
configuration with annotations and by JUnitPerf's idea of wrapping Unit tests for performance
testing, but more powerful and easier to use.
2.45 Coveriety
TBC
2.46 D-ITG (Distributed Internet Traffic Generator)
D-ITG (Distributed Internet Traffic Generator) is a platform capable to produce traffic at packet
level accurately replicating appropriate stochastic processes for both IDT (Inter Departure Time)
and PS (Packet Size) random variables.

2.47 Database Opensource Test Suit (DOTS)
The Database Opensource Test Suite (DOTS) is a set of test cases designed for the purpose of stress-
testing database server systems in order to measure database server performance and reliability.
2.48 DB Monster
DBMonster is an application to generate random data for testing SQL database driven
applications under heavy load.
2.49 Deluge
Deluge is an open-source web site stress test tool. It simulates multiple user types and counts,
includes proxy server for recording playback scripts, and log evaluator for generating result
statistics.
This tool has not been updated since 2002. This tool is no longer under active development
although it is still available on Sourceforge.
2.50 DieselTest
Dieseltest is a Windows application that simulates hundreds or thousands of users hitting a website.
BEWARE: This tool has not been updated since 2001. It remains listed here in case anybody wishes
to take it over.
2.51 Faban
Faban is a facility for developing and running benchmarks, developed by Sun. It has two major
components, the Faban harness and the Faban driver framework. The Faban harness is a harness
to automate running of server benchmarks as well as a container to host benchmarks allowing
new benchmarks to be deployed in a rapid manner. Faban provides a web interface to launch &
queue runs, and extensive functionality to view, compare and extract graph of outputs.

2.52 FunkLoad
FunkLoad is a functional and web performance testing tool, written in Python. It can be used for
functional and regression testing of web projects as well as performance testing and stress testing
to overwhelm the web application resources and test the application recoverability.
2.53 Curl-Loader
Written in C, Curl-Loader is a web application testing and load generating tool. Provides an open-
source alternative to Spirent Avalanche and IXIA IxLoad. The loader uses real HTTP, FTP and TLS/SSL
protocol stacks, simulating tens of thousand and hundred users/clients each with own IP-address.
The tool supports user authentication, login and a range of statistics.
2.54 Spirant Avalanche
TBC.
2.55 IXIA IxLoad
TBC.
2.56 Paessler
TBC
2.57 AQ TestComplete and AQ Time
AQTime is a performance profiler and memory allocation debugger for 32-bit and 64-bit Windows
and .NET-connected applications. AQtime can be integrated into Microsoft Visual Studio,
Borland Developer Studio or CodeGear RAD Studio or CodeGear RAD Studio or it can run as a
stand-alone application. AQtime can also determine what caused bottlenecks.
The tool provides no mechanism to test RIA based applications or WCF calls.

2.58 Test Maker
TestMaker offers building and running intelligent test agents that test Web-enabled applications for
scalability, functionality, and performance. It works with object-oriented scripting language
(Jython) to build intelligent test agents, an extensible library of protocol handlers (HTTP, HTTPS,
SOAP, XML-RPC, SMTP, POP3, IMAP), a new agent wizard featuring an Agent Recorder to write
scripts, a library of fully-functional sample test agents, and shell scripts to run test agents from the
command line and from unit test utilities.
The tool provides no mechanism to test RIA and smart client.
2.59 TP Test
TPTEST measures the throughput speed to and from various reference servers on the Internet. The
use of TPTEST may help increase the consumer/end user knowledge of how Internet services work.
The tool is very basic and cannot be graded as performance testing tool for applications.
2.60 Tsung
Tsung is a distributed load testing tool. It is protocol-independent and can currently be used to
stress HTTP, SOAP and Jabber servers (SSL is supported). It simulates complex user's behavior using
an XML description file, reports many measurements in real time (including response times, CPU
and memory usage from servers, customized transactions, etc.). HTML reports (with graphics) can
be generated during the load. For HTTP, it supports 1.0 and 1.1, has a proxy mode to record
sessions, supports GET and POST methods, Cookies, and Basic WWW-authentication. It has already
been used to simulate thousands of virtual users.
The tool requires further RND to experiment with OUR PRODUCT POS.
2.61 Tnsgen
Tnsgen is test data generation tool which operates under Solaris, Linux and Windows, capable of
generating thousands of records per second in XML, SQL or user-defined formats. Easily extensible
to include project-specific data sets.
The tool is a data generating tool, it cannot be graded as performance testing tool.
2.62 N-Time
N-Time tool is very similar to NUnit tool to perform repeatable tasks that help managers, architects,
developers and testers to test an application against its performance.

2.63 p-unit
p-unit is an open source framework for unit test and performance benchmark, under GPL license.
p-unit supports to run the same tests with single thread or multi-threads, tracks memory and time
consumption, and generates the result in the form of plain text, image or pdf file.
2.64 Lobo
Lobo tool supports performance testing of Java based applications only.
2.65 MessAdmin
MessAdmin is a light-weight and non-intrusive notification system and HttpSession administration for
J2EE Web Applications, giving detailed statistics and informations on the application. It installs as a
plug-in to any Java EE WebApp, and requires zero-code modification.
The tool is specific for java based application. It provides no mechanism to test desktop based
applications – this in turns disregards smart client applications as well.
2.66 Multi Router Traffic Grapher (MRTR)
Multi Router Traffic Grapher is written in perl and works on Unix/Linux as well as Windows and even
Netware systems. MRTG is free software licensed under the Gnu GPL.
The tool is specific for java based application. It provides no mechanism to test desktop based
2.67 RRRD Tool
RRRD Tool is data logging and graphing system for time series data. It can be used to write custom
monitoring shell scripts or create whole applications using its Perl, Python, Ruby, TCL or PHP
bindings.
The tool is barely into application performance testing.
2.68 Multi-Mechanize
Multi-Mechanize is an open source framework for performance testing. It allows running
simultaneous python scripts to generate load against a web site or web service.

The tool does not address modern needs of RIA based clients and smart clients.
2.69 Stress Driver
Stress Driver is a general-purpose stress test tool.
2.70 Load Impact
LoadImpact is online load testing service from loadimpact.com for load- and stress- testing; with
access to distributed network of load generator nodes – server clusters with very fast connections
to enable simulation of simultaneous users accessing your website concurrently. Free low level load
tests for 1-50 simulated users; higher levels have monthly fees.
No claim to test RIA client on their site. No support for smart client.
2.71 Pure Load
PureLoad is a load testing tool that simulates concurrent users executing requests against server
based applications. Use PureLoad to verify performance and stability of your server applications.
2.72 Pure Agent
PureAgent is a performance monitoring tool that replicates real users generating activities on
server applications. PureAgent measures the total round trip time it takes to execute one or many
user operations accessing either static or dynamic content. The results are presented as charts in a
standard web browser, alarm levels can be defined which might trigger notifications to be sent by
Email. The monitored scenario of user operations is created using the PureTest application.
2.73 FWPTT
fwptt is used for load testing of web applications only. It can record normal and ajax requests. It
can understand HTML being returned from server, hence, it works with asp.net, jsp, php or other
such applications. The tool recently supported AJAX as well.
Since AJAX is the extent of the tool, the tool is too basic to meet the complex requirements of OUR
PRODUCT where web is using silverlight and smart client.

2.74 JBlits Pro
JBlits gives continuous heavy loading and automated response validation. Easy to use, quick to
setup. Fully featured with logging, graphs, stats, scheduling, HTTP debugger etc. Full Java API.
Works in Java, works for Java. The tool is not meant for Silverlight and smart client.
2.75 JCrawler
JCrawler can start with a set of starting URLs and it will begin crawling from that point onwards,
going through any URLs it can find on its way and generating load on the web application.
Although, the load parameters (hits/sec) are configurable, JCrawler doesn’t look like a real
performance testing tool. There are no built-in mechanism for correlation and parameterization.
The tool, at best, can simulate load on static websites where no custom data is required.
2.76 PandoraFMS
Pandora FMS is a monitoring Open Source software and watches systems and applications.
Pandora FMS could detect a network interface down, defacement in website, a memory leak in
server application, or the movement of any value of the NASDAQ new technology market.
Pandora FMS could send out SMS message when system fails... or when Google's value drop below
US$ 500.
Although PandoraFMS provides features unique to itself and it can be very useful in many ways,
PandoraFMS fails to simulate load on RIA and Smart Client.
2.77 Siege
Siege can be used to test over HTTP and HTTPS. SIEGE is an http regression testing and
benchmarking utility, designed to measure the performance of code. It lets the user hit a
webserver with a configurable number of concurrent simulated users. SCOUT surveys a webserver
and prepares the urls.txt file for a siege. In order to perform regression testing, siege loads URLs from
a file and runs through them sequentially or randomly. Scout makes the process of populating that
file easier. You should send out the scout, before you lay siege.
With the above limitation of the tool, Siege is not a match to OUR PRODUCT requirements and
cannot facilitate at any level or phase.
2.78 HTTP Load
http_load is a multiprocessing http test client. As the name suggests, the tool works specifically for
HTTP protocol. This helps cater plain HTML applications only.
http_load runs multiple http fetches in parallel, to test the throughput of a web server in a single
process. It can be configured to do https fetches as well. It processes a file containing a list of URLs
that may be fetched, a flag specifying how to start connections (either by rate or by number of
simulated users), and a flag specifying when to quit (either after a given number of fetches or a

given elapsed time). There are also optional flags for checksums, throttling, random jitter, and
progress reports.
With all above, the tool serves very basic level requirement for OUR PRODUCT. Provides no
mechanism to test RIA based and desktop based applications – this in turns disregards smart client
2.79 httPerf
httperf provides facility for generating HTTP workloads. The three distinguishing characteristics of
httperf are:
• its robustness, which includes the ability to generate and sustain server overload,
• support for the HTTP/1.1 and SSL protocols, and
• its extensibility.
2.80 iPerf
iperf was developed as a modern alternative for measuring maximum TCP and UDP bandwidth
performance. iperf allows the tuning of various parameters and UDP characteristics. iperf reports
bandwidth, delay jitter, datagram loss.
2.81 Site Blaster
Site Blaster is a web site load and stress testing tool. It can be used to rapidly submit requests to a
site. Or, it can pause a random amount of time between submissions; approximating the behavior
of a user. While the testing is being performed, the pages being tested will be displayed. When
testing is complete, a report is available that can be viewed or printed.
It provides no mechanism to test RIA based and desktop based applications – this in turns
disregards smart client applications as well.
2.82 RawLoadTester
RLT calls the selected URL multiple times and tells server response time. It writes some additional
runtime details to the PHP log file to optionally do more granular analysis afterwards. Although the
server processes most of the statistics, all URL requests come from the browser. You can run as
many browsers and workstations simultaneously as you want.

RLT provides no mechanism to test RIA based and desktop based applications – this in turns
2.83 Allmon
Allmon is an open source, generic system collecting and storing metrics used for performance and
availability monitoring.
2.84 AppsWatch
Appswatch is an automated testing tool for performance, and GUI testing and SLA monitoring and
alerting. It is protocol independent and works with any application accessible from windows
desktop. It works for Citrix, Java, .NET, Oracle, SAP, Eclipsys, Mckessen, Meditech, GE Healthcare,
Epic, Cerner, Client-Server, Flex, Flash and even custom applications
2.85 BurnIn Test
BurnInTest allows sub-systems of a computer to be simultaneously stress tested for endurance,
reliability and stability. BurnInTest tests the CPU, hard drives, CD ROMs, DVDs, sound cards, 2D
graphics, 3D graphics, RAM, network connections & printers. The Pro version can also test tape
drives, USB, serial and parallel ports. This can all be done simultaneously as the application is
multithreaded.
2.86 Capacity Calibration (CapCal)
CapCal writes code in Java for measuring web site performance under various loads--by actually
generating those loads, in real time, from computers scattered all across the Internet. CapCal can
also be distributed across your LAN and WAN, giving realistic performance data that matches end
user's experience.

2.87 Chariot
Chariot tests and troubleshoots any segment of network and provides comprehensive reports of
the results by emulating transaction traffic from real applications. It can be used to predict how
well network infrastructure will handle popular applications prior to deployment, and to isolate and
evaluate the performance of new network devices. Chariot is highly scalable and can be used to
set up and control tests—emulating up to 10,000 connections and hundreds of thousands of end-
users—all from a central console.
It cannot be graded as real performance testing tool since it aims for network and infrastructure
against internet applications.
2.88 CitraTestVU
CitraTestVU is a load testing tools, works in Visual Basic scripting to simulate Load on Citrix &
Microsoft Terminal Services Hosted Environments.
The tool is focused on Java only. It provides no mechanism to test RIA based and desktop based
2.89 SSW Performance Pro
SSW Performance PRO! isolates performance bottle-necks in your Access 97, 2000, 2002 or 2003
app. It checks every form, combo and list box, and reviews SQL statements. It reports exactly what
to fix to improve the speed of your application. It’s especially useful after you've upsized an
application to SQL Server to pinpoint black spots in your app.
The tool is very basic and mainly targets MS Access. It cannot be graded a commercial level tool.
2.90 e-Load
e-Load is enables accurate testing of the response times and scalability of web applications and
web services. e-Load can concurrent users, executing real business transactions to analyze how
well web software applications will perform under load.
The tool provides no mechanism to test RIA based and desktop based applications. It supports
Web Services but not WCF.
2.91 MLoad
MLoad generates high levels of realistic network and user traffic ensuring that WAP Gateways,
MMSCs and web based mobile infrastructure elements meet the high-bandwidth, high-
performance demands of a mobile internet service. The toolset includes a test harness to allow
acceptance testing of mobile elements against performance requirements in the deployed
environment.
The tool focused on data flux and network bandwidth only. It provides no mechanism to test RIA
based and desktop based applications. It supports Web Services but not WCF.

2.92 Monitor Master
Monitor Master is a quality management solution for wireless data services - WML/XHTML, SMS and
MMS. Monitor Master helps Operators and Content Developers ensure that end-users of wireless
applications will have the best quality of experience possible - regardless of the handset being
used.
The tool focuses wireless protocols and not RIA based and desktop based applications. It supports
Web Services but not WCF.
2.93 Forecast
Forecast is a easily and accurately tests the performance and scalability of IT applications and
systems.
No support for WPF, WCF or Smart Client.
2.94 GS DataGenerator
GS DataGenerator generates high volumes of 'intelligent' test data on virtually any relational
database, creates business trends and patterns, simulates business workflow, mimics potential data
errors Employ number of predefined functions. This broad range of functionality allows testing
database performance and functionality during development.
It’s very basic tool, works in DOS and provides no support for WPF, WCF or Smart Client.
2.95 vPerformer
vPerformer is used to assess the performance and scalability of web applications. vPerformer
allows assesses how web application responds when it is concurrently accessed by a large number
of users. It can measure the performance characteristics of application by recording and replaying
automated scripts that simulate a large number of concurrent virtual users.
The tool is very basic and doesn’t address modern needs like Silverlight etc.
2.96 JUnit
JUnitPerf is a collection of JUnit test decorators used to measure the performance and scalability of
functionality contained within existing JUnit tests
The tool is specific for Java related apps.

2.97 Xceptance LoadTest
Xceptance LoadTest for web and Java and other apps includes recording capabilities. XLT Cloud
Service available. Tests implemented as JUnit 4 test cases. For web-based tests, the framework
provides a (headless) browser that can emulate Internet Explorer or Firefox behavior. Can execute
client-side JavaScript in the emulated web browsers and that way it simplifies the creation of test
cases for Web 2.0 applications. It is platform independent since the tool is implemented in Java;
test scripting in Java or Ruby. The tool is free for up to five virtual users.
The tool is focused on Java only. It provides no mechanism to test RIA based and desktop based
2.98 zapwireless
zapwireless is a wireless throughput test tool which can be used to fully characterize the statistical
performance of a wireless link.
2.99 Seagull
Seagull was primary aimed at IMS protocols, now it is a traffic generator for functional, load,
endurance, stress and performance tests for almost any kind of protocol. Currently supports
Diameter, XCAP over HTTP, TCAP (GSM Camel, MAP, Win) protocols.
2.100 SIPp
SIPp is a performance testing tool for the SIP protocol. Its main features are basic SIPStone
scenarios, TCP/UDP transport, customizable (xml based) scenarios, dynamic adjustement of call-
rate and a comprehensive set of real-time statistics. It can also generate media (RTP) traffic for
audio and video calls.
Although Sipp is a rich tool and may be useful for specific purposes, it provides no mechanism to
test RIA based and desktop based applications – this in turns disregards smart client applications as
well.
2.101 SLAMD
SLAMD Distributed Load Generation Engine is a Java-based application designed for stress testing
and performance analysis of network-based applications.
SLAMD does not provide any mechanism to test RIA based and desktop based applications – this
in turns disregards smart client applications as well.

2.102 IxoraRMS
IxoraRMS is a monitoring tool with great visualization and customization capabilities. But it severely
lacks capabilities to support RIA and modern complex architecture.
2.103 j-hawk
j-hawk is a Java based open source framework which can be incorporated within an application
for performance testing. The idea requires to define *module and its method inside the application
and register the same with j-Hawk. j-Hawk executes the modules and generates a graphical
performance report which can be analyzed to find performance bottleneck of your application.
The tool is specific for Java based application which is not the case with OUR PRODUCT.
2.104 Wireshark
TBC
2.105 Ostinato
Ostinato is an open-source, cross-platform packet/traffic generator and analyzer with a friendly
GUI. It aims to be "Wireshark in Reverse" and thus become complementary to Wireshark.The tool is
specific for Java based application which is not the case with OUR PRODUCT.
2.106 Eclipse Test & Performance Tools Platform Project
(EPTP)
EPTP supplies frameworks and services that allow software developers to build unique test and
performance tools, both open source and commercial, that can be easily integrated with the
platform and with other tools.
Eclipse Test & Performance Tools Platform Project (TPTP) evidently provides no mechanism to test
desktop based applications – this in turns disregards smart client applications as well.
2.107 Web Polygraph
Web Polygraph is a benchmarking tool for caching proxies, origin server accelerators, L4/7
switches, content filters, and other Web intermediaries. This tool focused mainly on Light Directory
Access Protocol (LDAP) and web applications working on HTTP protocol. The tool appears to be
limited in functionalities it offers. The tool is incapable of handling Rich Internet Application (RIA)
such as flex or Silverlight.

3. Tools Selection Criteria
Criteria for selection of Automated Performance Testing tool was defined before proceeding with
research activities. The criteria broadly address Business and Technical Requirements and
Objectives sought from Performance Testing. Below is a highlight of foundation of criteria:
3.1 Business and Technical Requirements
1. It is essential for the application is able to pass at least 1 test case to perform certain business
process. Although, it is best to have reasonably stable application before performance testing
starts. If there is urgency, the application should be able to work under at least 1
situation/record-set.
2. It is pivotal to finalize performance test objectives before test scripting is begun. Number of
users, data volume and other variables must be known beforehand otherwise unnecessary
delays may occur. For example, if the intention is to run endurance or longevity test with 1000
users over 4.5M record base, for 24 hours; then test data requirement will be massive as
compared to running 100 users for 1 hour.
3. Dedicated test environment and interface systems are identified and requisition (booking)
made in advance and certified to be used for testing. The hardware specifications should be
documented for web server, application server, db server and any other layer involved. This
should be documented for Load Injectors/Generators as well.
4. System components have been identified and deployed over test environment mimicking (or
a close match to) production server.
5. Test IDs/passwords and access rights have been identified, requisition made and approval
granted for all involved applications. Test IDs must have the correct access rights for testing for
application as required by user’s role.
6. Criteria for test data to be used for testing has been drawn, and reviewed by respective
parties involved.
7. Test Data has been created in testing database.
8. System infrastructure and connectivity test are performed prior to each test phase to ensure
connectivity is enabled.
3.2 Performance Test Objectives
In general, the test objective is to ensure the respective applications meet the stated and implied
performance requirements of business; as well as, the statistics remain within the SLA agreed with
end customer.

1. Identify any bottlenecks and slow system performance
2. Ensure application response time is acceptable with data volume of 4.5 Million records
3. Gauge application responsiveness under Stress
4. Validate application behavior under endurance and longevity (24+ hours)
5. Validate the behavior of Load balancing over cluster nodes equally
6. To ensure that application is functioning properly under load
7. Establish performance benchmarks
8. Validate transactions response time against customers’ SLA
9. Perform root cause analysis for any problems and errors appearing under load – and
suggest possible optimization.
10. Draw projection as measures for future needs
11. Ensure smooth implementations with no/minimum disruption to business going live, and
applications rollout with minimum risk.
12. Project implementation compliance to ChromeIS processes.
13. Project quality that meets entry and exit criteria for each performance testing phase.
There are specific goals of performance testing, like:
1. Measure and benchmark response time for each page/screen to open
2. Measure and benchmark response time against each transaction, like, how long it
takes when, saving a proposal, saving a contract, converting a proposal into contract,
calculating exposure of a customer, asset is attached to a proposal, rentals are
generated, financial analysis is done, early termination is performed etc.
3. Measure and benchmark response time for each service as an individual.
4. This section specifies the test types (with definition for each) to run. This also explains
when application will be considered fit to undergo each type of testing and when a
particular
5. Measure and benchmark maximum number of users a transaction/service can
entertain disregarding response time (peak point identification)
6. Benchmark maximum number of users achieved against test environment
(hardware/software specifications)
7. Benchmark maximum hardware utilization against desired number of users (against
test environment specifications)
8. Benchmark system behavior against cold startup and warm startup
9. The spike ratio that the system must be capable of handling
10. The peak ratio that the system must be capable of handling?
11. Memory leakage against each business process or individual service

12. Identify which SPs fail and at what circumstances, at what line, under what conditions.
13. Facilitate in ensuring load is distributed evenly over all nodes of clusters (as configured)
14. Facilitate ensure seamless mitigation of failover under high load
15. Tolerance ratio: Imposed load ? 25 %?
16. Spike ratio: Imposed load x 3?
17. Burstiness ratio: Imposed load x 5?
18. Increase the load by multiplying the load baseline by 1x, 2x, 3x, 4x, Nx gradually until
unacceptable response time is reached through optimization.
19. Pass recommendation in achieving optimal configuration, or identifying target
component where optimization could be required.
0.1 Second
10 Second
20 Second
30 Second
Acceptable User Experience
Unacceptable User Experience
Questionable User Experience
Business is Closed
Response
Time
Load
60 Second
Figure 1, Performance Acceptance at a glance
The above assumes, all stakeholders are on same page regarding:
• What is transaction response time and what is acceptable response time?
• The difference between simultaneous and concurrent users
• Which metrics should be collected during all types of performance testing?
• What is the correlation between demand and increased load?

• How do we determine which components are problematic?
3.3 Performance Test Execution
Performance testing usually follows a linear path of events:
• Create performance-testing scripts and scenarios.
• Create Task Distribution Diagram to elaborate maximum number of users over a
component as well as maximum number of users accessing system at any point of
time.
• Execute performance tests as standalone tests to detect issues within a particular
business workflow. Repeat execution multiple times to surface most errors, identify root
cause of failure.
• Execute performance scenarios as a package (i.e. with all BPs together) to simulate
real life activities that are measured against performance success criteria.
• Monitor all variables involved in Performance Testing. This includes all servers and their
components such as processor, memory, NIC, clusters, caching, heap, top SQL
queries, SPs, ISS caching, IIS sessions, Kernel, Number of processes, Number of threads,
Throughput, DB Sessions, Parameters, Data Buffer Cache Hit Ratio, Shared Pool Hit
Ratio, Query to Parse ratio, PGA Usage, I/O read wait time, I/O write wait time, CPU
wait time, configurations, hits per second, timeouts etc. These statistics will be
collected for Web Server, App Server, DB Server and any other server involved, as well
as load injectors.
• Collate all statistics and observations. Perform analysis and document
recommendations/suggestions. Compile a concise performance testing results report.
• Tune the system.
• Repeat testing as required.
3.4 Load Description
In the context of Performance Testing, “Load” can be broadly classified into 2 categories, namely,
Server Based Load and User Based Load. Below is the elicitation of both terminologies.
Server-based Load
• Number of users and/or sessions
• Average session time
• Number of page views
• Average page views per session
• Peak period (e.g., 75% of traffic is from 11:00 AM-4:00 PM)
• Number of hits
• Average page size
• Most requested pages
• Average time spend on page

• Frequency of visits (e.g., 75% of users made one visit)
• Demographics
• Client information such as browser, browser version, Java script support, Java script
enable/disable, and so on.
User Based Load:
• Number of users
• Session length
• User activities and frequency of activities per session
• Think/Read/Data-input time
• Percentage by functional group
• Percentage by human speed
• Percentage by domain expertise/ familiarity (speed)
• Percentage by demographics (arrival rates)
Other questions to consider when describing system Load:
• What is the definition of “workload”?
• How do we size the workload?
• What is the expected workload?
• What’s the mix ratio of static pages vs. code?
• What is the definition of “increased load”?
• What is future growth? Can it be quantified?
• What is the definition of scalability?
A requirements document is filled up for each testing cycles to further clarify performance test
requirements. Refer to below example file.
4. Conclusion
After assessing large number of tools available in market, we can conclude that HP LoadRunner,
IBM Rational Performance Center and JMeter (the top 3 market leaders) do not directly support
testing the complex architecture we’re building; at least not as yet.
A number of open source tools, freeware utilities, ASM based performance testing services and
other freeware utilities do not address the performance testing challenges we‘re facing.
JMeter could be a possible option if encryption can be removed from OUR PRODUCT for testing
purposes.
EggPlant Load Tester described in detail how it can test web services written in .net. This requires
further RND and experimentation if time allows.
Tsung is an open source framework which claims testing web services and other complex niche
technology frameworks. This requires further RND and experimentation if time allows.
We’ve had trial runs with Visual Studio and optimistically, we can hope to meet our objectives with
MSVS. This is further described in Section 4.1 below.

4.1 Selected Tools
The testing tool chosen is Microsoft Visual Studio 2013.
NeoLoad can be used to identify the sequence of calls made to web services.
Although, MSVS this is not specifically designed performance testing in mind, however, this has
been picked as a workaround for current situation. Visual Studio offers scripting capabilities, thus
enabling to simulate asynchronous multithreaded simultaneous users to the application. This is the
first stepping stone for us.
Once the load is simulated closed to real behavior, various other tools will be used to identify
application behavior and other environmental factors undergoing changes (as described in
Section 3)
Below table enlists tools which will be used for Performance Testing as well as other activities (e.g.
resources monitoring) during performance testing phase:
Tool Purpose
Visual Studio 2013 For Performance Testing of areas which can’t be worked out
with HP PC / LR.
HP LoadRunner 12 Automated performance and stress testing
NeoLoad Trace client/server communication
SolarWind Defects logging and tracking
Spotlight Enterprise Resource Monitoring
Windows Profiler Extensive Resource Monitoring
Table 2. Selected tools for Performance Testing phases
4.2 SWOT Analysis
In concluding this research, we’ve learned about numerous testing tools, what those are capable
of and their approach towards load simulation.
HP LoadRunner is a market leader in performance testing tools, with over 90% market share
globally. ChromeIS has mature strong expertise in HP LoadRunner and HP Performance Center and
that had always been strength of ChromeIS to address any performance concerns in timely
fashion.
Unfortunately, HP LoadRunner has very limited support for .net protocol and it induces a lot of errors
whilst preparing performance test scripts for OUR PRODUCT. Experiments show that the tool fails in
deserialization, decoding and parsing already desterilized fragments of scripts. We can conclude
that LoadRunner 12 does not offer what it takes to test a sophisticated .net based application with
complex architecture and modern clients.
Time has changed and ChromeIS has taken initiatives to re-invent the products using state of the
art technologies making Rich Internet Applications and optimizing performance using Smart Client.
While this sounds exciting, it brings unforeseen challenges in many ways. Performance and
efficiency of software is undoubtedly among most critical aspects of software and this cannot be
ascertained without undergoing aggressive performance test. Hence, a workaround has been
suggested and Microsoft Visual Studio 2013 will be used for conducting performance testing.

4.3 Training Needs
ChromeIS has mature expertise in HP LoadRunner and HP Performance Center; however, these
tools are out of option.
Since ChromeIS will be using Visual Studio 2013 for development of Performance test agents,
certain level of resource training is required. A team size of 4-6 dedicated resources is expected to
be working for Performance Testing. This team will be briefly trained by QAM on performance
testing goals, objectives and overall approach towards meeting quantitative goals. While the
conceptual and quantitative goals remain the same, working with the tool itself will be crucial to
achieve test objectives.
Training programs to be provided to the team to enable successful planning and execution. A
training plan can be published separately or as an addendum to this document.
4.4 Lessons Learnt
Selection of tool and technology for carrying out performance testing is of utmost significance. The
same applies to overall software development as well.
We’ve seen in past as well that development technology was picked without consensus, or even
consultation with Performance Testing team. Now we’ve understood that performance testing an
application involves sophisticated activities, induces unforeseen technical challenges, and
imposes technological constraints, thus essentially making it a challenge to workout performance
testing. This cannot be simply compared with business testing where there is no or little involvement
of technological constrains.
It is suggested to keep performance testing team in due consultation when making technology
selection or making technical changes to existing application (upgrade of Silverlight version,
bringing parts of application logic to thick client). Although the final decision of picking the
development technology or modifying underlying components will be under sole discretion of the
PM, this discussion will ensure any possible constraints are communicated and any possible risks are
addressed
5. Definition of Terms, Acronyms And Abbreviations
Term Description
DEV Development Team
PT Performance Testing
SQA Software Quality Assurance

Performance Test WCF/WPF app - Selecting right Tool

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Performance Test WCF/WPF app - Selecting right Tool

Semelhante a Performance Test WCF/WPF app - Selecting right Tool (20)

Último

Último (20)

Performance Test WCF/WPF app - Selecting right Tool