“Testing Experience”, a Germany-based online magazine for software testers and test managers, has published an article by Ramesh Viswanathan, Senior Test Architect, Marlabs Bangalore in their March 2013 edition. Ramesh has presented his observations on the topic ‘Need for Performance Requirements to Ensure Reliable Business Applications’ through this article. It has incorporated topics associated with application development for various industry domains, techniques and tools for non-functional requirements gathering, optimal performance target, and best practices for developing resource-specific outputs. Testing Experience serves as a platform for knowledge transfer in software testing projects, with more than 250000 downloads per issue, in over fifty countries. Marlabs congratulates Ramesh Viswanathan for this accomplishment. We hope and wish that such valuable tech-philosophies lead him to better avenues of success in career! For viewing the detailed version of Ramesh’s article, refer the following link - http://www.testingexperience.com/testingexperience21_03_13.pdf

1. 34 Testing Experience – 21/2013
Introduction
Applications developed for various domain verticals like BFS, Insurance,
Healthcare, e-Learning, etc. have a huge volume of transactions in a day.
While there is sufficient focus and due-diligence in the industry to de-
fine the functional requirements, the absence of proper non-functional
requirements in many instances can lead to failure of applications and
adverse business impact. Hence, performance requirements gathering
play a very important role in the software development life cycle.
Requirements gathering for performance testing can be for
a. a new business application, or
b. for an application already existing in production.
There is always the need to systematically collate requirements with
respect to performance testing. These need to be captured at an early
stage of the software life cycle and signed off by all the key stakeholders.
At a high level, performance requirements can be categorized as below:
Requirements Categories
Category 1 – WorkLoad
Studying and understanding the specifics that relate to load on the
system in terms of the number of transactions that need to be simulta-
neously processed, or the amount of processing such as arrival patterns,
navigation trends, and user behavior for completion.
Aspects related to Category 1 – Workload include:
▪▪ Online transactions processing
▪▪ Handling batch jobs
▪▪ Transactions related to reports
▪▪ Volume of data in the back end
▪▪ Different user roles in the application
▪▪ Other external interfaces that communicate with applications
Collating of the list of transactions that are Business Critical, High Re-
source consumption, High Frequency of Use and Most Commonly used,
reports that are generic/graphic formats and generic/critical batch jobs
is normally easy, but getting detailed information in respect of the rate
of transactions per minute or per second is not straight forward and
need some formulas to capture them.
For a system in production, this data can be obtained by means of ana-
lyzing transaction history data in databases, and web and system logs
that contain timestamps.
For a new system that is yet to be implemented, this is purely based on
business inputs and assumptions.
Data volumes are easier to estimate for an existing system and they
can easily be captured by analyzing counts from different tables in
the database. Statistics about the number of registered users are usu-
ally available, but estimating the number of concurrent users is never
straightforward.
Category 2 – Performance Targets
Gathering performance targets such as response time, throughput, and
resource utilization.
For performance targets, the related items below cover the majority
of needs:
▪▪ Online interaction response time:
▪▪ Based on the category and type of interaction, for example
in the case of a simple click such as viewing the page and
clicking the next button, it may have a smaller target than a
transaction submit operation that requires entering manda-
tory fields and clicking on a submit button.
▪▪ Network bandwidth considerations: As an example, for an
intranet based application like an HR portal, the response
time may be 8 seconds and applications accessed by end users
that are Internet facing may have response time of 15 seconds
based on the type of operation performed.
▪▪ Sub-transactions that form the complete transaction and the
time for overall processing.
▪▪ Handling and completing online transaction activity: For example,
if a user needs to complete a transaction in 3 minutes the transac-
tion has 5 web interactions to complete, and each transaction re-
quires 20 seconds for data entry, etc. to be successfully completed,
the calculated average response time for a single interaction should
not exceed (3*60−5*20)/5 seconds = 16 seconds.
▪▪ Delivery time for asynchronous transactions: Some factors to be
considered when setting completion time targets are:
▪▪ Based on the type of transaction.
▪▪ Based on the network bandwidth.
▪▪ Based on the number of interactions within the application/
system architecture before reaching the final destination.
▪▪ Transaction throughput: Throughput refers to transactions pro-
cessed per unit time as well as throughput for different type of
reports and their completion time, which need to be classified into
scheduled and ad hoc reports.
By Ramesh Viswanathan
Need for Performance Requirements to
Ensure Reliable Business Applications

2. Testing Experience – 21/2013 35
▪▪ Batch completion time: The completion time needs to be specified
based on what type of batch process/programs are running that
also involve any backup operations
▪▪ Understanding the system resources/resource consumption under
performance targets also plays an important role and a few are
mentioned below:
▪▪ CPU utilization: This is the percentage of time the CPUs of the
system are busy. It is desirable not to have CPU usage of more than
70 %.
3. Memory consumption: This is the number of MB or GB of the
system’s RAM consumed.
4. Disk Utilization: Disk utilization and I/Os per second of the
disks subsystem are often measured to plan for capacity
more holistically.
5. Network bandwidth: The metric can be either in Kbps or
Mbps. It is always a good practice to have a target set for
network consumption overall as well as per user. An example
could be 15Kbps per user and 1 Mbps overall from identified
critical branch to a centralized server. Network bandwidth
targets are dependent on available bandwidth and usage by
users of various roles and geographic locations.
▪▪ In addition to resource consumption counters such as CPU,
Memory, Disk and Network, application specific counters need to be
defined and added as part of monitoring, e.g. for an ASP.NET-based
application, few of the counters like Requests/sec, Requests Execut-
ing, Transactions Total, and Transactions/sec, for SQL counters like
Logins/sec, Logouts/sec, UserConnections, and Buffer Cache Hit
Ratio.
Techniques and Tools for Non-Functional Require-
ments Gathering
Itisalwaysagoodpracticetoproceedusingapropermeansofcapturing
the NFR details, including and not limited to the following:
▪▪ Non-functional requirements questionnaire: This is a list of items
that can assist in forming and understanding the requirements bet-
ter to ensure a quality output. Questionnaire collation for different
sections is provided below:
▪▪ Business test cases requirements
▪▪ Performance testing requirements
▪▪ Monitoring requirements
▪▪ POC requirements
▪▪ External application/server requirements
▪▪ Workload modeling and WLM tools:
▪▪ Identifying objectives and related sub-categories
▪▪ Collating scenarios that are critical and closely related to the
business
▪▪ Determining the associated navigation paths for the critical
and key scenarios
▪▪ Isolating the unique data for the associated navigation paths
as well for the simulated users
▪▪ Finding the distribution of scenarios and each scenario that is
a business test case
▪▪ Categorizing the load levels for different scenarios from the
identified target user load
▪▪ Formulating the engineering approach to instrument the
model
A common means of workload modeling can be performed by under-
standing the web-server logs. There are a few commercial and open
source tools to extract and report the details as needed.
▪▪ Understanding future growth: This forms the capacity planning
approach to cater for handling future loads as the application ma-
tures and more features are implemented to market it long-term
in the competitive e-industry. Some of the high level details are
provided below:
1. Determine service level requirements: The first step in the
capacity planning process is to categorize the work done by
systems and quantify users’ expectations of how that work
will get done. This is done by defining workloads, determin-
ing the unit of work, and identifying service levels for each
workload.
2. Analyze the current capacity: The existing capacity of the
system must be assessed and analyzed to determine how
the needs of the users will be met. This is done by measuring
the service levels and comparing them to objectives, gauging
overall resource usage, and computing the resource usage by
workload.
3. Planning for the future: This involves using forecasting meth-
odologies for future business activity, thereby assessing and
determining future system requirements. After assessment,
the immediately required changes are incorporated in system
configuration. This will ensure there is sufficient capacity and
is made available to maintain the service levels designed
Best Practices
The best practices mentioned below will assist in ensuring that the
requirements are accurate and testing is aligned with them.

3. 36 Testing Experience – 21/2013
1. Engage early with stakeholders and collate details such as trans-
actions performed for a specified period of time, time for each
transaction, duration of peak activity, and usage.
2. Process the collated details, segregate and avoid duplications, e.g.
ensuring requests are from unique IP addresses and filtering all
error-related pages to end up with a work load model.
3. Understand the business needs, e.g. number of peak usages, trans-
actions per second, etc. Map the collated details with business
needs.
4. Organizations need to have a technical and high quality engineer-
ing team that will assist in verifying and validating the require-
ments.
5. Develop a testing strategy and an appropriate testing tool based
on the platform and technology identified.
6. Deriving an accurate work load model with an appropriate test
strategy and supporting test tools ensures that the performance
testing outcome meets expectations and is aligned with business
objectives.
Summary
This article highlights the importance of defining and collecting re-
quirements which is usually given secondary importance compared to
functional requirements.
Theaspectsandparametersofperformancerequirementsareillustrated
with examples to provide a practical thought process.
By using tools, techniques and an engineering approach non-functional
requirements (NFR) gathering can be made more scientific.
Performance can be built into the system and validated at every stage
through the appropriate focus on business, technology, tools, and the
software testing processes as demonstrated in the “Best Practices” sec-
tion. ◼
Ramesh Viswanathan is a Technology Master of Engi-
neering in Communications Systems and a Master of
Business Administration in Operations Management,
and is both Siebel and ISTQB Certified. He currently
worksasaSeniorPerformanceTestArchitectatMarlabs
Inc, USA. He has been in the SoftwareTesting industry
since 1999, working with various organizations in the
past such as ReadyTestGo, CognizantTechnology Solu-
tions, Symphony Software Services, ANZ Bank Information and Technology,
and SunGard Global Solutions
> about the author
License ISTQB®
and IREB®
training materials!
Díaz & Hilterscheid creates and shares ISTQB®
and IREB®
training material that provides the resources you need to
quickly and successfully offer a comprehensive training
program preparing students for certification.
Save money and save time by quickly and easily incorpo-
rating our best practices and training experience into your
own training.
Our material consists of PowerPoint presentations
and exercises in the latest versions available.
Our special plus: we offer our material in four different
languages: English, German, Spanish and French.
Díaz & Hilterscheid Unternehmensberatung GmbH
Kurfürstendamm 179
10707 Berlin
Germany
Phone: +49 (0)30 74 76 28-0
Fax: +49 (0)30 74 76 28-99
E-mail: training@diazhilterscheid.com
Website: training.diazhilterscheid.com
For pricing information and other product licensing requests, please contact us either by phone or e-mail.