about performance testing of software.

1. Performance testing

2. • The testing performed to evaluate the response time, throughput,
and utilization of the system, to execute its required functions in
comparison with different versions of the same products or a
different competitive product is called performance testing.
• It is done to ensure that a product,
– # of transactions processed by the product in particular period of time
(throughput)
– Delay between the point request and the first response from the
product (response time)
– Delay caused by the application, OS, by the environment (latency)
– Is available and running under different load conditions (availability)
– deciding what kind of resources are needed for the product for
different load conditions (capacity planning )
– Is comparable to and better than that of the competitors for different
parameters.(bench marking)

3. Methodology for performance testing

4. • Performance testing is complex and expensive
due to large resources requirement and time
it takes.
• Hence this testing need careful planning.
• A good n# of defect uncover may require
design and architecture change

5. Collecting requirements
• It is the first step in planning the performance testing
• Testing need elaborate documentation and
environment setup and expected result may not well
known in advance.
• This requirement should be testable
– Feature involving manual intervention cannot be tested
because it depends on how fast user responds with input
• Requirement needs to clearly state what factor need to
be measured and improved
• Requirements need to be associated with numbers or
percentage that is desire.

6. • Sources for deriving requirements of performance
testing
– Performance compared to the previous release of the
same product
• Atm withdrawal will be faster than previous release by 10%
– Performance compared to the competitive products
• Faster than competitive bank ATM
– Performance compared to absolute numbers derived from
actual need
• Atm capable of 1000 transaction /day, each transaction not taking
more than 1 minute
– Performance numbers derived from architecture and
design
• There is an expectation that the source code is written in such a
way that those numbers are met

7. • Eg: performance test requirement

8. • Two type of requirements
– Generic requirement
• All the products in that area should meet those
performance expectation
– Time taken to load a page, time taken to navigate from one
screen to other
– Specific requirement
• Depends on implementation particular product
– Time taken to ATM withdrawal

9. Writing test cases
• Test case should have
– List of operations or business transactions to be tested
– Steps for executing those operations or transactions
– List of product, OS parameters that impact the
performance testing
– Resource and their configuration (network, hardware)
– The expected results (response time, throughput,
latency)
– Product version / competitive product to be compared

10. • Test cases execute repeatedly for different
parameter, configuration
• It need more effort and time so not all
transactions are included in testing
– Prioritize test cases, so that highest priority test
cases executes before other

11. Automating performance test cases
• Performance testing is automated because
– Performance testing is repetitive
– Performance test cases cannot be effective without
automation
– The results need to be accurate
– Performance test takes in to account several
parameter and combinations that need to
remembered
– Analysis of performance take in to account of resource
utilization, log file, trace file that need to be collected
at regular interval

12. • End to end automation is required for
performance testing.
• Not only the steps of test cases but also setup
required for test cases, creating different load
condition, executing the steps for transaction
of competitive product are have to be scripted
• So we need to use standard tools and
practices.

13. Executing performance test cases
• Performance testing involves less effort for execution
but more effort on planning, data collection and
analysis.
• Executing performance test cases may mean involving
certain automated script.
• Data need to executing performance test cases
– Start and end time of execution
– Log and trace file of the product
– Utilization of resources on a periodic basis
– Configuration of all environment
– Response time, throughput, latency and so on as specified
in test cases documentation at regular intervals

14. • What performance a product delivers for
different configuration of hardware and network
setup is need to be included during execution –
configuration performance testing
• This endures that performance of product is
compatible with different hardware.
• Once performance test are executed and various
data point are collected, next step is to plot them.
• Plotting the data help in making a quick analysis.

15. Analysis the performance test result
• Analysis the performance test result require multi-
dimensional thinking.
• This is the most complex part of performance testing where
product knowledge, analytical thinking and statistical
background are essential.
• Some calculations are required before analysis
– Calculating mean the performance test result data
– Calculating the standard deviation
– Removing noise and re-plotting and calculating mean, std
deviation
– Differentiating the performance data when the resources are
available completely as against when some background
activities were going on.

16. • For publishing performance numbers, performance test are
repeated multiple times and the average of those values
are taken.
• This increases the chance that performance data can be
reproduced at customer site.
• It is also depends on consistency of the product delivers
those performance number.
• Standard deviation is represents how much the data varies
from the mean.
• There are some data out of range may cause the graph to
be cluttered and prevent meaningful analysis, such value
has to removed – such process is called noise removal.

17. • Some of activities such as garbage collection/ defragmentation in
memory management of operating system, are initiated in
background and degradation in performance may be observed.
• Once all done, the analysis of performance data is carried out to
conclude
– Whether performance of the product is consistent
– What performance is expected for what types of configuration
– What parameters impact performance and how they can be used to
derive better performance
– What is the effect of product technologies such as caching on
performance improvement
– What is optimum throughput/response time of product for a set of
factors(resources and load)
– What performance requirements are met compared to old version and
competetitive product

18. Performance tuning
• Analysis performance data helps I narrowing
down the list of parameter that really impact the
performance result and improve product
performance.
• Performance test cases are repeated for those
parameters for further analysis.
• The combination of those parameter can also
cause changes in performance
• Steps to optimize performance tuning
– Tuning the product parameter
– Tuning operating system and parameter

19. • Important note for tuning product parameter
– Repeating the performance test for different
values of each parameter.
– Some parameter changes may also need to
change other parameter changes.
– Repeat the performance test for default values of
parameters.
– Repeat performance test for low and high values
of each parameter.

20. • Various parameters provided by the operating system
– File system relates parameters (# of open files permitted)
– Disk management parameter(simultaneous read/write)
– Memory management parameter (page size, number of
pages)
– Processor management parameter (multiprocessor)
– Network parameter (TCP/IP setting)
• Result of performance tuning are published in the form
of guide- called performance tuning guide.

21. Performance benchmarking
• Performance benchmarking is about comparing the
performance of the product with competitive product .
• No two product have same architecture, design,
functionality, ,hence it is difficult to compare.
• End user transaction/scenario could be one approach to
comparison.
• Steps involved in performance bench marking:
– Identifying the transactions / scenarios and the test
configuration
– Comparing the performance of different products
– Tuning the parameters of the products being compared fairly to
deliver the best performance
– Publish result of the performance bench marking

22. • Step1:
– Comparable transactions are selected for performance benchmarking
– Test cases for all the products are executed in the same test bed
• Step 2:
– Once tests are executed, compare the results.
– It is important that in performance benchmarking all product should be tuned to the
same degree.
– There are three outcomes
• Positive: outperform with respect to competition.
• Neutral: set of transaction are comparable with competition
• Negative: under-perform compared to competition
• Step 3:
– third outcome detrimental for the success of produce, hence performance tuning need
to be performed for this set of transactions using same configuration
– Repeat this tuning for all situations(+ve, neutral, -ve)
• Step 4:
– Internal publish has all three outcomes and recommended set of actions
– Positive outcomes are published as marketing guarantee

23. Capacity planning
• In capacity planning, performance result and requirements are taken as
input and the configuration needed to satisfy that set of requirements are
derived
• Capacity planning necessitates a clear understanding of the resource
requirement for transactions.
• Some transactions required CPU, some require network intensive and so
on
• Some transactions require combination of resources for performing better.
• This understanding is prerequisite for capacity planning
• Load pattern can
– Minimum required - immediate need (short term),
– Typical configuration, under that configuration product work fine for meeting
the performance requirements (medium term)
– Special configuration – planning for future requirements (long term)

24. • Load balancing
– Ensures that the multiple machines available are
used equally to service the transactions
• Availability
– Machine clusters ensures availability.
– In a cluster there are multiple machines with
shared data so that one machine goes down the
transaction can be handled by other machine

25. Tools for performance testing
• Two types
– Functional performance tools
• Help in recording the playing back the transactions and obtaining
performance numbers
• Eg:
– Winrunner from mecury
– QA partner from compuware
– Silktest from segue
– Load tools
• Simulate load conditions for performance testing without having
to keep that many users or machines
• eg:
– Loadrunner from mecury
– QA Load from compuware
– Silkperformer from segue

26. Process for performance testing

27. Obtain measurable testable
requirement
Create a performance test plan
Design test cases
Automate test cases
Evaluate exit criteria
Perform and analyze performance
test cases
Evaluate entry criteria

28. • Test plan need the following details
– Resource requirements
• Resources needed to perform performance testing need to be
planned and obtained
– Test bed, test lab setup
• Testing configuration in simulated or real life environment
– Responsibilities
• Multiple team are there to perform this test
• Prepare a matrix of responsibilities
– Setting up product trace, audits and traces
• What traces and audit has to be collected in planned in advance
– Entry and exit criteria
• Performance test require a stable product due to its complexity
• Performance test starts after product met certain criteria
• Similarly set of criteria is defined to conclude the result of
performance test