Optimize load handling for high-volume tests with IBM Rational Performance Tester

Optimize load handling for high-volume tests with
IBM Rational Performance Tester
Best practices for your load testing environment

Skill Level: Intermediate

Bharath Raj Keshavamurthy (bharathrajbk@in.ibm.com)
Enterprise Solutions Performance Analyst
IBM

24 May 2011

When using IBM® Rational® Performance Tester for testing diversified protocols and
large volume load simulations, it is essential to optimize the performance of your
testing machines and tools, as well as your network and infrastructure. In this article,
you will discover best practices that you can adopt to enhance the load generation
capability of Rational Performance Tester per machine by configuring both the testing
tool and the operating system. You will also learn about techniques that you can use
to alleviate trivial errors that occur during large volume load simulations.
This article is organized into three aspects of preparing your IBM® Rational®
Performance Tester environment. In the first section, you will learn about methods
for deploying the Rational Performance Tester load generation environment, and
information about deployment patterns to avoid. In the second section, you will learn
about tuning parameters that facilitate optimal performance test load generation and
operating system performance. The last section includes information about what you
can do before you launch a performance test run to run a successful performance
test.

Rational Performance Tester deployment practices to adopt
and to avoid
Consider the following suggestions for deploying your Rational Performance Tester
load generation environment:

Optimize load handling for high-volume tests with IBM Rational Performance Tester Trademarks
© Copyright IBM Corporation 2011 Page 1 of 13

developerWorks® ibm.com/developerWorks

• Use many agent machines with low-end configurations (in terms of
number of CPUs and amount of memory) and have these machines
generate load. If you use fewer agent machines with high-end
configurations, then the operating system is limited by the total number of
available sockets, processing parallelism, and limits on multi-threading.
Therefore, use many agent machines with low-end configurations that are
horizontally scaled out for performance testing, rather than vertically
scaling agents on a few high-end machines.
Note:
Low-end configuration machines are those with a low number of CPUs
and memory. However, the CPU speed should not be considered as low
configuration. Recommended CPU speed for Rational Performance
Tester agent machines is 2.33 GHz or greater.
Figure 1. Comparison of different topologies in designing the load generation
environment

Larger view of Figure 1.


ibm.com/developerWorks developerWorks®

• If the test agents are in a virtual environment, ensure that the logical
partitions on the machines where the performance test agent software is
installed have dedicated physical Ethernet controllers to support high
volume network traffic.
Important:
Do not use logical host Ethernet adapters or virtualized Ethernet devices
as networking interfaces for logical partitions that run performance test
agent services on Rational Performance Tester agent logical partitions.
• Ensure that all agents are in the same network as the controller and the
performance test setup. Doing so ensures that there is minimal network
latency and enables faster transfer of periodic statistics, test assets, and
data between the host controller and agent machines.
Good topology decisions

The first two topologies are illustrated in figures 2 and 3. They are best practices that
you can adopt for deploying Rational Performance Test environment

Topology 1: Best mechanism for performance testing the solution

During performance testing, you must host the load generation environment in the
same network as that of the solution that is being tested so that there is minimal
network latency between the load generation environment and the solution. After
you minimize network latencies, it is not only easier to diagnose performance
bottlenecks, but there is also a decrease in network errors and packet drops that can
be difficult to troubleshoot, and might lead to transaction errors, despite the server
operating correctly.

Figure 2. First topology for the performance test environment




Topology 2: Rational Performance Tester host controller is outside of the
solution network, but the performance test agents are inside

This topology is almost the same as the previous one. Here, there can be
communication problems between the host controller and its agents due to varying
networks issues, such as the bandwidth of the network pipe and the network cards
of the servers that host Rational Performance Tester. With that, the shortcoming of
this mechanism is primarily seen in issues that prevent smooth execution of
performance tests in the following situations:

a. General execution of performance test runs or execution of performance
test runs for large durations / Endurance performance tests which run for
days together and with quick sampling rates

b. There are many test agents involved for load generation purposes,
approximately 25 test agents or more.

If the network connectivity and bandwidth between the host controller and its agents
are not uniform, then this might cause communication errors (network packet losses)
and further result in abnormal termination of performance tests. The following errors
might occur:

• "Workbench did not receive any message from Agent for more than xx
milliseconds"



• "Could not connect to Agent. Connection timed out"
Figure 3. Second topology for performance testing


Deployment patterns to avoid
Avoid the topology patterns illustrated in Figures 4 and 5 when you deploy your
Rational Performance Tester environment.

Pattern 1: Rational Performance Tester agents are deployed on the same
hardware as the solution components

Although this topology of performance testing guarantees minimal network latency
between performance test agents and the solution, the operating system resources
are shared between the solution and performance test environment. In this case,
resource utilization analysis becomes very difficult due to extreme complexity
involved in differentiating resource utilization of the same operating system between
the solution environment and the performance test environment. Inaccurate analysis
leads to inaccurate data for performance issues identification, analysis, and
troubleshooting. Therefore, this topology is not suggested for performance test
environment deployment.

Consider the following operating system resources:

• Processor use
• Memory use
• TCP/IP sockets for both the solution component and agent processes



• Network bandwidth, including the number of bytes sent and received per
second for both the solution component and agent process
Note:
Deploying performance test agents on solution components also has an impact on
the use of available TCP/IP sockets at the operating system layer in addition to
process, memory and network bandwidth utilization.

Rational Performance Tester creates connections for the requests that it makes
during performance test load generation. Every connection uses TCP/IP sockets.
This number is limited to only 65,535 (as the TCP/IP protocol allows only 65,535
TCP/IP ports to be available at any operating system platform).

If the performance test agents use a large number of TCP/IP sockets, then the
solution components have a shortage of socket connections, which creates a
performance bottleneck and leads to poor performance.

Figure 4. First pattern to avoid for performance testing


Pattern 2: The entire load generation environment (host controller and test
agents) is out of the solution network

Avoid this topology for testing the solution because it results in network errors,
packet drops, and so forth. Thus this topology not only leads to disruptions and
errors during performance testing, but also becomes difficult to drive the desired
throughput and load on solution servers.

Figure 5. Second pattern to avoid for performance testing




Performance tuning for your test environment
The following tuning parameters for Rational Performance Tester have been tried
and tested successfully for generating a large user volume load of 35,000 virtual
users with a user concurrency of 3,000 virtual user against a 3 tier web-based
application that uses HTTP protocol.

You can apply these parameters when you test a web application against a user
load between 1000 – 10,000 concurrent virtual users.

Disclaimer: These tuning parameters have been noted to provide
25 – 40% improvement in the network throughput behavior of the
operating system and are subjected to the application workload
characteristics as well. The performance improvement that was
achieved varied by 5 – 10% on different platforms. The AIX platform
had the most significant improvement.
These parameters are a reference point. You can map them against
the application workloads and volumes that you are testing so that
you get an appropriate set of values for your tests.

Table 1: Operating system tuning for optimal performance in load generation
OS Parameter Value Implementation
details



Windows MaxUserPort 65534 Edit the registry under
HKEY_LOCAL_MACHINE
TcpTimedWaitDelay 30 > System >
MaxFreeTcbs 72000 CurrentControlSet >
Services > TCPIP >
MaxHashTableSize 65535 Parameters. All the
TcpWindowSize 65535 parameters have to
added as a DWORD
EnableDynamicBacklog 1 and the values set as
Decimal
MinimumDynamicBacklog20
MaximumDynamicBacklog000
1
DynamicBacklogGrowthDelta
10
TcpAckFrequency 1
Linux net.ipv4.ip_local_port_range 65000
2048 Edit /etc/sysctl.conf
with the following
net.core.rmem_default 262144 parameters and save
net.core.wmem_default 262144 the file. Run the
following command –
net.core.wmem_max 33554432 sysctl –p
net.core.rmem_max 33554432
net.core.netdev_max_backlog
5000
net.ipv4.tcp_no_metrics_save
1
net.ipv4.tcp_rmem 4096 16777216
33554432
net.ipv4.tcp_wmem 4096 16777216
33554432
net.core.optmem_max 40960
AIX sb_max 6192000 no –p –o
<parameter> =
clean_partial_conns 1 <value>
tcp_timewait 1 chdev -l sys0 -a
maxuproc=<value>
tcp_keepidle 600
tcp_keepinit 40
tcp_nodelayack 1
tcp_keepintvl 10
Maxuproc 8192
tcp_sendspace 4096000
tcp_recvspace 4096000
tcp_ephemeral_low 1024
rfc1323 1



Practices to consider before you launch a performance test

• Ensure that the Rational Performance Test agent service has been
started on the agent machines.
• Ensure that there are no connections in CLOSE_WAIT or TIME_WAIT
that use the netstat command before you start the tests from the agent.
Note:
TIME_WAIT connections happen when the server closes its connection,
but the client waits for an acknowledgement signal to close the
connection. Connections go to CLOSE_WAIT after exceeding the time
duration under the TIME_WAIT state and the operating system must
close this connection. Having a connection in any of these two states
before you launch a performance test prevents this connection from being
used by Rational Performance Tester during the test run, which suggests
that no occurrences of these types of connections are there before the
test. This ensures a clean network state before launch.
• For agent machines on UNIX operating systems, look for a process
named rpt_runner and stop any occurrence of it before you run a
performance test. In Windows, look for Java in the Task Manager, and
stop any occurrences of the same before you start your test.
Note:
Stopping the rpt_runner process in Linux, AIX, or Java in Windows
does not stop the Rational Performance Tester agent service. The agent
process is started at the beginning of every test launch. Hence, stopping
these processes before launch of next test (if any exist after a test run
completes) ensures that processes of the agent that is associated with
previous tests are cleaned up from the operating system. These
processes might continue to exist after test completion due to abrupt
termination of performance test or errors during performance test.
• On Microsoft Windows machines, enable the show heap status option on
the host controller: Windows > Preferences > General > Show heap
status. You can use this to keep track of memory utilization for Rational
Performance Tester during tests. If the memory utilization increases, force
the garbage collection manually by clicking on the garbage collection icon
in the heap status bar. Knowing this helps you size the memory for the
host controller machine.
• Close the host controller workbench and launch it again to achieve
minimal use of the JVM heap memory before you start your test. You
must force garbage collection once to clean up any unreferenced objects
in the Rational Performance Tester JVM heap memory before starting
your test.



• Delete all the contents in the following directories and restart the agent
server before you run your test:
• Temp directory of the operating system. (/tmp for Unix based
operating systems, and C:Documents and
Settings<User>temp directory for Windows)
• Deployment_root directories of agents
• Shared class resources that are present in C:Documents and
Settingsuser nameLocal SettingsApplication
Datajavasharedresources
• In certain cases, you must restart the operating system of the
performance test agent machines.
• Ensure that adequate disk space is available in temp directory and the file
system that contains the deployment_root folder. Recommended
minimum value for the temp directory is 10 GB, for the deployment
directory is 5 GB.
• When generating high loads and considering sync points, use many small
configuration agent machines rather than fewer large configuration
machines to get better results
• If the test duration is long or involves high throughput generation of
requests, then avoid statistics sampling at a fast rate. A high statistics
sampling rate is overhead for the performance test tool and must be kept
at an optimal rate. For example, in a 2-hour test run, a 15-second
sampling interval is an optimal statistics sampling rate.
Figure 6. Statistics sampling rate in Rational Performance Tester

• Set the error logs to minimum logging. Only errors and warning can be
logged for primary test actions when running large volume tests to
achieve optimal performance in load generation.
Figure 7. Logging level options in Rational Performance Tester



Resources
Learn
• Find out more on the Rational Performance Tester product overview page.
Then explore the Rational Performance Tester page on IBM®
developerWorks® for links to technical articles and browse the user assistance
in the Rational Performance Tester Information Center.
• Visit the Rational software area on developerWorks for technical resources and
best practices for Rational Software Delivery Platform products.
• Stay current with developerWorks technical events and webcasts focused on a
variety of IBM products and IT industry topics.
• Attend a free developerWorks Live! briefing to get up-to-speed quickly on
IBM products and tools, as well as IT industry trends.
• Watch developerWorks on-demand demos, ranging from product
installation and setup demos for beginners to advanced functionality for
experienced developers.

• Improve your skills. Check the Rational training and certification catalog, which
includes many types of courses on a wide range of topics. You can take some
of them anywhere, any time, and many of the "Getting Started" ones are free.
Get products and technologies
• Download the trial version of IBM Rational Performance Tester.
• Evaluate IBM software in the way that suits you best: Download it for a trial, try
it online, use it in a cloud environment, or spend a few hours in the SOA
Sandbox learning how to implement service-oriented architecture efficiently.
Discuss
• Join the Performance Testing forum, where you can share you questions and
knowledge about IBM performance testing products.
• Share your knowledge and help others who use Rational software by writing a
developerWorks article. You'll get worldwide exposure, RSS syndication, a
byline and a bio, and the benefit of professional editing and production on the
developerWorks Rational website.
• Follow Rational software on Facebook and Twitter (@ibmrational), and add your
comments and requests.
• Ask and answer questions and increase your expertise when you get involved
in the Rational forums, cafés, and wikis.
• Connect with others who share your interests by joining the developerWorks



community and responding to the developer-driven blogs.

About the author
Bharath Raj Keshavamurthy
Bharath Raj works at IBM India Software Labs with the High
Performance OnDemand Solutions (HiPODS) team as an Enterprise
Solutions Performance Analyst. He started his career with IBM as a
J2EE application developer using Web 2.0 technologies. He has
worked on the performance of middleware, Java, and databases that
are used for enterprise solutions in telecommunications, retail, and
other industries. With an intensive background and experience in
performance tuning, capacity planning, testing and engineering
methodologies, Bharath Raj applies a holistic approach to the
performance of an enterprise solution. He continues to use cutting edge
technologies and imbibe them in enterprise web applications to achieve
an optimized, faster and much more reliable solution.


Optimize load handling for high-volume tests with IBM Rational Performance Tester

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Optimize load handling for high-volume tests with IBM Rational Performance Tester

Semelhante a Optimize load handling for high-volume tests with IBM Rational Performance Tester (20)

Mais de Bill Duncan

Mais de Bill Duncan (20)

Último

Último (20)

Optimize load handling for high-volume tests with IBM Rational Performance Tester