The Art of Performance Strategy by Bharath Kumar Hemachandran & Samar Ranjan

1. © 2022 Thoughtworks | Confidential
The Art of Performance Strategy
Bharath Hemachandran - Principal Consultant
Samar Ranjan - Lead Consultant

2. © 2022 Thoughtworks | Confidential
Why did we want to present on this topic?
The reasons for the existence of
Performance testing is not well understood
2
Performance is commonly an after-thought
in the dev cycle
People do not think about performance
testing holistically
A well crafted performance strategy is
hard to come by
It is more about the tool than the process
leading up to testing
Performance is seen as primarily a
scripting (not coding exercise)

3. © 2022 Thoughtworks | Confidential
What is performance?
3

4. © 2022 Thoughtworks | Confidential
Performance of the System Under Test
4
What does it depend on?
Depends on the key business requirements, the Architecture and Tech Stack of the System Under
Test

5. © 2022 Thoughtworks | Confidential
Key Performance Parameters
5
What defines Performance of a System Under Test?
Response Time Responsiveness Availability
Reliability Resource Usage Throughput
Stability Error-Handling
Scalability

6. © 2022 Thoughtworks | Confidential
Key Parameters for Interfaces
6
Response Time Responsiveness Availability
Reliability Resource Usage Throughput
Stability Error-Handling
Scalability

7. © 2022 Thoughtworks | Confidential
Key Parameters of an IOT System
7
Response Time Responsiveness Availability
Reliability Resource Usage Throughput
Stability Error-Handling
Scalability

8. © 2022 Thoughtworks | Confidential
Key Parameters for an AI System
8
Response Time Responsiveness Availability
Reliability Resource Usage Throughput
Stability Error-Handling
Scalability

9. © 2022 Thoughtworks | Confidential
Creating a strategy
9

10. © 2022 Thoughtworks | Confidential 10
What would a perf strategy look like?
Purpose
Stakeholders
Goals & Objectives of Performance Tests
Tool Comparison
Converting workflows to Performance Workloads
Type of tests to run
Explanation/Evaluation of Tool Choice(s)
Executions
Reporting
Insights

11. © 2022 Thoughtworks | Confidential
Market analysis/research data (New
application)
Factors to consider
It is essential to understand the
business & as well as technological
factors. It determines the direction
where all the stakeholders interest lies.
11
Know the application usage
User workflow
Data interactions
Constraints & Dependencies
Analytics/Traffic (Existing application only)

12. © 2022 Thoughtworks | Confidential 12
Market analysis/research data
Image source: https://amazeemetrics.com

13. © 2022 Thoughtworks | Confidential 13
Know the application usage
Image source: https://indiamart.com

14. © 2022 Thoughtworks | Confidential 14
User workflow
Image source: https://zenaton.com

15. © 2022 Thoughtworks | Confidential 15
Data Interactions
Image source: https://sourcecodehero.com

16. © 2022 Thoughtworks | Confidential 16
Constraints & dependencies

17. © 2022 Thoughtworks | Confidential 17
Analytics/Traffic
Image source: https://oreilly.com

18. © 2022 Thoughtworks | Confidential
Who is who in your
performance test
Stakeholders play a big role in
performance test strategy. Each stake
holder provides certain information
without which the test might be useless.
Business/Domain
Specialist
Architect/Dev
Lead
18
Dev Ops
Product Owner

19. © 2022 Thoughtworks | Confidential
Identification of an actual user flow &
prioritizing the most expected user
journey
Creating a
performance load
design
Your stakeholders provides you the
pattern for the performance load & you
need to follow same to build
performance load design.
19
Pattern of user entry and exit
Test Data
Tools & monitoring
Load & Concurrency
Environment

20. © 2022 Thoughtworks | Confidential
Performance
Testing
Load
Stress
Soak
Spike
Configur
ation
Volume
Responsiveness Stability Resource Usage Scalability Reliability
SUT under
various loads
SUT under
heavy loads
X times peak
load
Sustained use
of SUT under
load
SUT with
different
configurations
SUT with sudden
increases/decrea
ses in load
SUT with sudden
influx of data
Types of Performance Tests
20

21. © 2022 Thoughtworks | Confidential
21
Planning for Performance Testing
Things to keep in mind
Measures
Reporting
Load & Concurrency
Response
Times
Resource
Usage
Benchmarks
Logging
Throughput
Load
Scenarios
Environment/
Infra Capacity
Virtual User
Simulation Mix
Load
Breakdown
Report
Templates
Data Store Data Viz
Failure Rate
21

22. © 2022 Thoughtworks | Confidential
Tools for monitoring
APM - Application Performance
Monitoring tools are essential to
performance test as they provide real
time information about the server side
during the test & post test analysis.
22
Cloudwatch
(AWS)

23. © 2022 Thoughtworks | Confidential
Tools for monitoring
Log management tools provides real
time application log information. You
will have data about application
response, errors and matrices around
received request & processed
transactions.
23

24. © 2022 Thoughtworks | Confidential
24
Which tool do I choose?
Gatling Locust Jmeter K6
Developed/Code in Scala, Java, Kotlin Python (v3.8 or above) Java/Groovy,beanshell Go,/Javascript
Open Source License Apache License v2.0 MIT License Apache License v2.0
GNU Affero General Public
License v3.0
Learning Curve Steep Easy Medium Easy
Load Design Flexible Flexible Medium Flexible
Reporting Out of the box Out of the box Out of the box/configurable
Integration with 3rd party
dashboard
Virtual User Simulation
Actor Model (Async Akka
Netty) gEvent Model JVM Thread based Goroutine based
Assertions Library Out of the box Out of the box Configurable Out of the box
Run different types of users in the
same Simulation Out of the box Out of the box Out of the box Out of the box
Readability Medium Easy Hard Easy
Extensibility Medium Easy Hard Easy
Code Reuse Easy Easy Medium Easy
Runs in Pipelines Out of the box Out of the box Out of the box Out of the box
Github Stars/Forks for Project 5.6k/1.1k 19.5k/2.5k 6.4k/1.8k 17.5k/913
Protocols supported HTTP/HTTPS/gRPC HTTP/HTTPS/gRPC
HTTP, HTTPS SOAP, FTP,
JDBC, LDAP, MOM via JMS,
SMTP, POP3, IMAP, shell
scripts, TCP, Java objects HTTP/HTTPS/gRPC

25. © 2022 Thoughtworks | Confidential 25
What would a perf strategy look like?
Purpose
Stakeholders
Goals & Objectives of Performance Tests
Tool Comparison
Converting workflows to Performance Workloads
Type of tests to run
Explanation/Evaluation of Tool Choice(s)
Executions
Reporting
Insights

26. © 2022 Thoughtworks | Confidential 26
Scripting best practices
What does a good framework look like?
26
Well Organized &
Structured
Separate data from
actions
Test Extensibility
Design Patterns are
your friend
Parameterization
Reporting and
Debugging
Workflows

27. © 2022 Thoughtworks | Confidential
Is CICD worth doing in Performance
testing?
27

28. © 2022 Thoughtworks | Confidential
Best practices around execution
28

29. © 2022 Thoughtworks | Confidential
Sample reports/Case studies/Examples
Typical output from a Performance Tool - What to look for
29

30. © 2022 Thoughtworks | Confidential
Sample reports/Case studies/Examples
Typical output from Monitoring Platforms - What to look for
30

31. © 2022 Thoughtworks | Confidential
What do you do with this data?
Engaging back with stakeholders with the data
31
Optimal Infrastructure
Requirements
Highlight Performance
Issues
Capacity Planning
● Assessing infrastructure needs
● Establishing benchmarks for resource usage and response times
● Identifying optimal configurations
● Planning future scalability of the system
● Benchmark comparison. Renegotiate/redo benchmarks if required
● Enabling team to identify bottlenecks and (potential) problem areas
● Engage devs to increase visibility/transparency of code
● Regression testing when new code/configurations are added
● Identify new data/scenarios that can lead to more discoveries
● Enable cost optimization through resource usage planning
How often should you run tests?

32. © 2022 Thoughtworks | Confidential 32
Response Time
Metrics to capture -
Server side
The total time between
sending the request
and receiving the
response
Throughput
The number of
transactions an
application receives per
second.
Request Per Second
The number of
requests handled by
the application per
second
Error Rate
The percentage of
requests resulting in
errors compared to the
total number of
requests.
CPU Utilization
The percentage of CPU
capacity utilized in
processing the
requests.
Memory Utilization
The utilization of the
primary memory of the
computer while
processing any
requests.
You need to use appropriate monitoring
tool for these matrices.

33. © 2022 Thoughtworks | Confidential 33
Time to interact
Metrics to capture -
Client side
The time taken by a
website to become fully
interactive
Speed Index
It measures how
quickly the content is
displayed during page
load
Load Time
The average amount of
time a page takes to
show up on your
screen.
Payload
It is the difference
between essential
information in a chunk
of data and the
information used to
support it.
Time to First Byte
The time between the
browser requesting a
page and when it
receives the first byte
of information from the
server.
Page Size/Weight
The overall size of a
particular webpage.
While load tool provides certain matrices,
you need another set tools for clients like
browser.

34. © 2022 Thoughtworks | Confidential
So what should we take away from this
discussion?
34
Performance of a system is more than just
about its response time The goal(s) and objectives of performance
should be defined by the key stakeholders
Always create a performance strategy before
starting testing. This should always address
the goals/objectives
A performance strategy should translate
workflows into scenarios to be evaluated
The kind of tests to be run should align with
the end goals/objectives
Bharath and Samar are the greatest QAs in the world
Do all of the above before scripting even a
single line of code

35. © 2022 Thoughtworks | Confidential
We look forward for the feedback!
bharathkumar.h@thoughtworks.com LinkedIn
samarjan@thoughtworks.com Linkedin
35

36. © 2022 Thoughtworks | Confidential
36
Planning for performance testing
Things to keep in mind
Measures
Reporting
Tooling
Load
Response
Times
Resource
Usage
Benchmarks
Logging
Throughput
Load
Scenarios
Environment/
Infra Capacity
Virtual User
Simulation Mix
Load
Breakdown
Report
Templates
Data Store Data Viz
Failure Rate
36

37. © 2022 Thoughtworks | Confidential
A performance test strategy reference
https://bahmni.atlassian.net/wiki/spaces/BAH/pages/3038445574/Performance+Benchmarking+and
+Capacity+Planning
37