3. Background
When software developers are concerned with performance of their system.They
may resort to these options
Performance Testing to determine the performance of an already built system.
MSDN provides a very thorough guide on the subject.
Profiling to analyze and investigate bottlenecks when a system is running.
Benchmarking to compare the relative performance of systems.
Analysis to determine the algorithmic complexity (Big-O notation).
4. Types of Benchmarking
This leads to two commonly known types of benchmarks:
Macrobenchmarks are used to test entire system configurations. Macro
benchmarks are done between different platforms to compare the efficiency
between them.
Microbenchmarks are used to compare different implementations in an isolated
context, for example a single component. Micro benchmarks are done within
the same platform for a small snippet of code.
5. MicroBenchmarking
Micro benchmarks are generally done for two reasons.
To compare different approaches of code which implements the same logic and
choose the best one to use.
To identify any bottlenecks in a suspected area of code during performance
optimization.
So benchmarking is used for comparisons.Benchmark is the process of recording
the performance of a system
6. Factors in benchmarking
Benchmark candidate: What piece of software do we benchmark?
Comparison against a baseline:determined by customer requirements or you might be just looking for
the best relative performance in a specific scenario among a set of benchmark candidates.
Metrics: Which metrics do we use to determine performance? Like Throughput or Average time
Benchmarking scenario: Do we consider single-threaded or multi-threaded performance? How does a
data structure behave when accessed concurrently by multiple writers?
Benchmarking duration
7. Why are hand-written Benchmarks bad
Because you need to take into account these factors
JVM consists of three main components that work together: the runtime including the interpreter, the
garbage collector and the JIT-compiler. Due to these components, we neither know in advance
which machine code will be executed nor how it behaves exactly at runtime
Oracle's HotSpot JVM applies a vast amount of optimizations on Java code(more than 70 optimization
technique)
Some compiler optimizations like dead code elimination, loop unrolling, lock coalescing and in-
lining. You might be benchmarking a different code than what you are thinking.
8. Why are hand-written Benchmarks bad
Each method is executed in interpreted mode at first. The Java interpreter requests that it should be
JIT-compiled. Consequently, we have to run the benchmarked code often enough before the actual
measurement starts to ensure that all benchmarked code has been JIT-compiled beforehand.. You
should not see any JIT-compiler activity after the warmup phase.
Benchmark code falls victim to dead code elimination: In certain circumstances the JIT-compiler may
be able to detect that the benchmark does not do anything and eliminates large parts or even the
whole benchmark code.
False sharing: In multithreaded microbenchmarks, false sharing can severely affect measured
performance. See false sharing
9. Why are hand-written Benchmarks bad
Reliance on a specific environment: The JVM version, the OS and the hardware
could be different in a microbenchmark and an application. whether its is
single core or multi-core or hyper threaded and its impact on your program to
benchmark.
When running on the same environment, we need to remember to switch off all
other programs. Machine should be silent. Background processes can
compete for resources and cause delay.
10. Warm up phase in Benchmarking
Before recording the numbers, do multiple runs of the code snippet to warm up
the environment. This is to initialize the environment. Java JIT takes time to
analyze and optimize the code on initial runs. We should give enough number
of iterations for it to stabilize otherwise we will end up adding the JIT
overheads to the performance.
Similarly we may not get the caching benefits that happens at different levels.
11. Creating your first benchmark
mvn archetype:generate
-DinteractiveMode=false
-DarchetypeGroupId=org.openjdk.jmh
-DarchetypeArtifactId=jmh-java-benchmark-archetype
-DgroupId=org.sample
-DartifactId=test
-Dversion=1.0
If you want to benchmark an alternative JVM language, use another archetype
artifact ID from the list of existing ones,
12. Creating your first benchmark
Building the benchmarks. After the project is generated, you can build it with the
following Maven command:
$ cd test/
$ mvn clean install
Running the benchmarks: $ java -jar target/benchmarks.jar
Archetypes for kotlin, groovy, scala and java are provided.
13. Understanding JMH code
We have already completed the first step by annotating a method with
@Benchmark.
JMH implements multiple annotation processors that generate the final
microbenchmark class. This generated class contains setup and
measurement code as well as code that's required to minimize unwanted
optimizations of the JIT compiler in the microbenchmark.
JMH contains a Runner class somewhat similar to JUnit so it is possible to run
embedded microbenchmarks using the JMH Java API.
14. Understanding JMH
You can see that JMH creates multiple JVM forks. For each for fork, it runs n
warmup iterations (shown in blue in the picture below), which do not get
measured and are just needed to reach steady state before m iterations are run
(shown in red in the picture below).
15. BenchMark Modes
Throughput: Rate at which the processing is done.
@BenchmarkMode({Mode.Throughput}) calculates the operations per
second. The timebound can be configured.
Average Time: Measures the average execution time.
@BenchmarkMode({Mode.AverageTime}) calculates seconds by operations.
The timebound can be configured. Its the reciprocal of throughput.
16. Benchmark Time Unit
JMH enables you to specify what time units you want the benchmark results
printed in. The time unit will be used for all benchmark modes your
benchmark is executed in.
You specify the benchmark time unit using the JMH annotation
@OutputTimeUnit. The @OutputTimeUnit annotation takes a
java.util.concurrent.TimeUnit as parameter to specify the actual time unit to
use.
17. Benchmark State
Sometimes you way want to initialize some variables that your benchmark code needs, but which you
do not want to be part of the code your benchmark measures. Such variables are called "state"
variables. State variables are declared in special state classes, and an instance of that state class
can then be provided as parameter to the benchmark method.
@State annotation signals to JMH that this is a state class.
A state object can be reused across multiple calls to your benchmark method. JMH provides different
"scopes" that the state object can be reused in. There state scope is specified in the parameter of
the @State annotation.
18. Benchmark State
State Scopes
A state object can be reused across multiple calls to your benchmark method.
JMH provides different "scopes" that the state object can be reused in. Their state
scope is specified in the parameter of the @State annotation.The Scope class
contains the following scope constants:
Thread - Each thread running the benchmark will create its own instance of the
state object.
Benchmark-All threads running the benchmark share the same state object.
19. Benchmark State class Requirments
A JMH state class must obey the following rules:
The class must be declared public
If the class is a nested class, it must be declared static (e.g. public static class ...)
The class must have a public no-arg constructor (no parameters to the constructor).
When these rules are obeyed you can annotate the class with the @State annotation to make JMH
recognize it as a state class.
