In this session, we will introduce you to JMH, an OpenJDK harness for building, running and analysing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM. It should help you find spots to optimise performance and, which may be even more important, it will show you parts that you don't really need to optimise. It not only will make your benchmarks more accurate, but also much easier to write.
4. Donald E. Knuth
Professor Emeritus at Stanford University
ACM Grace Murray Hopper Award
Turing Award
Author of The Art Of Computer Programming
Creator of TeX
3
6. Donald E. Knuth
We should forget about small
efficiencies, say about 97% of the
time: premature optimization is
the root of all evil.
Yet we should not pass up our
opportunity in that critical 3%.
5
7. Donald E. Knuth
A good programmer will not be
lulled into complacency by such
reasoning, he will be wise to
look carefully at the critical code;
but only after that code has
been identified.
6
8. Donald E. Knuth
It is often a mistake to make
a priori judgements about
what parts of a program are
really critical.
7
9. Donald E. Knuth
The universal experience of
programmers who have been
using measurement tools has been
that the intuitive guesses fail.
8
40. Built-in profilers can show
If compilation is happening while measuring
If class loading is happening while measuring
How much object allocation is happening
Which methods are consuming CPU time
39
41. External profilers can be used
Linux perf_events
Windows xperf
Java Mission Control (pluggable)
Yourkit, etc.
40
44. Verify that new code matches expectations
Check that no regression is introduced
Validate optimization ideas
Cover performance fixes with related test
Motivation:
Professional programmers should take responsibility on the performance of the code that they are developing. Performance is frequently something that is overlooked until last phases of the development process, whereas it should actually be integrated in the development process.
TODO: Add some motivation around environmental benefits.
This catching phrase is usually used without much context. Just like biblical citations it can lead to religious wars.
Let’s check the context around that phrase.
Let’s put some context around that citation. Look into what’s written before and after that phrase in the paper where it appeared.
Knuth is the guy who said that.
There’s some polemic about whether the quote is originally from Knuth or if he was citing Tony Hoare.
This article tries to ”prove” that it’s actually from Knuth:
https://shreevatsa.wordpress.com/2008/05/16/premature-optimization-is-the-root-of-all-evil/
A little bit of context:
"Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunity in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgements about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that the intuitive guesses fail.”
Page 268
Donald Knuth
Structured programming with go to statements
Computing Surveys, Vol. 6, No. 4, December 1974
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.6084&rep=rep1&type=pdf
A little bit of context:
"Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunity in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgements about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that the intuitive guesses fail.”
Page 268
Donald Knuth
Structured programming with go to statements
Computing Surveys, Vol. 6, No. 4, December 1974
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.6084&rep=rep1&type=pdf
A little bit of context:
"Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunity in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgements about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that the intuitive guesses fail.”
Page 268
Donald Knuth
Structured programming with go to statements
Computing Surveys, Vol. 6, No. 4, December 1974
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.6084&rep=rep1&type=pdf
A little bit of context:
"Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunity in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgements about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that the intuitive guesses fail.”
Page 268
Donald Knuth
Structured programming with go to statements
Computing Surveys, Vol. 6, No. 4, December 1974
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.6084&rep=rep1&type=pdf
A little bit of context:
"Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunity in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified. It is often a mistake to make a priori judgements about what parts of a program are really critical, since the universal experience of programmers who have been using measurement tools has been that the intuitive guesses fail.”
Page 268
Donald Knuth
Structured programming with go to statements
Computing Surveys, Vol. 6, No. 4, December 1974
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.103.6084&rep=rep1&type=pdf
Now you’re all convinced that you should be measuring the performance of your code. But wait, don’t just put timers on your unit-tests .
Now you’re all convinced that you should be measuring the performance of your code. But wait, don’t just put timers on your unit-tests .
Explain how code is initially interpreted;
Then, compiled at runtime;
Then, it runs in compiled mode.
Explain how code is initially interpreted;
Then, compiled at runtime;
Then, it runs in compiled mode.
Explain how code is initially interpreted;
Then, compiled at runtime;
Then, it runs in compiled mode.
Branch prediction
Loop unrolling
Dead code elimination
Autobox elimitation
Constant propagation
Null check elimination
Algebraic simplification
Devirtualisation
Range check elimitation
Etc.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
The thread scope matches well the concept of application server, because usually Java app servers have scope per thread.
This would be like processing 20 requests in parallel.
Benchmark scope would be a cache that all your requests are accessing. It should be guarded by synchronization mechanisms to make sure that it remains consistent.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
Write to a volatile happens-before every subsequent read of that volatile
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
Instead of using a single lock for the shared data, the shared data is segmented with each segment having its own lock. Uncontended lock acquisition is very cheap; it's the contented locks that cause scalability issues.
With a different lock for each partition, ConcurrentHashMap effectively reduces how often a lock is requested by the number of partitions. You can think of ConcurrentHashMap as made up of n separate hash tables.
Write to a volatile happens-before every subsequent read of that volatile
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
JMH is an open-source project that does exactly that. It’s part of the OpenJDK project and we will see how it can help you.
LMAX: Our micro-benchmarks currently take over an hour to run, though with more hardware we could run them in parallel to improve this. That's still not bad, but for comparison, our suite of ~11k acceptance tests only takes ~25mins...
Reduce noise / Isolate your benchmarks as much as possible (using cpu isolation, sched_setaffinity);
Care about a correct warmup phase / Give benchmarks enough time to run;
Don't do nanosecond per operation benchmarks in Continuous integration;
Define regression / Some variance is expected;
Define well your baseline;
Differentiate inter-version, intra-version regressions;
Make sure issues backlog is tracked and handled.