Slide deck from http://jugekb.ru/ event

1. Java black box profiling
Alexey Ragozin
alexey.ragozin@gmail.com
jugekb.ru 2016

2. The Problem
• There are application
• It doesn’t meat its SLA
• Users/client/stakeholders are unhappy
You are one to fix it!

3. What to do?
• Do not panic!
• Elaborate acceptance criteria
• Make sure you understand KPI and SLA
• Write out all moving parts in system
• Is problem in Java application?
 Yes - You are ready to start profiling

4. Moving parts
Server
Browser
Application
Database
Caching
JavaScript
HTTP
Network Network
SQL performance
CPU Memory / Swapping
Disk IO Virtualization

5. Understanding targets
• Business transaction ≠ Page
• Business transaction ≠ SQL transaction
• Page ≠ HTTP request
• HTTP request ≠ SQL transaction
Make sure you know how your KPI
are translating into milliseconds

6. Just be before you start profiling
 Take a deep breath
 Are you sure your problem is in Java part?
 Are you sure you understand target state?
Is it physically possible?
These points are trivial so it is very easy to
overlook them under stress.
Do not make that mistake!

7. Few more rule before start
Every system is a black box
 Do you think that know what is inside?
 You don’t!
There 3 kinds of data produced by profiling
 Lies
 Darn lies
 Statistics

8. Profiling your
Java application

9. Do not rush!
Do not jump to profiler just yet!
Profilers employs smart techniques
to pin point specific set of bottlenecks.
Profiler’s job is to help to find you some
problem in your code,
either real or just possible!

10. Narrow problem area
What kind of problem do you have?
• 100% CPU utilization?
• 100% of all cores or just single thread 100%
• Memory problems?
• Frequent young GCs? Full GCs?
• Contention?
Identify area and employ right tool
Even if tool covers all this areas you should
understand what are you searching for?

11. Process under profiling
Stationary process
• Uniform load reasonable simulating real one
• Goal is optimizing average performance indicators
Transition process
• Goal is optimize prolonged process
• Load profile is changing over time systematically
Profiling single operation
• Goal is to optimize particular operation

12. Profiling in JVM
• JVMTI – used by almost all 3rd party profilers
• Stack trace sampling
• Byte code instrumentation
• JMX: Thread Mbeans, etc
• JVM perf counters
• GC logs
• Heap dumps / Heap histogram
• Flight recorder / mission control

13. Stack trace sampling
• Work best with stationary process
• Uniformly impacts system under test
• Do not require “calibrating”
• Result from multiple runs could be combined
• You deal with probabilities, not hard numbers
• Measure wall clock, not CPU time

14. Byte code instrumentation
• Useful for all types of experiments
• Produce some absolute numbers
 Number of calls is accurate
 Time is predictably skewed
• Significantlyskewperformanceofsystemundertest
• Require careful targeting
• Could measure CPU time

15. Flight recorder
• Exploit JVM internals
• Provide good context (stack traces)
• Suitable long run profiling
• Completely ignore certain aspects
• Prone to finding false problems

16. Threading MBean
Standard JVM Threading MBean
 CPU usage per thread (user/sys)
 Memory allocation per thread
 Blocking / waiting statistics

17. Start broad
Performance
• Per thread CPU usage / allocation rate
• Stack sampling
Memory
• Heap histogram
• “Garbage” histogram

18. Missing tooling
SwissJavaKnife(SJK)
https://github.com/aragozin/jvm-tools
• Thread monitor
• Stack Trace Recorder/Analyzer
• Dead object histogram
and counting …

19. Tracking down CPU hogs
1. You sampling to identify suspects
• Plain frame histogram + meditation
• Call tree + deep meditation
• Hierarchical classification
2. Instrument methods to get hard prove
• Relative number of calls between methods
3. Iterate to pin point root cause

20. Flame Graphs

21. Tracking down CPU hogs
Command
sjk ssa -f tracedump.std --categorize -tf **.CoyoteAdapter.service -nc
JDBC=**.jdbc
Hibernate=org.hibernate
"Facelets compile=com.sun.faces.facelets.compiler.Compiler.compile"
"Seam bijection=org.jboss.seam.**.aroundInvoke/!**.proceed"
JSF.execute=com.sun.faces.lifecycle.LifecycleImpl.execute
JSF.render=com.sun.faces.lifecycle.LifecycleImpl.render
Other=**
Report
Total samples 2732050 100.00%
JDBC 405439 14.84%
Hibernate 802932 29.39%
Facelets compile 395784 14.49%
Seam bijection 385491 14.11%
JSF.execute 290355 10.63%
JSF.render 297868 10.90%
Other 154181 5.64% 0.00%
20.00%
40.00%
60.00%
80.00%
100.00%
Time
Other
JSF.render
JSF.execute
Seam bijection
Facelets compile
Hibernate
JDBC
Excel

22. Tracking down CPU hogs
Stack frame frequency histogram
Command
sjk ssa -f tracedump --histo -tf **!**.jdbc -tt ogr.hibernate
Report
Trc (%)FrmNTerm (%)Frame
69950687%699506 0 0%org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1204)
68937085%689370 10 0%org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
67652484%676524 0 0%org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:58)
67513684%675136 0 0%org.hibernate.internal.SessionImpl.list(SessionImpl.java:1261)
57383671%573836 4 0%org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:264)
55096868%550968 1 0%org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:99)
53389266%533892 132 0%org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:227)
38151447%381514 882 0%org.hibernate.event.internal.AbstractVisitor.processEntityPropertyValues(AbstractVisitor.java:76)
27101833%271018 0 0%org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:161)
http://blog.ragozin.info/2016/01/flame-graphs-vs-cold-numbers.html

23. Btrace: CLI profiler
BTrace
• Instrumenting
profiler
• Used via CLI or API
• Scriptable
with Java
BTrace script
@Property
Profiler prof = Profiling.newProfiler();
@OnMethod(clazz = "org.jboss.seam.Component",
method = "/(inject|disinject|outject)/")
void entryByMethod2(@ProbeClassName String className,
@ProbeMethodName String methodName, @Self Object component) {
if (component != null) {
Field nameField = field(classOf(component), "name", true);
if (nameField != null) {
String name = (String)get(nameField, component);
Profiling.recordEntry(prof, concat("org.jboss.seam.Component.",
concat(methodName, concat(":", name))));
}
}
}
@OnMethod(clazz = "org.jboss.seam.Component",
method = "/(inject|disinject|outject)/",
location = @Location(value = Kind.RETURN))
void exitByMthd2(@ProbeClassName String className,
@ProbeMethodName String methodName, @Self Object component,
@Duration long duration) {
if (component != null) {
Field nameField = field(classOf(component), "name", true);
if (nameField != null) {
String name = (String)get(nameField, component);
Profiling.recordExit(prof, concat("org.jboss.seam.Component.",
concat(methodName, concat(":", name))), duration);
}
}
}
https://github.com/jbachorik/btrace2

24. Tracking memory littering
1. Indentify thread producing garbage
• SJK: ttop command
2. Classify garbage
• Class histogram of dead object in heap
• SJK: hh --dead / hh --dead-young
3. Investigate suspect classes
• Heap dump
• Debugging / Instrumentation / Flight recorder

25. Tracking contention
• Stack trace sampling
 Good for “bad” contention cases
• Flight recorder
 Excellent at contention analysis
• BTrace
 Analyze lock access pattern using your
codebase knowledge, useful if you need gather
more application context

26. Conclusion
Understand desired target state
Progress slow and steady
• Do not get side tracked from real problem
Start from broad techniques
• Stack trace sampling
• Garbage histogram
Dig deeper with more advanced tools
• Targeted instrumentation
• Flight recorder metrics

27. KEEP
CALM
AND CRACK YOUR
JAVA PROBLEMS
Alexey Ragozin
alexey.ragozin@gmail.com
http://blog.ragozin.info
- my articles
http://github.com/aragozin
http://github.com/gridkit
- my open source code