1. Java profiling
Do It Yourself
Alexey Ragozin
alexey.ragozin@gmail.com
JUG.MSK.RU 2016
2. JVM diagnostic interfaces
• JMX
• JVMTI – native API only
• Attach Protocol
Ad hoc instrumentation
and more
• Perf counters
• Heap dump
• Flight recorder
• Serviceability agent
3. MBeans: threading
CPU usage per thread (user / sys)
Memory allocation per thread
Block / wait times
Should be enabled
Stack traces
Invaluable
5. MBeans: memory
• Memory geometry information
• Collection count
• Last collection details
for each collector
• GC events available as notifications
since Java 7
8. JVM Attach Protocol
• List JVM processes
• Attach to JVM by PID
• Send control commands
heap dump / histogram
stack dump
• Inspect system properties and VM options
• Launch instrumentation agents
https://github.com/gridkit/jvm-attach
9. SJK: hh --dead
Dead object histogram
Similar to jmap –histo
Invoke jmap –histo two time
all heap objects
live heap object
calculates difference
Can show top N rows
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#hh-command
11. SJK: jps
JDK’s jps on steroid
Uses attach API
Lists VMs
Filtering by JVM system properties
Prints property values
Prints effective –XX options
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#jps-command
13. Perf counters
Based on shared memory
safe for target JVM
Flat data model
misc JVM counters
safepoint statistics
GC tennuring stats
you can add own counter programmatically
jcmd PID PerfCounter.print
14. Stack Trace Sampling
Capture
• Dump stack traces via local connection
• Store in highly compressed dump
10-30 bytes per trace
Analysis
• Frame frequency
• Conditional frame frequency
• Traces classification histogram
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#stcap-command
18. Working with heap dumps
Java API to traverse heap dump object graph
Available at https://github.com/aragozin/jvm-tools/tree/master/hprof-heap
Based on NetBeans profiler library
No temporary files used
Fixed generic method signatures
Improved performance
Useful for
In-place processing of large heap dumps
150 GiB is my personal record
Write domain specific heap usage reports
19. Working with heap dumps
HeapPath
Convenient way to extract value from dump
Error proof
Handles String, primitives/boxed and arrays
myfield1.myfield2.myfield3
myarrayfield[0].myfield
myarrayfield[*].myfield
myarrayfield[*][*]
myfield1.*.myfield3
[*].value(MyClass)
myhashmap?entrySet[key=description].value
20. Working with heap dumps
See also
https://github.com/vlsi/mat-calcite-plugin
Heap dump meets SQL
21. SJK Summary
Visit https://github.com/aragozin/jvm-tools
Single executable JAR
Command line interface
Exploits JMX/JVMAttachProtocol/PerfCounters
Sampling profiler included
Extensible commands
Write commands for your own application
22. Btrace: CLI profiler
BTrace
• Instrumenting
profiler
• Used via CLI or API
• Scriptable
with Java
BTrace script
@Property
Profiler prof = Profiling.newProfiler();
@OnMethod(clazz = "org.jboss.seam.Component",
method = "/(inject|disinject|outject)/")
void entryByMethod2(@ProbeClassName String className,
@ProbeMethodName String methodName, @Self Object component) {
if (component != null) {
Field nameField = field(classOf(component), "name", true);
if (nameField != null) {
String name = (String)get(nameField, component);
Profiling.recordEntry(prof, concat("org.jboss.seam.Component.",
concat(methodName, concat(":", name))));
}
}
}
@OnMethod(clazz = "org.jboss.seam.Component",
method = "/(inject|disinject|outject)/",
location = @Location(value = Kind.RETURN))
void exitByMthd2(@ProbeClassName String className,
@ProbeMethodName String methodName, @Self Object component,
@Duration long duration) {
if (component != null) {
Field nameField = field(classOf(component), "name", true);
if (nameField != null) {
String name = (String)get(nameField, component);
Profiling.recordExit(prof, concat("org.jboss.seam.Component.",
concat(methodName, concat(":", name))), duration);
}
}
}
https://github.com/jbachorik/btrace2
23. Sigar
System Information Gatherer And Reporter
https://github.com/hyperic/sigar
• Cross platform
• Common system metrics
CPU, Context switches, IO, etc
• Java bindings
Self extracting JAR: org.gridkit.lab:sigar-lib:1.6.4
24. Flight Recorder
+ Accessible via JMX
+ Targeting JVM internals
+ Low overhead
‐ Non-compact file format
‐ Biased profiling
‐ Weak support for thread sampling
26. Serviceability agent
Ultimate JVM debugging tool
• Use binding to platform debugger (windbg, gdbg)
can work with core dumps
• Use RTTI information from JVM binaries
OpenJDK requires debug symbols to be installed for SA to work
• Introspect JVM internals by inspecting process memory
• Used by jstack -F and other JVM tools accepting core dumps
• Very buggy!
• Very slow!
• Abandoned?
28. Serviceability Agent Summary
Hybrid stack trace sampling
• Slow and intrusive
• Stack parsing issues (-XX:+PreserveFramePointer should help)
• There are alternatives:
Honest Profiler - https://github.com/RichardWarburton/honest-profiler
Perf Map Agent - https://github.com/jrudolph/perf-map-agent
Heap walking with SA
• May be faster for large heaps
No need heap to disk serialization!
• Inspecting portion of heap (e.g. Eden only)
• Some code in jdi-sa.jar should be optimized to fix performance
