Memory leaks are not always simple or easy to find. Heap dumps from production systems are often gigantic (4+ gigs) with millions of objects in memory. Simple spot checking with traditional tools is woefully inadequate in these situations, especially with real data. Leaks can be entire object graphs with enormous amounts of noise. This session will show you how to build custom tools using the Apache NetBeans Profiler/Heapwalker APIs. Using these APIs, you can read and analyze Java heaps programmatically to ask really hard questions. This gives you the power to analyze complex object graphs with tens of thousands of objects in seconds.

1. Exploring Java Heap
Dumps
Ryan Cuprak

2. Java Heap Review
• Java objects are stored in the heap
• All objects are globally reachable in the heap
• Heap is created when an application starts
• Size of heap is configured using –Xmx and –Xmx
• Garbage collection prunes the heap and removes objects
no longer reachable
• Stack memory - variable values are stored when their
methods are invoked
Heap Contains Everything and can be DUMPED to DISK

3. Why Analyze Heaps?
• IT reports Java EE/Spring server memory footprint has
grown to 9 gigs
• Server app logs contain OutOfMemoryExceptions
• Connections to queueing or database are exhausted
• Serialized Java objects in queue are unreasonably large
• Desktop application becomes unresponsive
• Excessive amount of garbage collection

4. Java Heap Analysis
MAT
Profiler
All you need is a profiler right?

5. Memory Leaks
Textbook memory
leaks - easy to find
and fix.
JConsole

6. Production Heap Dumps
• 9 gigs of data
• 13k classes loaded
• ~ 136 million instances
• ~6,000 GC roots

7. Production Heap Dumps
Capture the
Heap Dump
and…

8. Production Heap Dumps

9. Heap Dump Panic
Too much data!
• Impossible to comprehend
• No human way to explore the data
• Application data model is too
complicated

10. Real Memory Leaks
Bank Account:
1231209
Owner
Bob
Owner
JulieReport
January 2018
Bank Account:
1231210
Bank Account:
1231209
Report
January 2018
Owner
Bob
Challenge:
Data looks good
everywhere…

11. Real Memory Leaks
Causes:
• Faulty clone methods
• Duplicate singletons
• Accidently cached data
• Cache logic bugs
Complications
• May NOT GROW over time (leaks gets cleaned-up)
• More than one non-trivial memory leak

12. What about OQL?
• OQL
• Object Query Language – used for querying heaps
• SQL-like language
• Supports JavaScript expressions
• Supported in NetBeans and VisualVM
• Downside
• Poorly documented and hard to use
• Easy to create runaway queries

13. Heap Analysis Solution

14. NetBeans Profiler
• NetBeans is open source IDE/platform
• Modular architecture
• Clean code base
Profiler GUI
Profiler API

15. NetBeans Profiler API
• Parses hprof files
• Creates an object model representing the hprof file
• Pages data in from disk
• Simple API (master in about 10 minutes)
• Independent of NetBeans
• Can be extract and use in any IDE – Plain old Java
Talk is really about how to build a custom heap analysis tool:
• To answer specific data model questions
• With custom logic for your data model

16. Generating Heap Dumps

17. Generating Heap Dumps
• Command line parameter:
• -XX +HeapDumpOnOutOfMemoryError
• Command line:
• jmap –dump:format=b,file=dump.hprof <pid>
• jhsdb jmap --binaryheap --pid <pid>
• jcmd <pid> GC.heap_dump <file name>
• Ctrl-Break
Command Line

18. Generating Heap Dumps
Programmatic

19. Generating Heap Dumps
JMX

20. Heap Dump Warning
Dumping the heap:
• Takes time
• Consumes diskspace
• Negatively affects performance

21. Targeted Heap Dumps
• Serialize object graphs from application to a file.
• Read the serialized data into another tool and then
programmatically create a heap dump.

22. Building a Profiler

23. Building Custom Profiler
Create NetBeans
Platform App
Copy API src out of
NetBeans

24. NetBeans Platform App

25. NetBeans Platform App
Add dependency on
“Java Profiler (JFluid)”

26. Profiler Sources
Checkout source:
https://github.com/apache/incubator-netbeans.git
Profiler code:
netbeans/profiler/lib.profiler/src/netbeans/lib.profiler/heap
Copy heap directory

27. Which Approach?
• Copying sources easiest
• Most analysis apps are command line (one-offs)
- Note -
You don’t need the classpath of the application from which
the heap was generated.

28. NetBeans Profiler API

29. Opening a Heap
That’s All!

30. Heap Object Methods
getJavaClassByName(String fqn) : JavaClass
getAllClasses() : List
getBiggestObjectsByRetainedSize(int number) : List
getGCRoots(): GCRoot
getInstanceByID(long instanceId) : Instance
getJavaClassByID(long javaclassId) : JavaClass
getJavaClassesByRegExp(String regexp) : HeapSummary
getSummary() : Properties

31. System Properties

32. Heap Summary
• getTotalLiveInstances() : long
• getTime() : long
• getTotalAllocatedBytes() : long
• getTotalAllocatedInstances() : long
• getTotalLiveBytes() : long

33. Exploration Starting Points
• GCRoots
• Threads (really GCRoots)
• Class Types

34. GC Roots
• Garbage Collection Root is an object that is accessible from
outside the heap.
• Objects that aren’t accessible from a GC Root are garbage
collected
• GC root categorization:
• Class loaded by system class loader
• Thread
• Stack Local
• Monitor
• JNI Reference
• Held by JVM

35. Garbage Collection Roots
Java frame: 44
thread object: 5
JNI global: 29
sticky class: 1284

36. GCRoot Objects

37. GC Roots
JNI_GLOBAL = "JNI global";
JNI_LOCAL = "JNI local";
JAVA_FRAME = "Java frame";
NATIVE_STACK = "native stack";
STICKY_CLASS = "sticky class";
THREAD_BLOCK = "thread block";
MONITOR_USED = "monitor used";
THREAD_OBJECT = "thread object";
UNKNOWN = "unknown";
INTERNED_STRING = "interned string";
FINALIZING = "finalizing";
DEBUGGER = "debugger";
REFERENCE_CLEANUP = "reference cleanup";
VM_INTERNAL = "VM internal";
JNI_MONITOR = "JNI monitor";
root.getKind() : String

38. Finding Classes
• Can perform lookup using:
• Fully qualified class name (ex. java.lang.String)
• Class ID
• Instance ID
• IDs are unique to heap dump
• Hash codes are not available!

39. Profiler Data Model
JavaClass
Instance B
Value
Value
Instance A
Value
Value

40. Class
Java.lang.String
Java.util.List

41. Instances
From an instance:
• Who references the instance
• Who does the instance
reference
Perform instanceof to find out:
• ObjectArray
• PrimitiveArray
GCRoot can take forever…

42. Values
If you ask an instance for its
references, you get a list of Value
objects.

43. Example: Member Variables
Iterates over all Person objects and prints member variables.

44. Example: Static Variables

45. Example: References

46. String Implementation

47. String Extraction
Strings are objects – array of characters

48. LinkedList Implementation

49. LinkedList Implementation

50. LinkedList Extract

51. ArrayList

52. ArrayList

53. Thread Extraction

54. Noise: Ignore Internal Classes
Ignore internal JVM classes

55. Puzzler
Prints: Count: 231
• 230 entries have 1 are referenced by one other object
• 1 entry is “owned” by 822 other objects

56. Demo App Exploration

57. Demo App
Note: Used HashSets, Arrays[][], Lists, and Vectors

58. Demo App

59. Demo App

60. Demo App

61. String Utilization
• 5159 Strings in heap dump
• 15 associated with data model

62. String Utilization
Output

63. Data Model Leak
Add logic to fire an employee:

64. Data Model Leak
Fired Adam – shouldn’t be in the
system!

65. Data Model Leak
Found!

66. Data Model Leak
Leaking here!

67. Best Practices
• Be mindful of your heap
• Cache analysis on disk when processing large heaps
• Heap processing is I/O bound
• Not all profiler calls are the same
• Look for Javadoc: Speed: normal
• Maintain a list of processed objects
• Easy to run in circles
• Exclude JVM internal classes from analysis
• Revisit graph algorithms!

68. Summary
• Heap snapshot can be easily explored
• Excellent way to verify application logic
• Only way to identify deep data model/logic errors
• Can be used to recover data
• Generate a heap snapshot from a frozen/corrupted application and
then mine

69. Q&A
Twitter: @ctjava
Email: rcuprak@gmail.com / r5k@3ds.com
Blog: cuprak.info
Linkedin: www.linkedin.com/in/rcuprak
Slides: www.slideshare.net/rcuprak/presentations