2. The Developer Swiss Knife: Troubleshooting
Tools for Java EE Applications

3. Assumptions
•This presentation aims experienced developers
with a little (but good) knowledge about how the
JVM works
•The presentation focus only on the tools, not on
the troubleshooting methods
•Some of the tools are only for Linux. Windows
users, get out! =P
•All tools presented in this presentation are Open
Source projects and some of them are
developed by JBoss

4. The Application stack
Our primary focus

5. The Tools
•JDK CLI Tools (jstack, jmap, jps, etc.)
•JDK GUI Tools (JVisualVM, JConsole)
•Eclipse Memory Analyzer Tool (MAT)
•Thermostat
•Garbagecat
•Samurai
•AS7 JDR (JBoss Disaster Report) tool
•Tattletale
•Byteman*

6. JDK has its own troubleshooting tools
● jps
● jmap
● jinfo
● jhat
● jstack

7. Some of them are Graphical
● JConsole
● JVisualVM

8. What should I do when my application has a
memory problem?

9. Generating a Heap Dump
● -XX:-HeapDumpOnOutOfMemoryError
● jmap
● JVisualVM

10. Ok, I have a Heap Dump. Now can I analyze it?

11. Meet Eclipse MAT

12. What should I do if my application have a CPU
problem? e.g. High CPU consumption

13. Generating a Thread Dump
● jstack
● kill -3
● AS7 CLI command:
– /core-service=platform-
mbean/type=threading:dump-all-threads(locked-
monitors=true,locked-synchronizers=true)

14. Now with a Thread Dump, what should I do with
it?

15. Samurai – A Thread Dump viewer

16. High CPU consumption can be high GC activity

17. Enabling GC Logging
● Add to java command:
-XX:+PrintGC -Xloggc:gc.log -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps

18. GC Log content
3.122: [GC 132096K->24592K(504320K), 0.0321910 secs]
4.558: [GC 156688K->26752K(504320K), 0.0340520 secs]
6.609: [GC 158848K->50117K(504320K), 0.0628430 secs]
7.077: [GC 80334K->54921K(504320K), 0.0416730 secs]
7.119: [Full GC 54921K->46716K(504320K), 0.2522170 secs]
8.510: [GC 178812K->60924K(504320K), 0.0190690 secs]
9.683: [GC 193020K->69447K(461696K), 0.0308020 secs]

19. Garbagecat
java -jar garbagecat-1.0.0.jar --help
usage: garbagecat [OPTION]... [FILE]
-h,--help help
-o,--options <arg> JVM options used during JVM run
-p,--preprocess preprocessing flag
-s,--startdatetime <arg> JVM start datetime (yyyy-MM-dd HH:mm:ss,SSS)
for converting GC logging timestamps to datetime
-t,--threshold <arg> threshold (0-100) for throughput bottleneck
reporting

20. Garbagecat report
========================================
Throughput less than 90% for PARALLEL_SCAVENGE
========================================
18020.492: [GC [PSYoungGen: 161888K->1376K(166016K)] 949359K->789159K(1214592K), 0.0294550 secs]
...
========================================
SUMMARY:
========================================
# GC Events: 46044
GC Event Types: PARALLEL_SCAVENGE, PARALLEL_SERIAL_OLD
Max Heap Space: 1566912K
Max Heap Occupancy: 1562164K
Max Perm Space: 77864K
Max Perm Occupancy: 46645K
Throughput: 99%
Max Pause: 4098 ms
Total Pause: 1703657 ms
First Timestamp: 4616 ms
Last Timestamp: 232465068 ms
========================================
0 UNIDENTIFIED LOG LINE(S):

21. Applications also might have classloading
problems

22. Tattletale

23. JBoss AS7 has JDR

24. Wait a minute! I tried to run jvisualvm in OpenJDK
and it says “command not found”

25. Thermostat – The next heat

26. Please Remember
● Don't develop applications without make use
of testing approaches (unit, integration,
functional, ...)
● Although all testing efforts, there is no 100%
bug free applications
● For all unpredictable and complex problems,
this presentation will help you troubleshoot, if
not all, most of the problem

27. The links
● http://icedtea.classpath.org/wiki/Thermostat
● http://eclipse.org/mat/
● http://yusuke.homeip.net/samurai/en/index.html
● https://code.google.com/a/eclipselabs.org/p/garbagecat/
● http://docs.oracle.com/javase/7/docs/index.html
● http://www.jboss.org/tattletale
● http://jboss.org/byteman
Twitter: @rimolive
http://aprendendo-cloud-computing.blogspot.com

28. Thank you!

29. What about Byteman?

30. Race Condition Tests with Byteman
Flávia Rainone
Don't miss it!