Presentation section from Splunk Live content

1. Splunking the JVM
Damien Dallimore
Developer Evangelist

2. Copyright©2013,SplunkInc.
What is this JVM thing ?
2
• Circa 1991, Dr. James Gosling at Sun started developing a technology
for next generation smart devices/appliances
• “Green” became “Oak” which became “Java”
• Java 1.0 first appeared in January 1996.
• The JVM is a virtual machine that runs programs that are compiled
into Java bytecode
• Available for many hardware and software platforms
• 17 years later , the JVM has evolved from a consumer device
technology,to a browser oriented technology with the explosion of
the web , to now becoming deeply rooted in the enterprisesoftware
landscape on the server side and in the cloud

3. Copyright©2013,SplunkInc.
17 years later
3
• Oracle took ownership of Java from Sun in
January 2010
• The Java Community Process(JCP) is the
forum where members develop
specifications for Java technology
• Java Specification Requests(JSR) get
submitted for new features, are reviewed and
then voted on by the JCP Executive
committee.
• Editions
• Embedded Java, Java ME , Java SE , Java EE
• Current Version is Java 7 (Dolphin)
• Java 8 scheduled for 2013
Application Servers Enterprise Service Buses Databases
NoSQL Distributed Big Data Web Servers
Directory Servers Search Engines Build Systems
Gaming Platforms Trading Systems Reservation Systems
Core Banking Messaging Infrastructure Proprietary Systems

4. Copyright©2013,SplunkInc.
JVM Variants
4
• Oracle Hotspot (formerly SUN)
– theprimaryreferenceJVMimplementation
• Oracle JRockit (formerly BEA)
– freesinceMay2011
– codebasecurrentlybeingmergedwithHotspot,ETA~JDK 8
• Open JDK
– SUN opensourcedHotspotand the Java classlibraryin 2006
– SlightdifferenceswithOracleJava still
– OpenJDKis the official JavaSE7 ReferenceImplementation
• J9
– IBM’sJVMforAIX,Linux,MVS, OS/400, PocketPC, z/OS
• Azul Systems Zing
– basedonHotSpot
– supportsmemoryheapsup to 512 GB withoutGCpausesand is ableto growand shrinkthe heap
basedonload

5. Copyright©2013,SplunkInc.
The JVM has a healthy future
5
• Hotspot/JRockitcodemergecreating abestofbreedJVM,OracletocontributethistoOpenJDK
• OpenJDKisthriving,OraclearecontributingandbeinggoodstewardsofJava(despiteinitialskepticism)
• Proliferationof alternativeJVMlanguagesthatcanallco-habitateintheJVMandnewfeaturesinJava8tofurtherenhance
thismultilanguageplatform
– Scala
– Groovy
– Clojure
• TheJVMisevolvingorganicallywiththeshiftingtidesofEnterprisesoftware,itisn’taboutthe“J”anymore.
• FromtheclusteredApplicationServerdominationofthe00’swenowseeanexplosionofBigDataproductsrunningin
massivelydistributedenvironmentsoncommodityhardwareorinthecloud
– ApacheHadoopfamily(MapReduce,Hive,Hbase,Cassandra,HDFS)

6. Copyright©2013,SplunkInc.
What is running in JVMs ?
6

7. Copyright©2013,SplunkInc.
JVM “Fanboi”
7
Dr. GoslingFanboi
Speaking of Java as a language as opposed
to the JVM platform, James Gosling, the
Father of Java, said "Most people talk
about Java the language, and this may
sound odd coming from me, but I could
hardly care less."
He went on to explain, "What I really care
about is the Java Virtual Machine as a
concept, because that is the thing that ties
it all together."

8. Copyright©2013,SplunkInc.
JVM Machine Data
8
• The JVM footprint cross cuts the data centre and represents a massive source of valuable machine data
• Large scale Application/Web Server clusters
• Hadoop & Cassandra Node topologies in the 10’000s !!!
Custom Developed
Code
WAR file
Application Code
Tomcat
JVM
Hotspot
Operating System
Linux
JMX, SNMP, HPROF,GC Logs, Custom Agents(AppDynamics/SplunkJavaAgent)
JMX, Application Logs
JMX, Developer Logs, Splunk Java SDK, SplunkJavaLogging
JVM process OS resource metrics
CORRELATE

9. Copyright©2013,SplunkInc.
Application & Developer Logs
9
• Application logs
• default logs that are part of the product
• Developer logs
• any custom code created and deployed
to the application that has it’s own
logging
• Written to local disk or a mounted network
volume
• Monitor with a Splunk UF
Splunk Indexer
Splunk Universal Forwarder
Monitor Log Files/ Directorys
Developed Code
Application
JVM
OS

10. Copyright©2013,SplunkInc.
Splunk Java SDK / SplunkJavaLogging
10
Splunk Indexer
Developed Code
Application
JVM
OS
HTTP$REST$/$TCP$/$UDP • Alternative to writing to log file or
needing to deploy a Splunk Universal
Forwarder
• Use the Splunk Java SDK to input events
directly to Splunk via HTTP Rest.
• Use SplunkJavaLogging to input events
directly to Splunk using custom logging
appenders.

11. Copyright©2013,SplunkInc.
JVM Process OS Metrics
11
• By JVM Process ID : Process State, Memory, CPU,
Disk Usage, Disk I/O, Network I/O, File
Descriptor Usage.
• Some OS metrics also exposed via JMX
• Splunk for Unix and Linux
• Splunk for Windows
• Correlate this OS data across your JVM and
Application events ie: your JVM may have hung
because of CPU starvation caused by some other
process thrashing
Splunk Indexer
Splunk for Unix or Linux
Monitor Log Files &
Directorys
Developed Code
Application
JVM
OS
Poll output from
commands

12. Copyright©2013,SplunkInc.
Garbage Collection logs
12
Splunk Indexer
Splunk Universal Forwarder
Monitor GC Log Files
Developed Code
Application
JVM
OS
• Extended Hotspot JVM options
-verbose:gc
-Xloggc:/home/damien/jvm_logs/gc.log
-XX:+PrintGC
-XX:+PrintGCTimeStamps
-XX:+PrintGCDetails
• The log is written to at Garbage Collection time
• Useful for tracing full GC cycles
• Need to perform field extractions in Splunk
• Many GC metrics also available via JMX
54.736: [Full GC 54.737:
[Tenured: 172798K->18092K(174784K), 2.3792658 secs] 257598K->18092K(259584K),
[Perm : 20476K->20476K(20480K)], 2.4715398 secs] [Times: user=0.56 sys=0.05, real=0.07 secs]

13. Copyright©2013,SplunkInc.
Custom Instrumentation Agents (Advanced)
13
Splunk Indexer
Splunk Universal
Forwarder
Monitor Agent Log Files
Developed Code
Application
JVM
OS
REST/TCP/UDP
• JVM BCI (byte code instrumentation)
• Write custom agents that get injected into
the running JVM
• Dynamically inspect the state of
applications running in the JVM
• Profiling, debugging, monitoring,
thread/memory analysis
• As you write the agent code , the data
output can be file based or over the
network
• Check out my SplunkJavaAgent on github
• Also AppDynamics have some pretty cool
kung fu in this area, we integrate !

14. Copyright©2013,SplunkInc.
HPROF Profiling Dumps
14
Splunk Indexer
Splunk Universal
Forwarder
Binary HPROF dump file
Developed Code
Application
JVM
OS
Monitor and decode into
textual key=value pairs
• Binary JVM dumps that allow for deeper JVM resource
inspection
• Typical use case is diagnosing memory issues after JVM
crashes with java.lang.OutOfMemoryError
• Binary file is usually batch loaded into a third party
memory analysis tool like Eclipse MAT
• SplunkJavaAgent can dynamically dump and decode
hprof output and send to Splunk
• Awesome source of information for dev/test
Warning : heap dumping is an expensive operation as a full GC gets performed

15. Copyright©2013,SplunkInc.
SNMP
15
• The JVM SNMP Agent provides a single MIB that exposes the
JVM’s Management and Monitoring API
http://docs.oracle.com/javase/1.5.0/docs/guide/management/JVM-MANAGEMENT-MIB.mib
• Setup the JVM (just the basic settings shown)
Open a UDP Port : -Dcom.sun.management.snmp.port=9004
Configure the ACL : $JAVA_HOME/jre/lib/management/snmp.acl
• Traps can be caught locally to file and monitored
• Splunk SNMP Modular Input can poll the JVM SNMP Objects
(coming soon to a theatre near you)
Splunk Indexer
Splunk Universal
Forwarder
Developed Code
Application
JVM
OS
SNMP%Objects%
Polled
JVM MIB
snmptrapd UDP:162
SNMP%Traps%
wri6en%to%file

16. Copyright©2013,SplunkInc.
JMX (Java Management Extensions)
16
Splunk Indexer
Developed Code
Application
JVM
OS
Splunk Universal
Forwarder
JMX
• Manage and Monitor the JVM and Application via
exposed MBeans
• JVM MBeans (java.lang domain)
• Vendor MBeans (most vendors ship their products with
extensive MBean coverage)
• Custom Coded MBeans (whatever your devs wish to
code)
• MBeans expose attributes, operations and
notifications to give you a powerfully dynamic
insight into the runtime state of the JVM and your
application.
• Add Splunk to the mix for historical and realtime
operational visibility, pro-active issue detection
etc..
• Splunk for JMX app on SplunkBase

17. Copyright©2013,SplunkInc.
JMX vs SNMP
17
JMX
• Open and easily extensible
• Developers can simply create new MBeans
• Vendor products(JBoss, Cassandra, Hadoop etc..) ship with thorough MBean coverage, not MIBs
SNMP
• The built-in SNMP agent of the JVM is not extensible.
• You will not be able to use it in order to expose your own custom MIB
• If you do want to expose your own MIB, you’d have to create a custom agent

18. Copyright©2013,SplunkInc.
Putting it all together, JVM Splunking Nirvana
18
Splunk Indexer Cluster
Developed Code
Application
JVM
OS
JMX
HPROF
OS*Metrics/Logs
Splunk Forwarder
Logs
JMX
REST/TCP/UDP
Auto Load Balanced
JMX
Logs
Distributed Search

19. Copyright©2013,SplunkInc.
Contact me
19
Email : ddallimore@splunk.com
Twitter : @damiendallimore
Skype : damien.dallimore
Github : damiendallimore
Splunkbase : damiend
Slideshare : http://www.slideshare.net/damiendallimore
Blogs : http://blogs.splunk.com/dev
Web : http://dev.splunk.com