SlideShare uma empresa Scribd logo
1 de 16
Splunk for JMX
Damien Dallimore
Developer Evangelist
Copyright©2013,SplunkInc.
Splunk for JMX
2
• ConnecttoanylocalorremoteJVM'sJMXserver,Hotspot/JRockit/IBMJ9/OpenJDK/AzulZing
• QueryanyMBeanrunningonthatserver
• ExtractanyMBeanattributes(simple,compositeortabular)
• InvokeMBeanoperations
• Writeattributesandoperationresultsoutinadefaultkey/valueformat,orpluginyourowncustomformat,forSPLUNK
indexingandsearching
• TransporteventsoverSTDOUT(default),TCP,Syslog,SplunkRESTendpointordirecttofile.
• DeclareclustersofJVM'sforlargerscaleJVMdeployments
• Runson*NixandWindows
• OutoftheboxdashboardsforcommonJVMMBeans
• FreelyavailablefromSplunkBase,allsourcecodeisonGitHub
Copyright©2013,SplunkInc.
Connectivity Options
3
• Remote JMX interface
– rmi (JSR160StandardImplementationandMX4J'sJSR160Implementation)
– iiop(JSR160StandardImplementationandMX4J'sJSR160Implementation)
• Direct Process attachment
– ConnectdirectlytoalocallyrunningJVMprocess
• MX4J HTTP connectors (requires MX4J in the target JVM also)
– soap,soap+sssl
– hessian,hessian+ssl
– burlap,burlap+ssl
Copyright©2013,SplunkInc.
Setup and Configuration
4
• The main goalof the app was to make it as simple and intuitive as possible to connect to
your JVMs and start Splunking JMX data
• Enableyour targetJVM’s remoteJMX interface, test connectivitywith JConsole
• InstallSplunk for JMX
• ExtractSplunkforJMXtarballtoSPLUNK_HOME/etc/apps
• RestartSplunk
• Atthesetupscreen,choosea scriptedinputforyourplatform(Nix/Windows)
• Setup your JMX configurationfile
• Thedefaultconfig.xmlfileispreconfiguredforcommonJVMMBeans
• BrowseyourJVM(usingJConsole)forotherMBeansthatyouwishtopollandconfigurethese
• Youcanhaveasmanyconfigfilesasyourequire,andyoumightsettheseuptofireoffatdifferent
scheduledfrequencies
Copyright©2013,SplunkInc.
Configuration Examples - Simple
5
Copyright©2013,SplunkInc.
Configuration Examples - Wildcards
6
MBean Object name format “domain:key=value,key2=value2”
Around 25KBytes per dump on Hotspot JVMs
Don’t know the domain or properties name or have many Mbeans
with the same attributes ?
* and ? wildcards are supported in the Mbean name
Copyright©2013,SplunkInc.
Configuration Examples - Clusters
7
• Define clusters of JVM’s that share the
same MBean definitions
• Note , in these examples, for brevity I am
using “dumpAllAttributes” , but in
production you’d want to pick and
choose specific MBean attributes you
are interested in, and perhaps split
definitions over multiple files run at
varying frequencies
Copyright©2013,SplunkInc.
Configuration Examples - Operations
8
• Invoke JMX operations that return a value or simply
perform some action on the target JVM
• Operation definitions can take parameters
Use Case 1 : your developers might code a JMX
operation that returns a CSV or JSON formatted
snapshot of some metrics for Splunking
Use Case 2 : dynamically trigger HPROF dumps.
The “com.sun.management:type=HotSpotDiagnostic”
Mbean exposes a “dumpHeap” operation
Copyright©2013,SplunkInc.
Configuration Examples - Connecting
9
• IP Address with credentials
• Hostname
• Static Process ID
• Process ID lookup from file
• Process ID lookup from command output
• Raw JMX Service URL
• MX4J HTTP Connector
Copyright©2013,SplunkInc.
Custom Formatters/Transports
10
• The Splunk for JMX configuration is user extensible
• You can code and configure your own Formatters and Transports
Formatters
• Takes the raw MBean polled output and formats it for Splunking
• A Java implementation of the "com.dtdsoftware.splunk.formatter.Formatter" interface
• If the optional formatter declaration is omitted, then the default formatter will be used
Transports
• Takes the formatted output and transports it to a destination
• A Java implementation of the "com.dtdsoftware.splunk.transport.Transport" interface
• If the optional transport declaration is omitted, then the default transport(STD out) will be used
Copyright©2013,SplunkInc.
Formatter Examples
11
Copyright©2013,SplunkInc.
Transport Examples
12
Copyright©2013,SplunkInc.
Deployment Architectures 1
13
• Simplest scenario
• Monolithic Splunk installation
• Splunk for JMX polling 1 or more remote/local JVMs via
the remote JMX interface
• There is support for many target JVM’s in the
configuration schema but to really scale out, you need
a more advanced Splunk architecture
Copyright©2013,SplunkInc.
Deployment Architectures 2
14
Splunk UF running locally
with target JVM
Splunk Indexer
Cluster
Splunk Search Head
Pool
Load Balancer
• Run Splunk UF locally with target
JVM.Can connect use remote JMX
interface or direct process attachment.
• Each tier scales out horizontally.
• Can overcome firewall issues that are
sometimes inherent with Java RMI
• Deploy Splunk for JMX components and
configurations with Splunk Deployment
Server, Puppet or Chef.
Demo
Copyright©2013,SplunkInc.
Contact me
16
Email : ddallimore@splunk.com
Twitter : @damiendallimore
Skype : damien.dallimore
Github : damiendallimore
Splunkbase : damiend
Slideshare : http://www.slideshare.net/damiendallimore
Blogs : http://blogs.splunk.com/dev
Web : http://dev.splunk.com

Mais conteúdo relacionado

Mais procurados

Introduction to Web Application Clustering
Introduction to Web Application ClusteringIntroduction to Web Application Clustering
Introduction to Web Application Clustering
Piyush Katariya
 
Apache Karaf - Building OSGi applications on Apache Karaf - T Frank & A Grzesik
Apache Karaf - Building OSGi applications on Apache Karaf - T Frank & A GrzesikApache Karaf - Building OSGi applications on Apache Karaf - T Frank & A Grzesik
Apache Karaf - Building OSGi applications on Apache Karaf - T Frank & A Grzesik
mfrancis
 

Mais procurados (20)

OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
OSMC 2021 | Robotmk: You don’t run IT – you deliver services!OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
OSMC 2021 | Robotmk: You don’t run IT – you deliver services!
 
Managing ejabberd Platforms with Docker - ejabberd Workshop #1
Managing ejabberd Platforms with Docker - ejabberd Workshop #1Managing ejabberd Platforms with Docker - ejabberd Workshop #1
Managing ejabberd Platforms with Docker - ejabberd Workshop #1
 
Introduction to Web Application Clustering
Introduction to Web Application ClusteringIntroduction to Web Application Clustering
Introduction to Web Application Clustering
 
Simple tweaks to get the most out of your jvm
Simple tweaks to get the most out of your jvmSimple tweaks to get the most out of your jvm
Simple tweaks to get the most out of your jvm
 
Liberty management
Liberty managementLiberty management
Liberty management
 
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
 
No Surprises Geo Replication - Pulsar Virtual Summit Europe 2021
No Surprises Geo Replication - Pulsar Virtual Summit Europe 2021No Surprises Geo Replication - Pulsar Virtual Summit Europe 2021
No Surprises Geo Replication - Pulsar Virtual Summit Europe 2021
 
Performance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverPerformance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & Webdriver
 
Batching and Java EE (jdk.io)
Batching and Java EE (jdk.io)Batching and Java EE (jdk.io)
Batching and Java EE (jdk.io)
 
Yang in OpenDaylight
Yang in OpenDaylightYang in OpenDaylight
Yang in OpenDaylight
 
Dragonflow Austin Summit Talk
Dragonflow Austin Summit Talk Dragonflow Austin Summit Talk
Dragonflow Austin Summit Talk
 
High density deployments using weblogic multitenancy
High density deployments using weblogic multitenancyHigh density deployments using weblogic multitenancy
High density deployments using weblogic multitenancy
 
High performance network programming on the jvm oscon 2012
High performance network programming on the jvm   oscon 2012 High performance network programming on the jvm   oscon 2012
High performance network programming on the jvm oscon 2012
 
KubeFuse - A File-System for Kubernetes
KubeFuse - A File-System for KubernetesKubeFuse - A File-System for Kubernetes
KubeFuse - A File-System for Kubernetes
 
Docker Swarm and Traefik 2.0
Docker Swarm and Traefik 2.0Docker Swarm and Traefik 2.0
Docker Swarm and Traefik 2.0
 
Common Sense Performance Indicators in the Cloud
Common Sense Performance Indicators in the CloudCommon Sense Performance Indicators in the Cloud
Common Sense Performance Indicators in the Cloud
 
Jakarta EE 8 on JDK17
Jakarta EE 8 on JDK17Jakarta EE 8 on JDK17
Jakarta EE 8 on JDK17
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
The best of Apache Kafka Architecture
The best of Apache Kafka ArchitectureThe best of Apache Kafka Architecture
The best of Apache Kafka Architecture
 
Apache Karaf - Building OSGi applications on Apache Karaf - T Frank & A Grzesik
Apache Karaf - Building OSGi applications on Apache Karaf - T Frank & A GrzesikApache Karaf - Building OSGi applications on Apache Karaf - T Frank & A Grzesik
Apache Karaf - Building OSGi applications on Apache Karaf - T Frank & A Grzesik
 

Semelhante a Splunk for JMX

Tornado Web Server Internals
Tornado Web Server InternalsTornado Web Server Internals
Tornado Web Server Internals
Praveen Gollakota
 
AMF Flash and .NET
AMF Flash and .NETAMF Flash and .NET
AMF Flash and .NET
Yaniv Uriel
 
WebLogic JMX for DevOps
WebLogic JMX for DevOpsWebLogic JMX for DevOps
WebLogic JMX for DevOps
Frank Munz
 
We4IT LCTY 2013 - x-pages-men - ibm domino xpages - performance in a nutshell
We4IT LCTY 2013 - x-pages-men - ibm domino xpages - performance in a nutshellWe4IT LCTY 2013 - x-pages-men - ibm domino xpages - performance in a nutshell
We4IT LCTY 2013 - x-pages-men - ibm domino xpages - performance in a nutshell
We4IT Group
 
Mom those things v1
Mom those things v1 Mom those things v1
Mom those things v1
von gosling
 

Semelhante a Splunk for JMX (20)

Linked Process
Linked ProcessLinked Process
Linked Process
 
Five cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark fasterFive cool ways the JVM can run Apache Spark faster
Five cool ways the JVM can run Apache Spark faster
 
A Java Implementer's Guide to Boosting Apache Spark Performance by Tim Ellison.
A Java Implementer's Guide to Boosting Apache Spark Performance by Tim Ellison.A Java Implementer's Guide to Boosting Apache Spark Performance by Tim Ellison.
A Java Implementer's Guide to Boosting Apache Spark Performance by Tim Ellison.
 
Tornado Web Server Internals
Tornado Web Server InternalsTornado Web Server Internals
Tornado Web Server Internals
 
FOSDEM 2017 - Open J9 The Next Free Java VM
FOSDEM 2017 - Open J9 The Next Free Java VMFOSDEM 2017 - Open J9 The Next Free Java VM
FOSDEM 2017 - Open J9 The Next Free Java VM
 
Getting Started with JDK Mission Control
Getting Started with JDK Mission ControlGetting Started with JDK Mission Control
Getting Started with JDK Mission Control
 
You Can't Correlate what you don't have - ArcSight Protect 2011
You Can't Correlate what you don't have - ArcSight Protect 2011You Can't Correlate what you don't have - ArcSight Protect 2011
You Can't Correlate what you don't have - ArcSight Protect 2011
 
jcmd #javacasual
jcmd #javacasualjcmd #javacasual
jcmd #javacasual
 
Like loggly using open source
Like loggly using open sourceLike loggly using open source
Like loggly using open source
 
Effective out-of-container Integration Testing
Effective out-of-container Integration TestingEffective out-of-container Integration Testing
Effective out-of-container Integration Testing
 
AMF Flash and .NET
AMF Flash and .NETAMF Flash and .NET
AMF Flash and .NET
 
WebLogic JMX for DevOps
WebLogic JMX for DevOpsWebLogic JMX for DevOps
WebLogic JMX for DevOps
 
Ingesting hdfs intosolrusingsparktrimmed
Ingesting hdfs intosolrusingsparktrimmedIngesting hdfs intosolrusingsparktrimmed
Ingesting hdfs intosolrusingsparktrimmed
 
We4IT LCTY 2013 - x-pages-men - ibm domino xpages - performance in a nutshell
We4IT LCTY 2013 - x-pages-men - ibm domino xpages - performance in a nutshellWe4IT LCTY 2013 - x-pages-men - ibm domino xpages - performance in a nutshell
We4IT LCTY 2013 - x-pages-men - ibm domino xpages - performance in a nutshell
 
Mom those things v1
Mom those things v1 Mom those things v1
Mom those things v1
 
CCNP Data Center Centralized Management Automation
CCNP Data Center Centralized Management AutomationCCNP Data Center Centralized Management Automation
CCNP Data Center Centralized Management Automation
 
Oracle WebLogic Diagnostics & Perfomance tuning
Oracle WebLogic Diagnostics & Perfomance tuningOracle WebLogic Diagnostics & Perfomance tuning
Oracle WebLogic Diagnostics & Perfomance tuning
 
Being HAPI! Reverse Proxying on Purpose
Being HAPI! Reverse Proxying on PurposeBeing HAPI! Reverse Proxying on Purpose
Being HAPI! Reverse Proxying on Purpose
 
OpenSouthCode 2018 - Integrating your applications easily with Apache Camel
OpenSouthCode 2018 - Integrating your applications easily with Apache CamelOpenSouthCode 2018 - Integrating your applications easily with Apache Camel
OpenSouthCode 2018 - Integrating your applications easily with Apache Camel
 
BSides LV 2016 - Beyond the tip of the iceberg - fuzzing binary protocols for...
BSides LV 2016 - Beyond the tip of the iceberg - fuzzing binary protocols for...BSides LV 2016 - Beyond the tip of the iceberg - fuzzing binary protocols for...
BSides LV 2016 - Beyond the tip of the iceberg - fuzzing binary protocols for...
 

Mais de Damien Dallimore

QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT Rodeo
Damien Dallimore
 
Splunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gxSplunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gx
Damien Dallimore
 

Mais de Damien Dallimore (9)

QCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT RodeoQCon London 2015 - Wrangling Data at the IOT Rodeo
QCon London 2015 - Wrangling Data at the IOT Rodeo
 
SpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk PresentationSpringOne2GX 2014 Splunk Presentation
SpringOne2GX 2014 Splunk Presentation
 
A Brief History Of Data
A Brief History Of DataA Brief History Of Data
A Brief History Of Data
 
Integrating Splunk into your Spring Applications
Integrating Splunk into your Spring ApplicationsIntegrating Splunk into your Spring Applications
Integrating Splunk into your Spring Applications
 
Spring Integration Splunk
Spring Integration SplunkSpring Integration Splunk
Spring Integration Splunk
 
Splunk Developer Platform
Splunk Developer PlatformSplunk Developer Platform
Splunk Developer Platform
 
Splunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gxSplunk as a_big_data_platform_for_developers_spring_one2gx
Splunk as a_big_data_platform_for_developers_spring_one2gx
 
Using the Splunk Java SDK
Using the Splunk Java SDKUsing the Splunk Java SDK
Using the Splunk Java SDK
 
Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)Splunking the JVM (Java Virtual Machine)
Splunking the JVM (Java Virtual Machine)
 

Último

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Último (20)

MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot ModelMcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)
 

Splunk for JMX

  • 1. Splunk for JMX Damien Dallimore Developer Evangelist
  • 2. Copyright©2013,SplunkInc. Splunk for JMX 2 • ConnecttoanylocalorremoteJVM'sJMXserver,Hotspot/JRockit/IBMJ9/OpenJDK/AzulZing • QueryanyMBeanrunningonthatserver • ExtractanyMBeanattributes(simple,compositeortabular) • InvokeMBeanoperations • Writeattributesandoperationresultsoutinadefaultkey/valueformat,orpluginyourowncustomformat,forSPLUNK indexingandsearching • TransporteventsoverSTDOUT(default),TCP,Syslog,SplunkRESTendpointordirecttofile. • DeclareclustersofJVM'sforlargerscaleJVMdeployments • Runson*NixandWindows • OutoftheboxdashboardsforcommonJVMMBeans • FreelyavailablefromSplunkBase,allsourcecodeisonGitHub
  • 3. Copyright©2013,SplunkInc. Connectivity Options 3 • Remote JMX interface – rmi (JSR160StandardImplementationandMX4J'sJSR160Implementation) – iiop(JSR160StandardImplementationandMX4J'sJSR160Implementation) • Direct Process attachment – ConnectdirectlytoalocallyrunningJVMprocess • MX4J HTTP connectors (requires MX4J in the target JVM also) – soap,soap+sssl – hessian,hessian+ssl – burlap,burlap+ssl
  • 4. Copyright©2013,SplunkInc. Setup and Configuration 4 • The main goalof the app was to make it as simple and intuitive as possible to connect to your JVMs and start Splunking JMX data • Enableyour targetJVM’s remoteJMX interface, test connectivitywith JConsole • InstallSplunk for JMX • ExtractSplunkforJMXtarballtoSPLUNK_HOME/etc/apps • RestartSplunk • Atthesetupscreen,choosea scriptedinputforyourplatform(Nix/Windows) • Setup your JMX configurationfile • Thedefaultconfig.xmlfileispreconfiguredforcommonJVMMBeans • BrowseyourJVM(usingJConsole)forotherMBeansthatyouwishtopollandconfigurethese • Youcanhaveasmanyconfigfilesasyourequire,andyoumightsettheseuptofireoffatdifferent scheduledfrequencies
  • 6. Copyright©2013,SplunkInc. Configuration Examples - Wildcards 6 MBean Object name format “domain:key=value,key2=value2” Around 25KBytes per dump on Hotspot JVMs Don’t know the domain or properties name or have many Mbeans with the same attributes ? * and ? wildcards are supported in the Mbean name
  • 7. Copyright©2013,SplunkInc. Configuration Examples - Clusters 7 • Define clusters of JVM’s that share the same MBean definitions • Note , in these examples, for brevity I am using “dumpAllAttributes” , but in production you’d want to pick and choose specific MBean attributes you are interested in, and perhaps split definitions over multiple files run at varying frequencies
  • 8. Copyright©2013,SplunkInc. Configuration Examples - Operations 8 • Invoke JMX operations that return a value or simply perform some action on the target JVM • Operation definitions can take parameters Use Case 1 : your developers might code a JMX operation that returns a CSV or JSON formatted snapshot of some metrics for Splunking Use Case 2 : dynamically trigger HPROF dumps. The “com.sun.management:type=HotSpotDiagnostic” Mbean exposes a “dumpHeap” operation
  • 9. Copyright©2013,SplunkInc. Configuration Examples - Connecting 9 • IP Address with credentials • Hostname • Static Process ID • Process ID lookup from file • Process ID lookup from command output • Raw JMX Service URL • MX4J HTTP Connector
  • 10. Copyright©2013,SplunkInc. Custom Formatters/Transports 10 • The Splunk for JMX configuration is user extensible • You can code and configure your own Formatters and Transports Formatters • Takes the raw MBean polled output and formats it for Splunking • A Java implementation of the "com.dtdsoftware.splunk.formatter.Formatter" interface • If the optional formatter declaration is omitted, then the default formatter will be used Transports • Takes the formatted output and transports it to a destination • A Java implementation of the "com.dtdsoftware.splunk.transport.Transport" interface • If the optional transport declaration is omitted, then the default transport(STD out) will be used
  • 13. Copyright©2013,SplunkInc. Deployment Architectures 1 13 • Simplest scenario • Monolithic Splunk installation • Splunk for JMX polling 1 or more remote/local JVMs via the remote JMX interface • There is support for many target JVM’s in the configuration schema but to really scale out, you need a more advanced Splunk architecture
  • 14. Copyright©2013,SplunkInc. Deployment Architectures 2 14 Splunk UF running locally with target JVM Splunk Indexer Cluster Splunk Search Head Pool Load Balancer • Run Splunk UF locally with target JVM.Can connect use remote JMX interface or direct process attachment. • Each tier scales out horizontally. • Can overcome firewall issues that are sometimes inherent with Java RMI • Deploy Splunk for JMX components and configurations with Splunk Deployment Server, Puppet or Chef.
  • 15. Demo
  • 16. Copyright©2013,SplunkInc. Contact me 16 Email : ddallimore@splunk.com Twitter : @damiendallimore Skype : damien.dallimore Github : damiendallimore Splunkbase : damiend Slideshare : http://www.slideshare.net/damiendallimore Blogs : http://blogs.splunk.com/dev Web : http://dev.splunk.com