This document provides an overview of using Splunk for application development. It discusses using the Splunk developer platform, APIs, and SDKs to index, search, manage and visualize data in Splunk from other applications. The agenda covers assumptions, using Splunk for development, an overview of the Splunk developer platform including the REST API and SDKs for Java, JavaScript, Python, and PHP. Code samples are provided for logging events and performing searches using the Java SDK. Support resources for developers are also listed.
4. You Are in This Session Because….
• You are an experienced Splunk user (search, dashboards,
sourcetyping, extracting fields)
• You are a developer and want to use your development skills to
customize and extend your Splunk experience
• You love REST and love developing with APIs
• You are interested in using SDKs to index, search, manage and
visualize data in Splunk
• You have http://dev.splunk.com bookmarked
4
6. Using Splunk for Application Development
Accelerate Dev & Test
– Every developer should use Splunk to find and fix bugs, trace transactions in real
time & build intelligence into your apps without defining a schema with
semantic logging
Integrate date from Splunk into other applications
– Search, manage and visualize data in other applications with the REST API and
SDKs for Java, Python, JavaScript and PHP
Build Real-time Big Data Applications
– Collection, storage, query language, visualization “out-of-the-box”
– Real-time insights: clickstream analysis, IT early-warning systems, security and
fraud protection
6
10. The Splunk Platform
Inputs, Apps, Other
Operational Intelligence Platform Content
UI SDK
Content REST API
User and Developer Interfaces Core Functions
Core Engine
Search Processing Language
Indexing
Collection
10
11. What can you do with the APIs and SDKs?
Index
– Log directly to Splunk (TCP, UDP, HTTP)
Search
– Including saved searches
– Extract data from Splunk
Visualize
– Integrate search results with third-party reporting tools, portals and other
custom applications
Manage
– Add/remove users and roles
– Create inputs
11
12. The Splunk REST API
Exposes an API method for every feature in the product
– Whatever you can do in the UI – you can do through the API
– Run searches
– Manage Splunk configurations
API is RESTful
– Endpoints are served by splunkd
– Requests are GET, POST, and DELETE HTTP methods
– Responses are Atom XML Feeds
– JSON coming in 5.0
– Versioning coming in 5.0
– Search results can be output in CSV/JSON/XML
12
13. Spring Integration Splunk Inbound Adaptor
• Blocking, Non Blocking, Saved & Realtime Searches
• Exporting
13
17. SDK Design Concepts
• Stay true to the semantics of the particular language
• E.g. Keep Python “pythonic”
• Provide implementation that feels to the developer
• E.g. Project, build, IDE (where applicable) support
• Cover REST API endpoints based on use cases of language
• E.g. Java SDK has most comprehensive coverage. JavaScript has fewer management facilities
• Initially stay true to REST API semantics and abstract based on feedback
• Namespaces
• owner: splunk username (defaults to current user)
• app: app context (defaults to default app)
• sharing: user | app | global | system
• Service Class
• Instantiate an object to connect and login
• Entry point for REST API calls
17
18. Java SDK
• Client/Server state
• Need to maintain state explicitly
• update() : to push changes to splunkd
• refresh() : to get changes from splunkd
• Getting Started - http://dev.splunk.com/view/java-sdk/SP-CAAAECN
• Open sourced under the Apache v2.0 License
• Current release status is “beta”
• Clone from Github : git clone https://github.com/splunk/splunk-sdk-java.git
• Project level support for Eclipse and IntelliJ (git plugins available)
• Pre-requisites
• Splunk installed
• JRE 6+
• Ant (test, build, generate javadocs)
• Run the unit tests and examples
• Set up a “.splunkrc” file in your user’s home directory
• Ant (build, test, generate javadocs)
18
19. JavaScript SDK
• 2 main components
• Data SDK – Manage Splunk objects, input and search data etc.
• UI SDK – Includes Splunk UI components like Charting and Timeline controls
• Use of native JavaScript objects
• Resource, Entity and Collection objects provide the necessary abstraction
• Client/Server state
• Need to maintain state explicitly
• update() : to push changes to splunkd
• fetch() : to get changes from splunkd
• Getting Started - http://dev.splunk.com/view/javascript-sdk/SP-CAAAECM
• Open sourced under the Apache v2.0 License
• Current release status is “beta”
• Clone from Github : git clone https://github.com/splunk/splunk-sdk-javascript.git
• Pre-requisites
• Splunk installed
• Node.js for server side scripting, building and running tests and examples
• Run the unit tests and examples using node.
19
20. Python SDK
• 4 main modules
• binding: Provides thin abstraction over raw HTTP.
• client: Provides an abstraction layer over REST APIs.
• results: Provides a Splunk specific streaming XML reader.
• data: Converts Splunk’s Atom feed response into Pythonic structure – directory or list
• Client/Server state
• Need to maintain state explicitly
• update() : to push changes to splunkd
• refresh() : to get changes from splunkd
• Getting Started - http://dev.splunk.com/view/python-sdk/SP-CAAAEBB
• Open sourced under the Apache v2.0 License
• Current release status is “beta”
• Clone from Github : git clone https://github.com/splunk/splunk-sdk-python.git
• Pre-requisites
• Splunk installed
• Python 2.6+
• easy_install or pip
• Run the unit tests and examples
• Set up a “.splunkrc” file in your user’s home directory
20
21. PHP SDK
• Client/Server state
• Need to maintain state explicitly
• update() : to push changes to splunkd
• fetch() : to get changes from splunkd
• Getting Started - http://dev.splunk.com/view/php-sdk/SP-CAAAEJM
• Open sourced under the Apache v2.0 License
• Current release status is “preview”
• Clone from Github : git clone https://github.com/splunk/splunk-sdk-php.git
• Pre-requisites
• Splunk installed
• PHP 5.2.11+
• Web Server that supports PHP (e.g. MAMP) – for running examples
• PHPUnit 3.6+ - for running the unit tests
• Run the unit tests and examples
• Set up a “settings.default.php” file in the examples and tests directory
21
29. Paginating Results
• “maxresultrows” in Splunk config default 50K
• Not recommended to change this
• If result set > 50K , then page through results
29
34. The Splunk Developer Community
Splunkbase
• Over 1,000 unique visitors/week to dev.spunk.com
• Over 650 followers of @splunkdev
34
35. Where to Go for More Info
Portal
– http://dev.splunk.com/
GitHub
– https://github.com/splunk/
Twitter
– https://twitter.com/splunkdev
Blog
– http://blogs.splunk.com/dev/
Support
35
We’re extending Splunk so it’s easier for your to leverage it’s capabilities using technologies you’re familiar with. We’re delivering SDKs on top of our REST API to help you integrate Splunk data with other applications. Splunk is a fully-integrated platform that delivers rapid “time-to-value” to developers. Many of our customers are building robust applications on Splunk today that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at a scale that their businesses demand.
Whatdoes this platform look like?The platform consists of 2 layer:A core engine and an interface layerOn top of the platform you can’t run a broad spectrum of content that supports use casesUse cases range from application mgmt. and IT operations, to ES and PCI compliance, to web analyticsThe core engine provides the basic services for real time data input, indexing and search as well alerting, large scale distributed processing and role based accessThe Interface layer consist of the basic UI for search, reporting and visualization– it contains developer interfaces, the REST API and SDKsThe SDKs provide a convenient access to core engine services in a variety of programing language environments. These programmatic interfaces allow you to eithe:r:extend Splunkintegrate Splunk with other applicationsbuild completely new applications from scratch that require OI or analytical services that Splunk provides
There is code in the develop branch (which we should probably push into main before .conf) that obviates the need for job.refresh()isDone() and isReady() refresh behind your back.
In order to get all events, you have to use the export endpoint. But the export endpoint has different behavior than a normal job. An export cannot be "restarted" when getting events if the network hiccups. A search job can just do another getResults() with the appropriate offset — this is because the export endpoint doesn't save the results like a search job does. But a search job has a limited number of events it will store on the server — which can be affected by status_buckets — but there is no way to guarantee the upper limit. With the default status_buckets we can get to 500K events. Itay and I experimented with hundreds of stratus_buckets but were only to get up to about 1M events, out of 13M available events.