In this presentation, Glenn Block discusses how Splunk uses node.js in its products. Surprising, huh? Node shows up in various islands of Splunk’s architecture from the Splunk Server, to middleware components, and finally in the SDKs. The focus here is on real world usage, the specific places where we chose to use node and why, as well as the team’s experiences deploying it into production. What’s Splunk? It’s a product designed for data ingest and query for massive realtime data work loads. It is used by some of the largest organizations in the world.
6. What is Splunk?
Any Machine Data
HA Indexes
and Storage
Search and
Investigation
Proactive
Monitoring
Operational
Visibility
Real-time
Business
Insights
Commodity
Servers
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
11. Why node?
11
• We needed an easy way to convert a web page into PDF
Node has modules for rendering web pages on the server
(JSDOM)
Node has modules for PDF generation (PDFKIT)
16. Why node?
16
We needed a mobile-friendly façade for our non-mobile
friendly API
The server is heavily IO bound / async, which node is perfect
for.
It is easy and lightweight to spin up API servers and proxies in
node. Great support for Auth as well.
19. Log directly to
Splunk via TCP,
UDP, HTTP
Integrate search
results with other
applications using
custom
visualizations
Create and run
searches from
other applications
Splunk SDK
19
VisualizeSearch Manage
Add/Delete Users
Manage Inputs
Index
22. Why node?
22
Inputs by their nature are heavily IO bound and async
Inputs are streaming results back to Splunk and node has great
support in the box for streams.
It has a great ecosystem of modules
It’s JavaScript!
24. Where to go for More Info
• Tutorials, Code Samples, Getting Started, Downloads
– http://dev.splunk.com/javascript
– https://github.com/glennblock/splunk-socket.io
• Support
– https://www.splunk.com/page/submit_issue
• GitHub
– https://github.com/splunk/splunk-sdk-javascript
• Twitter
– https://twitter.com/splunkdev or follow me:
https://twitter.com/gblock
• Blog
– http://blogs.splunk.com/dev/
24
Splunk is the leading platform for machine data analytics with over 5,200 organizations using Splunk (as of 7/1/13) – from tens of GB to many tens of TBs of data PER DAY.
Splunk software is optimized for real-time, low latency and interactivity.
Splunk software reliably collects and indexes all the streaming data from IT systems and technology devices in real-time - tens of thousands of sources in unpredictable formats and types.
The value from Splunking machine data is described as Operational Intelligence. This enables organizations to:
1. Find and fix problems dramatically faster
2. Automatically monitor to identify issues, problems and attacks
3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions
4. Gain real-time insight from operational data to make better-informed business decisions
The Splunk SDKs empower developers to deliver greater operational agility throughout the enterprise by making it easy to integrate data from Splunk with other applications. Splunk provides a fully-documented and supported REST API with nearly 200 endpoints that let developers programmatically index, search and visualize data in Splunk from any application. Splunk’s SDKs, built on that API, make it easy for developers to integrate data from Splunk with other applications across the enterprise, from custom-built mobile reporting apps to off-the-shelf CRM solutions. Splunk offers SDKs for Python, Java, JavaScript, PHP, Ruby and C#.
Developers can use the Splunk SDKs to:
Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications
Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards
Build mobile applications with real-time KPI dashboards and alerts powered by Splunk
Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP
Build customer-facing dashboards in your applications powered by user-specific data in Splunk
Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk
Programmatically extract data from Splunk for long-term data warehousing
Interact with data stored in Hadoop using HUNK