Splunk is a powerful platform for understanding your data. The preview of the Machine Learning Toolkit and Showcase App extends Splunk with a rich suite of advanced analytics and machine learning algorithms, which are exposed via an API and demonstrated in a showcase. In this session, we'll present an overview of the app architecture and API and then show you how to use Splunk to easily perform a wide variety of tasks, including outlier detection, predictive analytics, event clustering, and anomaly detection. We’ll use real data to explore these techniques and explain the intuition behind the analytics.

1. Copyright © 2015 Splunk Inc.
Machine Learning + Analy>cs in Splunk
Andrei Kastulin
Senior Sales Engineer
Splunk, Inc.

2. 2
Disclaimer
During the course of this presenta>on, we may make forward looking statements regarding future
events or the expected performance of the company. We cau>on you that such statements reflect our
current expecta>ons and es>mates based on factors currently known to us and that actual events or
results could differ materially. For important factors that may cause actual results to differ from those
contained in our forward-looking statements, please review our filings with the SEC. The forward-
looking statements made in the this presenta>on are being made as of the >me and date of its live
presenta>on. If reviewed aQer its live presenta>on, this presenta>on may not contain current or
accurate informa>on. We do not assume any obliga>on to update any forward looking statements we
may make.
In addi>on, any informa>on about our roadmap outlines our general product direc>on and is subject to
change at any >me without no>ce. It is for informa>onal purposes only and shall not, be incorporated
into any contract or other commitment. Splunk undertakes no obliga>on either to develop the features
or func>onality described or to include any such feature or func>onality in a future release.

3. Copyright © 2015 Splunk Inc.
Why do we need ML?

4. Copyright © 2016 Splunk Inc.
Historical Data Real->me Data Sta>s>cal Models
DB, HDFS, NoSQL, Splunk, etc Machine Learning
T – a few days T + a few days
Why is this so challenging using tradi2onal methods?
• DATA IS STILL IN MOTION, s>ll in a BUSINESS PROCESS.
• Enrich real->me MACHINE DATA with structured HISTORICAL DATA
• Make decisions IN REAL TIME using ALL THE DATA
Splunk
Security Opera>ons Center
Network Opera>ons Center
Business Opera>ons Center

5. Copyright © 2015 Splunk Inc.
What is ML?

6. 6
ML 101: What is it?
• Machine Learning (ML) is a process for generalizing from examples
– Examples = example or “training” data
– Generalizing = building “sta>s>cal models” to capture correla>ons
– Process = ML is never done, you must keep valida>ng & refigng models
• Simple ML workflow:
– Explore data
– FIT models based on data
– APPLY models in produc>on
– Keep valida>ng models
“All models are wrong, but some are useful.”
- George Box

7. 7
2 Types of Machine Learning
1. Supervised Learning: generalizing from labeled data

8. 8
2 Types of Machine Learning
2. Unsupervised Learning: generalizing from unlabeled data

9. 9
3 Types of Machine Learning
3. Reinforcement Learning: generalizing from rewards in >me
Leitner System Recommender systems

10. 10
How-To: Opera>onalize Machine Learning
• Reinforcement learning is key
• Reduce false posi>ves:
– Recommend events/incidents to analyst
– Record their workflow
– Build models to learn
• Reward analysts for good analysis:
– Give bonuses for successful inves>ga>ons
& detailed inves>ga>ve reports
– Carrots not s>cks
• Reward machines for good learning:
– Calibrate data, risk/rewards, analyst outcome

11. Copyright © 2015 Splunk Inc.
ML Use Cases

12. 12
IT Ops: Predic>ve Maintenance
1. Get resource usage data (CPU, latency, outage reports)
2. Explore data, and fit predic>ve models on past / real->me data
3. Apply & validate models un>l predic>ons are accurate
4. Forecast resource satura>on, demand & usage
5. Surface incidents to IT Ops, who INVESTIGATES & ACTS
Problem: Network outages and truck rolls cause big >me & money expense
Solu>on: Build predic>ve model to forecast outage scenarios, act pre-emp>vely & learn

13. 13
Security: Find Insider Threats
Problem: Security breaches cause big >me & money expense
Solu>on: Build predic>ve model to forecast threat scenarios, act pre-emp>vely & learn
1. Get security data (data transfers, authen>ca>on, incidents)
2. Explore data, and fit predic>ve models on past / real->me data
3. Apply & validate models un>l predic>ons are accurate
4. Forecast abnormal behavior, risk scores & notable events
5. Surface incidents to Security Ops, who INVESTIGATES & ACTS

14. 14
Business Analy>cs: Predict Customer Churn
Problem: Customer churn causes big >me & money expense
Solu>on: Build predic>ve model to forecast possible churn, act pre-emp>vely & learn
1. Get customer data (set-top boxes, web logs, transac>on history)
2. Explore data, and fit predic>ve models on past / real->me data
3. Apply & validate models un>l predic>ons are accurate
4. Iden>fy customers likely to churn
5. Surface incidents to Business Ops, who INVESTIGATES & ACTS

15. 15
Summary: The ML Process
Problem: <Stuff in the world> causes big >me & money expense
Solu>on: Build predic>ve model to forecast <possible incidents>, act pre-emp>vely & learn
1. Get all relevant data to problem
2. Explore data, and fit predic>ve models on past / real->me data
3. Apply & validate models un>l predic>ons are accurate
4. Forecast KPIs & metrics associated to use case
5. Surface incidents to X Ops, who INVESTIGATES & ACTS
Opera>onalize

16. Copyright © 2015 Splunk Inc.
How do we do ML with Splunk?

17. 17
Analysts Business Users
1. Get Data & Find Decision-Makers!
17
IT Users
ODBC
SDK
API
DB Connect
Look-Ups
Ad Hoc
Search
Monitor
and Alert
Reports /
Analyze
Custom
Dashboards
GPS /
Cellular
Devices Networks Hadoop
Servers Applica>ons Online
Shopping Carts
Analysts Business Users
Structured Data Sources
CRM ERP HR Billing Product Finance
Data Warehouse
Clickstreams

18. 18
2. Explore Data, Build Searches & Dashboards
• Start with the Exploratory Data Analysis phase
– “80% of data science is sourcing, cleaning, and preparing the data”
• For each data source, build “data diagnos>c” dashboard
– What’s interes>ng? Throw up some basic charts.
– What’s relevant for this use case?
– Any anomalies? Are thresholds useful?
• Mix data streams & compute aggregates
– Compute KPIs & sta>s>cs w/ stats, eventstats, etc.
– Enrich data streams with useful structured data
– stats count by X Y – where X,Y from different sources

19. 19
3. Get the ML Toolkit & Showcase App
• Get the App: hwp://>ny.cc/splunkmlapp
• Leverages Python for Scien2fic Compu2ng (PSC) add-on:
– Open-source Python data science ecosystem
– NumPy, SciPy, scitkit-learn, pandas, statsmodels
• Showcase use cases: Hard Drive Failure, Server Power consump>on,
Server Response Time, Applica>on Usage
• Standard algorithms out of the box:
– Supervised: Logis2c Regression, SVM, Linear Regression, Random Forest
– Unsupervised: KMeans, DBSCAN, Spectral Clustering
• Implement one of 300+ algorithms by edi>ng Python scripts

20. 20
4. Fit, Apply & Validate Models
• ML SPL – New grammar for doing ML in Splunk
• fit – fit models based on training data
– [training data] | fit LinearRegression costly_KPI
from feature1 feature2 feature3 into my_model
• apply – apply models on tes>ng and produc>on data
– [testing/production data] | apply my_model
• Validate Your Model (The Hard Part)
– Why hard? Because sta>s>cs is hard! Also: model error ≠ real world risk.
– Analyze residuals, mean-square error, goodness of fit, cross-validate, etc.
– Take Splunk’s Analy>cs & Data Science Educa>on course

21. 21
5. Opera>onalize Your Models
• Remember the ML Process:
1. Get data
2. Explore data & fit models
3. Apply & validate models
4. Forecast KPIs
5. Surface incidents to Ops team
• Then opera>onalize: feed back Ops analysis to data inputs, repeat
• Lots of hard work & stats, but lots of value will come out.
Opera>onalize

22. Copyright © 2015 Splunk Inc.
Show me the ML!

23. 23
What’s new in GA
• New Algorithms (Random Forest, Lasso, Kernel PCA, and more…)
• More use cases to explore
• Support added for Search Head Clustering
• Removed 50k limit on model figng
• Sampling for training/test data
• Guided ML via a ML Assistant aka Model / Query Builder
• Install on 6.4 Search Head

24. 24
Next Steps with Splunk ML
• Reach out to your Tech Team! We can help architect ML workflows.
• Lots of ML commands in Core Splunk (predict, anomalydetec>on, stats)
• ML Toolkit & Showcase – available and free, ready to use
• Splunk ITSI: Applied ML for ITOA use cases
– Manage 1000s of KPIs & alerts
– Adap>ve Thresholding & Anomaly Detec>on
• Splunk UBA: Applied ML for Security
– Unsupervised learning of Users & En>>es
– Surfaces Anomalies & Threats
• ML Customer Advisory Program:
– Connect with Product & Engineering teams - mlprogram@splunk.com

25. 25
SEPT 26-29, 2016
WALT DISNEY WORLD, ORLANDO
SWAN AND DOLPHIN RESORTS
• 5000+ IT & Business Professionals
• 3 days of technical content
• 165+ sessions
• 80+ Customer Speakers
• 35+ Apps in Splunk Apps Showcase
• 75+ Technology Partners
• 1:1 networking: Ask The Experts and Security
Experts, Birds of a Feather and Chalk Talks
• NEW hands-on labs!
• Expanded show floor, Dashboards Control
Room & Clinic, and MORE!
The 7th Annual Splunk Worldwide Users’ Conference
PLUS Splunk University
• Three days: Sept 24-26, 2016
• Get Splunk Cer>fied for FREE!
• Get CPE credits for CISSP, CAP, SSCP
• Save thousands on Splunk educa>on!

26. Copyright © 2015 Splunk Inc.
Thank you!