Hortonworks sqrrl webinar v5.pptx

Cyber Pattern Discovery using Linked Data Analysis
November 12, 2014
Page 1 © Hortonworks Inc. 2014
Webinar with Hortonworks and Sqrrl

Webinar with Hortonworks and Sqrrl
Cyber Pattern Discovery using Linked Data Analysis
November 12, 2014
Joe Travaglini
Director of Products
Sqrrl
John Kreisa
VP Strategic Marketing
Hortonworks

The Modern Data Architecture
Hortonworks. We do Hadoop.
John Kreisa,
VP Strategic Marketing
Hortonworks

Agenda
• Apache Hadoop and a Modern Data Architecture
• Security in a comprehensive data management platform
• Security Analytics using (Big) Cybersecurity Data
• Case study: Internal network breach

Our Mission: Power your Modern Data Architecture
with HDP and Enterprise Apache Hadoop
Who we are
June 2011: Original 24 architects, developers, operators of Hadoop from Yahoo!
June 2014: An enterprise software company with 420+ Employees
Our model
Innovate and deliver Apache Hadoop as a complete enterprise data platform
completely in the open, backed by a world class support organization
Key Partners

Why a Modern Data Architecture?
Business
Analytics
LIMITATIONS
Silos & Expensive
Single Purpose
DATA SYSTEM APPLICATIONS
Custom
Applications
Packaged
Applications
RDBMS EDW MPP
MDA: Key Drivers
1. Leverage new types of data
2. IT optimization
3. Enable a data lake
GOALS
• Extend new data sets across
existing data platforms
• Common data platform, multiple
processing engines
• Batch, interactive and real time
on a single data platform
EXISTING
Systems
Clickstream
Web
&Social
Geoloca9on
Sensor
&
Machine
Server
Logs
Unstructured
SOURCES

A Modern Data Architecture Includes Hadoop
Hadoop compliments and
enhances existing technologies
Common data set, multiple applications
• Optionally land all data in a single cluster
• Batch, interactive & real-time use cases
• Support multi-tenant access, processing
& segmentation of data
YARN: Architectural center of Hadoop
• Consistent security, governance & operations
• Ecosystem applications certified "
by Hortonworks to run natively in Hadoop
SOURCES
EXISTING
Systems
Clickstream
Web
&Social
Batch Interactive Real-Time
HDFS
(Hadoop Distributed File System)
Geoloca9on
Sensor
&
Machine
Server
Logs
Unstructured
DATA SYSTEM APPLICATIONS
Business
Analytics
Custom
Applications
Packaged
Applications
RDBMS EDW MPP YARN: Data Operating System
1 ° ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° ° N

Unlock New Applications from New Types of Data
INDUSTRY USE CASE Sentiment
& Web
Clickstream
& Behavior
Machine
& Sensor Geographic Server Logs Structured &
Unstructured
Financial Services
New Account Risk Screens ✔ ✔
Trading Risk ✔
Insurance Underwriting ✔ ✔ ✔
Telecom
Call Detail Records (CDR) ✔ ✔
Infrastructure Investment ✔ ✔
Real-time Bandwidth Allocation ✔ ✔ ✔
Retail
360° View of the Customer ✔ ✔ ✔
Localized, Personalized Promotions ✔
Website Optimization ✔
Manufacturing
Supply Chain and Logistics ✔
Assembly Line Quality Assurance ✔
Crowd-sourced Quality Assurance ✔
Healthcare
Use Genomic Data in Medial Trials ✔ ✔ ✔
Monitor Patient Vitals in Real-Time ✔ ✔
Pharmaceuticals
Recruit and Retain Patients for Drug Trials ✔ ✔
Improve Prescription Adherence ✔ ✔ ✔ ✔
Oil & Gas
Unify Exploration & Production Data ✔ ✔ ✔ ✔
Monitor Rig Safety in Real-Time ✔ ✔ ✔
Government
ETL Offload/Federal Budgetary Pressures ✔ ✔
Sentiment Analysis for Government
Programs ✔

Break Down Silos with a Security Data Lake
SCALE
SCOPE
Unlocking the Data Lake
RDBMS
MPP
EDW
• Data Lake
Enabled by YARN
• Single data repository,
shared infrastructure
• Multiple security apps
accessing all the data
• Enable a shift from
reactive to proactive
interactions
• Gain new insight across
the entire enterprise
New Analytic Apps
or IT Optimization
HDP 2.1
Governance
& Integration
Security
Operations
Data Access
YARN
Data Management

Big Data is Changing Cyber Security
“By 2016, more than 25 percent of global firms will adopt big data
analytics for at least one security and fraud detection use case, up from
current eight percent.”
– Gartner Cyber Security report Feb 2014
Gartner recommendations
• Align security capabilities in a holistic security strategy tailored to the threats and risks
• Target a single architecture to collect, index, normalize, analyze and share all information
• Organizations should look for profile accounts, users or other entities, and look for
anomalous transactions against those profiles

How Can Big Data Analytics Help Cyber Security?
• To prioritize threats, vulnerabilities, and attacks
• To control endpoints and mobile connections /devices
• To prevent insecure devices from accessing secure systems
• To provide intelligence about the threat land- scape
• To reduce false positives

© Hortonworks Inc. 2014
Securely explore your data
CYBER PATTERN
DISCOVERY USING
LINKED DATA ANALYSIS
A Big Data Solution with Hortonworks and Sqrrl
Joe Travaglini, Director of Products, Sqrrl
© 2014 Sqrrl Data, Inc. | All Rights Reserved

Who We Are

Agenda
• Security Analytics using (Big) Cybersecurity Data
• Dealing with the new security dilemma
• Why Hadoop and HDP are the perfect fit
• The ‘Linked Data’ Approach
• Case study: internal network breach
• Overview of scenario
• Data modeling with Sqrrl
• Visual, contextual research and analysis

The Numbers Don’t Lie
229 87%
90% $12.7M
Source: Mandiant Source: Verizon
Source: Verizon Source: Ponemon

Targeted Attacks Have Changed the Game
Source:
Battery Ventures

What Does This Mean For Us?
Dissolution of the Secure Perimeter
• You’ve been breached. Deal with it.
• Empower the investigator
• Research and respond: better, faster, smarter
• It’s all about speed to understanding

The Security Data Dilemma
Detecting attacks requires more (i.e. BIG) data
But your tools
can’t handle the
big data wave
So attackers are spilling in

A Modern Data Architecture
Hortonworks and Sqrrl Solution
• Hortonworks Data Platform
at the core
• Sqrrl Enterprise stack at the
app layer
Hadoop enables us to look at
data differently

Sqrrl Enterprise Architecture
Interface
Processing
Data Model
Data Storage
Visualization / API ML + Anomaly Detection
Query Engine Bulk/Graph Processing
Raw Events Linked Data Model
HDFS + Accumulo
Commodity Hardware
Physical
Security
Audit
Cryptography
Labeling + Policy

Big Data Transformed
Data Sources Linked Contextual Knowledge Analysis
Email
Security Data
VPN FW
Network Data
Proxy NetFlow
Application Data
HR USB

Linked Data Analysis
Adding structure to the noise

Case Study:
Compromised Network

Breach Detection Scenario

Case Study Model
Data Sources
Linked Meta Model
Users
Hosts
login
flow
login
DNS
records
Netflow
Host logs
Database
logs
External
Alerts

Case Study Example Mapping
Netflow Records
startTime endTime sourceIP destIP sourcePort destPort protocol tcpFlags bytesIn bytesOut
10/22/14 8:58
10/22/14 8:58
10.0.2.15
192.168.0.123
37051
139
TCP
...RS.
100
3355
10/22/14 8:45
10/22/14 8:45
10.0.2.15
192.168.0.6
0
3328
ICMP
......
40
100
10/22/14 8:59
10/22/14 8:59
192.168.0.119
10.0.2.15
139
60071
TCP
.A..S.
46
351
10.0.2.15
192.168.
0.123
Class=Flow,
totalBytes = 3455
192.168.
Class=Flow, 0.6
totalBytes = 140

Case Study Example Data

Investigation Process
1. Set the Stage 2. Enable Search
and Discovery
3. Automate
Analysis
• Define the
security-centric
entity/
relationship
model
• Extract and
maintain the
model
• Visually
navigate assets
and actors in
the network
• Drill down to the
raw data
seeding the
model
• Use behavioral
analytics to build
expectations of
‘normal’
• Flag entities as
potentially
‘abnormal’ and
sniff them out

Visualizing the Threat

Thanks!
Joe Travaglini
Director of Products, Sqrrl Data, Inc.
@joe_travaglini
jtrav@sqrrl.com
http://www.sqrrl.com

Hortonworks sqrrl webinar v5.pptx

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (6)

Semelhante a Hortonworks sqrrl webinar v5.pptx

Semelhante a Hortonworks sqrrl webinar v5.pptx (20)

Mais de Hortonworks

Mais de Hortonworks (20)

Último

Último (20)

Hortonworks sqrrl webinar v5.pptx