Following up on last year's Threat Analytics 101 talk, I'll go over last year's lessons learned, the creation of the open source Apache Metron project, and where I see the future of security analytics may take us.

1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
A year's lesson learned
Michael Schiebel
Cybersecurity Strategist
3/30/2016
Security analytics journey

2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Today’s talk regards:
Community Driven,
Open Source Projects

3. 3 © Hortonworks Inc. 2011 – 2016. All Rights ReservedPage3 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
To be 100% Open you must
influence the roadmap
We Employ the Committers
--one third of all committers to the Apache®
Hadoop™ project, and a majority in other
important projects
Our Committers Innovate
and expand both Open Enterprise Hadoop and
Apache NiFi
We Influence the Hadoop Roadmap
by communicating important requirements to the
community through our leaders
A PA C H E H A D O O P C O M M I T T E R S

4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda • Who am I?
• A year in review
• Core lessons
• Apache Nifi
• Apache Metron
• Putting it all together
• Next steps
• Questions

5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Who am I?
 +15 years information security experience
– Financial & Healthcare
– Incident Response
– Computer Forensics & Malware analysis
– Application Security
– Security Architecture
– Security Executive
• Strategy
• Business case development
 Hortonworks Cybersecurity Strategist
– Your Advocate
– Hortonworks’ vision for Apache Metron
– Evangelize an open source approach
Cybercrime is largest transfer of wealth in the history of civilization.
We need an open solution available to everyone.
Only together can we solve this problem.

6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
A year in review
• Asked to create an open source organization
• Met the Hortonworks folks
• OpenSOC forked
• Apache Foundation accepted
• 12/15 Apache (Incubating) Metron created
• ~30 committers
• Multi-company support
• Decided to join Hortonworks
• Apache Nifi!
• The single (IMHO) most important open source project
in 2015.
• WYSIWYG data ingest and transformation
• Data Provenance(Chain of custody)

7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core Lesson – Data Ingestion
 Finding and getting data is hard
 Secure delivery of data is hard
Cloud
Data
Center
Data
Center
DMZ
Data
Lake
Traditional tools generate tons of data as signature based events.
Analytics needs access to raw activity data
Don’t underestimate the challenge – it’s a people and process problem

8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core Lesson – Don’t be the new shiny
 Focus on the value proposition
– Show how it makes the job easier and more efficient at each phase of implementation.
– How current state isn’t effective.
– Metrics, Metrics, Metrics
 Don’t hype the technology – this can be viewed as scary or untried
– Analytics is mature – predates computers
– Unlike other technologies, analytics shows the evidence the prediction is based on.
– It’s just math.
“Why are you building some new Star Trek thing? We need to focus on the basics.”

9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Core Lesson – Avoid YADB (Yet another dashboard)
 It’s about the people
– Make the tool fit the people and process
– Does this make their job easier?
 Integration and workflow are key
 Focus on automated detection and response

10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Nifi
 WYSIWYG Data Ingest
 Secure data
transmission
 Chain of custody
 Data enrichment
pipeline

11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
PCAP
NETFLOW
DPI
Network
Tap
IDS
AV
EMAIL
FIREWALL
HOST LOGS
PARSE
NORMALIZE
USER
ASSET
GEO
WHOIS
CONN
TAG
VALIDATE
PROCESS
ENRICH
STIX
Flat Files
Aggregators
Model As A
Service
Cloud
Services
LABEL
Real-Time
Search
Interactive
Dashboards
Data
Modelling
Knowledge
Graphs
PCAP
Store
Integration
Layer
PCAP
Replay
Security
Layer
Workflow
Engine
Rules
Engine
Apache Metron

12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Putting it all together

13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Next Steps
 Call to action:
– Continue to build community: Apache Metron, Apache Nifi
– Become a committer
– Help write documentation
– Try out projects and provide feedback and suggestions
 Help develop open data schema
– How can we extend to solve other security problems? (FAIR actuarial tables…)
 Other opens source projects
– Workflow
– UI
– Sensors
– Automated Response
– Container/DevOps extensions

14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
Blog: http://www.hortonworks.com/blog/author/mschiebel/
Email: mschiebel@hortonworks.com
Twitter: @mgschiebel