O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
A streaming architecture
for Cyber Security
with NiFi, Hadoop, Storm and Metron
Simon Elliston Ball
• Product Manager
• Data Scientist
• Elephant herder
• @sireb
IoT: Mirai
Reports of 1.2 Tbps
500,000 devices at peak
DDoS attacks on Dyn DNS services
Drowning in
Data
The value of real time
Data in Motion: why wait until it’s at rest?
Correct context: the world moved on
Better data = analyst efficiency
Fully enriched data
Real context
Consistency
= faster triage and better coverage
Network Level Taps
Data Sources and Aggregation
Open standards for data models =
more productive data scientists +
shareable models
Business ...
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
DataServicesandIntegrationLayer
ModulesReal-time Processing
Cyber Se...
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment is the key
to context
Human
Resources
Database
Metron
Da...
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
But time is context too… profiling by time
t = 1 t = 2 t = 3 t = n
...
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Stellar: Excel functions for Cyber security
{
"profile": "auth_dist...
Thank you!
Apache Metron: http://metron.apache.org
Twitter: @sireb
Próximos SlideShares
Carregando em…5
×

A streaming architecture for Cyber Security - Apache Metron

1.470 visualizações

Publicada em

Apache Metron, introduced at Big Data Week London, 2017, along with a few of the reasons for a data science driven approach to cyber security

Publicada em: Tecnologia
  • Seja o primeiro a comentar

A streaming architecture for Cyber Security - Apache Metron

  1. 1. A streaming architecture for Cyber Security with NiFi, Hadoop, Storm and Metron
  2. 2. Simon Elliston Ball • Product Manager • Data Scientist • Elephant herder • @sireb
  3. 3. IoT: Mirai Reports of 1.2 Tbps 500,000 devices at peak DDoS attacks on Dyn DNS services
  4. 4. Drowning in Data
  5. 5. The value of real time Data in Motion: why wait until it’s at rest? Correct context: the world moved on
  6. 6. Better data = analyst efficiency Fully enriched data Real context Consistency = faster triage and better coverage
  7. 7. Network Level Taps
  8. 8. Data Sources and Aggregation Open standards for data models = more productive data scientists + shareable models Business level data sources link security to real business risk.
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataServicesandIntegrationLayer ModulesReal-time Processing Cyber Security Engine Telemetry Parsers Apache Metron: a framework for Big Data Driven cyber security TelemetryIngestBuffer Telemetry Data Collectors Real-time Enrich / Threat Intel Streams Performance Network Ingest Probes / OtherMachine Generated Logs (AD, App / Web Server, firewall, VPN, etc.) Security Endpoint Devices (Fireye, Palo Alto, BlueCoat, etc.) Network Data (PCAP, Netflow, Bro, etc.) IDS (Suricata, Snort, etc.) Threat Intelligence Feeds (Soltra, OpenTaxi, third-party feeds) Telemetry Data Sources Data Vault Real-Time Search Evidentiary Store Threat Intelligence Platform Model as a Service Community Models Data Science Workbench PCAP Forensics Threat IntelligenceEnrichment Indexers and WriterProfiler Alert Triage Cyber Security Stream Processing Pipeline
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Enrichment is the key to context Human Resources Database Metron Data Metron Data App Logs Active Directory Network Traffic Logs IoT Asset Database Geo, Threat, Traditional Security data sources Business Risk Data Metron Data Standard, Consistent Data Format Streaming enrichment Batch enrichment Fully Enriched data ready for analysis Wide variety of real- time and batch sources
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved But time is context too… profiling by time t = 1 t = 2 t = 3 t = n Wide range of algorithms including:  HyperLogLogPlus  Bloom filters  T-digests  Statistical Baselining  Hashing functions  Outlier detection  GeoHashing over time  Locality Sensitive Hashing Approx. Data Sketch Approx. Data Sketch Approx. Data Sketch Approx. Data Sketch Combined Baseline Statistic
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Stellar: Excel functions for Cyber security { "profile": "auth_distribution", "foreach": "'global'", "onlyif": "profile == 'attempts_by_user'", "init": { "s": "STATS_INIT()" }, "update": { "s": "STATS_ADD(s, total_count)" }, "result": "s" } Building a Profile Using a Profile window := PROFILE_WINDOW('...') profile := PROFILE_GET('attempts_by_user', user, window) distinct_auth_attempts := HLLP_CARDINALITY(GET_LAST(profile)) distribution_profile := PROFILE_GET('auth_distribution', 'global', window) stats := STATS_MERGE(distribution_profile) distinct_auth_attempts_median := STATS_PERCENTILE(stats, 0.5) distinct_auth_attempts_stddev := STATS_SD(stats) • Simple • Expression based • Function composition • Boolean operators • In-stream
  13. 13. Thank you! Apache Metron: http://metron.apache.org Twitter: @sireb

×