A Community Approach to Fighting Cyber Threats
•
1 like•593 views
3 Things to Learn About: *Infinitely scale data storage, access, and machine learning *Provide community defined open data models for complete enterprise visibility *Open up application flexibility while building on a future proofed architecture
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A Community Approach to Fighting Cyber Threats
Similar to A Community Approach to Fighting Cyber Threats (20)
More from Cloudera, Inc.
More from Cloudera, Inc. (20)
Recently uploaded
Recently uploaded (20)
A Community Approach to Fighting Cyber Threats
- 1. 1© Cloudera, Inc. All rights reserved. A Community Approach to Fighting Cyber Threats The vision for Apache Spot (Incubating)
- 2. 2© Cloudera, Inc. All rights reserved. Agenda • Introductions • Apache Spot Overview - Alan Ross • Future Proofing Cybersecurity with Open source – Doug Cutting • Live Q&A
- 3. 3© Cloudera, Inc. All rights reserved.
- 4. 4© Cloudera, Inc. All rights reserved. Poll Question Number 1 • How much security data do you currently store? • >500TB • 500TB to 999TB • 1PB to 5PB • 5PB+
- 5. 5© Cloudera, Inc. All rights reserved. Apache Spot Overview
- 6. 6© Cloudera, Inc. All rights reserved. SIEMSIEM Our story at Intel…
- 7. 7© Cloudera, Inc. All rights reserved. SIEMSIEM
- 8. 8© Cloudera, Inc. All rights reserved. 1,000,000,000,000+ [ events per day ]
- 9. 9© Cloudera, Inc. All rights reserved. Stop Independently Solving Universal Challenges Popular cyber platforms can not cost effectively scale to the volume and variety of modern data Only partial view of the enterprise limits analytics and slows investigations Difficult to deploy advanced machine learning detection capabilities Explosion of Data Limited Enterprise Visibility Limited Analytic Processing DataAccess 1%50%100% DataVolume 10PB1PB1TB IF (X) AND (Y) THEN (Z) Time User Network Endpoint Archived Data Emerging Data
- 10. 10© Cloudera, Inc. All rights reserved. A community approach to fighting cyber threats.
- 11. 11© Cloudera, Inc. All rights reserved. Detect advanced threats faster via machine learning and artificial intelligence Faster time to incident investigation and response with comprehensive enterprise visibility Change the economics of cybersecurity with an open source platform that supports multiple LOB workloads The Value of Apache Spot
- 12. 12© Cloudera, Inc. All rights reserved. Integrate Spot with Existing Investments Threat Intelligence Network User Endpoint Data Sources PackagedApplications Analytic Processing (Spark, Impala, Solr) Data and Analytic Management Apache Spot’s Platform SIEM Spot Algorithms Custom Spot Test Data Open Data Models (HDFS, Hbase) Ingestion (Kafka, Streamsets) (On premise or Cloud)
- 13. 13© Cloudera, Inc. All rights reserved. Many application on one shared data set and architecture Visualization & machine learning cybersecurity applications can share common data set & infrastructure CustomPackaged Spot community is developing out machine learning (e.g. network threat detection) Open Source Build custom applications & analytics using Spot without having to buy new infrastructure
- 14. 14© Cloudera, Inc. All rights reserved. Tier 3 SME & Hunter (Network) Tier 3 SME & Hunter (Users) Tier 3 SME & Hunter (Endpoint) Tier 3 SME & Hunter (Threat Intel) SOC Manager Tier 2 Incident Responder Tier 2 Incident Responder Tier 1 Alert Analyst Tier 1 Alert Analyst Tier 1 Alert Analyst Tier 1 Alert Analyst Empowering the Security Operations Center Tier 1 Alert Analyst Tier 2 Incident Responder Tier 3 SME & Hunter SOC Manager Fewer false positives due to ML based detection Comprehensive contextual data for faster investigation and response Wider and deeper data for large scale advanced analytics Everyone is leveraging same data and architecture at a lower cost
- 15. 15© Cloudera, Inc. All rights reserved. Poll Question Number 2 • How many cybersecurity applications do you use? • 1 • 2-5 • 6-9 • 10+
- 16. 16© Cloudera, Inc. All rights reserved. Future Proofing Cybersecurity With Open Source Faster Innovation Open Data Models Partner Ecosystem Any Data Any Analytics Any Scale
- 17. 17© Cloudera, Inc. All rights reserved. SIEM (TBs) Aggregated Events Raw System Logs Network Flows/ DNS Full Packet Capture Video, Text, Images User Data Data Types (MBs>PBs) Native Analytics (Basic > Advanced)Search Correlations SQL Machine Learning Advanced Statistics 1 10 20 40 Time (Months) 3 CDH (PBs) Industry leaders already build on CDH 1 Market Trends: User and Entity Behavior Analytics (UEBA) Expand Their Market Reach – Gartner April 2016 2 Magic Quadrant for Managed Security Services, Worldwide – Gartner December 2015 •60% of UEBA Vendors to Watch use CDH •25% of Network Traffic Analysis Vendors to Watch use CDH •50% of MSSP ‘Leaders’ use CDH •Gartner named Cloudera Non- Security-Specific Analytics Vendors to Watch1
- 18. 18© Cloudera, Inc. All rights reserved. CDHTraditional (e.g. RDBMS) DataAccess 1%50%100% DataVolume 10PB1PB1TB Time DataAccess 1%50%100% DataVolume 10PB1PB1TB Time Scaling Data Storage and Access
- 19. 19© Cloudera, Inc. All rights reserved. Any Data from Any Source
- 20. 20© Cloudera, Inc. All rights reserved. Community Defined Open Data Models (ODM) Endpoint User Network DIVERSE DATA SOURCES SINGLE ACCESS CYBERSECURITY ITOPTIM IZATION USEREXPERIENCEFRAUD
- 21. 21© Cloudera, Inc. All rights reserved. Large Scale Analytics A Google search experience across all of your cybersecurity data Advanced AnalyticsSearch Large scale visualizations and SQL queries for descriptive and diagnostic analytics Visualizations Execute large scale machine learning queries against PBs worth of data for threat detection
- 22. 22© Cloudera, Inc. All rights reserved. NEW PROJECTS EXISTING PROJECTS *CDH SUPPORTED Core Hadoop (HDFS, MapReduce) Solr* Pig* Core Hadoop HBase ZooKeeper Solr Pig Core Hadoop Hive Mahout HBase ZooKeeper Solr Pig Core Hadoop Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig Core Hadoop Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Knox Flink Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie HCatalog Hue Sqoop Avro Hive Mahout HBase ZooKeeper Solr Pig YARN Core Hadoop Kudu RecordService Ibis Falcon Nifi Knox Flink Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie Hcatalog Hue Sqoop Avro Hive Mahout Hbase ZooKeeper Solr Pig YARN Core Hadoop 200 6 2008 2009 2010 2011 2012 20132007 2014 SparklyR Arrow Kudu RecordService Ibis Falcon Nifi Knox Flink Parquet Sentry Spark Tez Impala Kafka Drill Flume Bigtop Oozie Hcatalog Hue Sqoop Avro Hive Mahout Hbase ZooKeeper Solr Pig YARN Core Hadoop Tensorflow Spot Eagle Beam SparklyR Arrow Kudu* RecordService* Ibis* Falcon Nifi Knox Flink Parquet* Sentry* Spark* Tez Impala* Kafka* Drill Flume* Bigtop* Oozie* Hcatalog* Hue* Sqoop* Avro* Hive* Mahout* Hbase* ZooKeeper* Solr* Pig* YARN* Core Hadoop* 2016 Present2015 Innovate Faster than a Single Vendor
- 23. 23© Cloudera, Inc. All rights reserved. Partners easily build off of open architecture…
- 24. 24© Cloudera, Inc. All rights reserved. Q&A
- 25. 25© Cloudera, Inc. All rights reserved. Subscribe on…
- 26. 26© Cloudera, Inc. All rights reserved. Thank you
Editor's Notes
- Intel story…
- Intel story…
- Intel story…
- How can we solve these universal challenges…
- Apache Spot was the answer
- Talk SIEM integration Mention Kudu
- Packaged: Visualization & machine learning cybersecurity applications can share common data set & infrastructure Open Source: Apache Spot (incubating) community is developing out machine learning (e.g. network threat detection) [Image is Arcadia app on top of the open source ML] Customer: Build custom applications & analytics using Cloudera’s components without having to buy new infrastructure
- It helps the SOC team…
- Having a standard, enhances collaboration (data sharing and analytics sharing. A single copy of enriched data improves detection and accelerates investigation and response Open data model for network, user, endpoint is defined by the Apache Spot community, not a single vendor Open source storage of security data breaks vendor lock in
- As you can see, Hadoop is not the same Hadoop from a few years ago. We have come a long way and Hadoop ecosystem continues to evolve and expand by the day Every year we bring in new and exciting projects that help solve some of the challenges for our customers and support new and compelling use cases using Hadoop Some of the recent examples include projects such Kudu & Record Service which are now an integral part of the platform. We currently integrate 25+ independent open source projects – All of which have different release schedules and cadences. We bring these together, integrate these, test these, ensure that security attributes are functioning and make sure all of these work together to deliver a stable, production-ready platform that enterprises can leverage. ---------------- Little incentive in ASF to look cross project, though multi-component use is the primary path to customer success. This ever growing community continues to grow as. As more businesses continue to use Hadoop, more use cases continue to emerge creating additional projects that allow for more robust data applications. With Hadoop you don’t just get the code your team built, you get the code the community built.