Real Time BI with Hadoop

•Transferir como KEY, PDF•

16 gostaram•2,091 visualizações

This document discusses using Hadoop for real-time business intelligence (BI). It notes that scalability is now critical for BI due to the growth of data and social media. Hadoop provides a scalable and cost-effective solution compared to traditional SQL databases. Key components are distributed search with Katta and faceted search for interactive exploration and aggregation of both structured and unstructured data. The goal is responses in under 5 seconds to support real-time BI.

Tecnologia Negócios

Real-Time BI in
Hadoop
Bradford Stephens

Lead Engineer, Visible Technologies
Principal Consultant, Drawn to Scale Consulting

Topics

• Scalability and BI
• Costs and Abilities
• Search as BI

What is “Real-Time”

• Understanding Latency
• We aim for <5 secs.

Scalability in BI

• Scalbility matters now
• Social Media: Catalyst
• All data is important
• Data doesn’t scale with business size any
more

Search as BI

• Katta = Distributed Search on Haddoop
• Bobo = Faceted Lucene

Doing it Cheap

• 100 TB, Structured and Unstructured
• Oracle- $100,000,000
• “NewSQL” - $4,000,000
• Hadoop + Katta - $250,000

Why We Need Hadoop

• Need to process high-latency data to get
the “small stuff” fast
• Robust Ecosystem
• Need more than SQL. RDBMS not a Swiss-
Army Knife

Aggregation is Real-
Time

• Distributed Search w/ Katta + Facets =
Aggregation-Based BI
• Sum, Count, Filter, Avg, Group

Protips: Review

• Understand High vs. Low Latency data
• Hadoop makes it cheap
• Pre-aggregate w/ Hadoop, Explore w/ Katta
+ Faceted Search

The Future

• Search/BI as a Platform: “Google my Data
Warehouse”
• Real-Time MR on HBase

Mais conteúdo relacionado

Mais procurados

Netflix: Using Big Data in the Cloud to Drive EngagementCoy Dean

(ARC312) Processing Money in the Cloud | AWS re:Invent 2014Amazon Web Services

Analyzing Billions of Data Rows with Alteryx, Amazon Redshift, and TableauDATAVERSITY

Qubole presentation for the Cleveland Big Data and Hadoop Meetup Qubole

Webinar with SnagAJob, HP Vertica and Looker - Data at the speed of busines s...Looker

AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...Amazon Web Services

Tips in migrating to SharePoint 2016 or O365, to avoid a migration headacheMike Maadarani

The Future of Content is Real-Time: Leveraging Artificial Intelligence to Del...Karl Johnson, MBA

Apache Cassandra: NoSQL in the enterprisejbellis

Netflix - Elevating Your Data Platform - TDWI Keynote - San Diego 2015Kurt Brown

Know and Grow Alignment 1Mahmoud M. Selim

Big data from the trenchesAzrul MADISA

The practice of big data - making big data approachablekcmallu

Kyvos insightsrebeccatho

Build your first spark big data environment in azureDiego Nogare

Gab Genai Cloudera - Going Beyond Traditional Analytic IntelAPAC

Sap amazon and Kogent DemoJon Boutelle

Big Data Science Challenges in MediaChandan Rajah

AWS Summit 2013 | Singapore - Delivering Search for Today's Local, Social, an...Amazon Web Services

Ran Rothschild - CloudZoneIdan Tohami

Mais procurados (20)

Netflix: Using Big Data in the Cloud to Drive Engagement

(ARC312) Processing Money in the Cloud | AWS re:Invent 2014

Analyzing Billions of Data Rows with Alteryx, Amazon Redshift, and Tableau

Qubole presentation for the Cleveland Big Data and Hadoop Meetup

Webinar with SnagAJob, HP Vertica and Looker - Data at the speed of busines s...

AWS Partner Webcast - Analyze Big Data for Consumer Applications with Looker ...

Tips in migrating to SharePoint 2016 or O365, to avoid a migration headache

The Future of Content is Real-Time: Leveraging Artificial Intelligence to Del...

Apache Cassandra: NoSQL in the enterprise

Netflix - Elevating Your Data Platform - TDWI Keynote - San Diego 2015

Know and Grow Alignment 1

Big data from the trenches

The practice of big data - making big data approachable

Kyvos insights

Build your first spark big data environment in azure

Gab Genai Cloudera - Going Beyond Traditional Analytic

Sap amazon and Kogent Demo

Big Data Science Challenges in Media

AWS Summit 2013 | Singapore - Delivering Search for Today's Local, Social, an...

Ran Rothschild - CloudZone

Destaque

ETL big data with apache hadoopMaulik Thaker

Omid: A Transactional Framework for HBaseDataWorks Summit/Hadoop Summit

IoT Crash Course Hadoop Summit SJDaniel Madrigal

Making the leap to BI on Hadoop by Mariani, dave @ atscaleTin Ho

Using Hadoop for Cognitive AnalyticsDataWorks Summit/Hadoop Summit

Curb your insecurity with HDPDataWorks Summit/Hadoop Summit

The Path to Wellness through Big DataDataWorks Summit/Hadoop Summit

Combining Machine Learning frameworks with Apache SparkDataWorks Summit/Hadoop Summit

What the #$* is a Business Catalog and why you need it DataWorks Summit/Hadoop Summit

The Future of Apache StormDataWorks Summit/Hadoop Summit

HIPAA Compliance in the CloudDataWorks Summit/Hadoop Summit

Real Time Machine Learning Visualization with SparkDataWorks Summit/Hadoop Summit

Open Source Ingredients for Interactive Data Analysis in Spark DataWorks Summit/Hadoop Summit

Machine Learning for Any Size of Data, Any Type of DataDataWorks Summit/Hadoop Summit

A New "Sparkitecture" for modernizing your data warehouseDataWorks Summit/Hadoop Summit

Extreme Analytics @ eBayDataWorks Summit/Hadoop Summit

Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on HiveDataWorks Summit/Hadoop Summit

Intro to Spark with Zeppelin Crash Course Hadoop Summit SJDaniel Madrigal

The Future of Apache Hadoop an Enterprise Architecture ViewDataWorks Summit/Hadoop Summit

Accelerating Data Warehouse ModernizationDataWorks Summit/Hadoop Summit

Destaque (20)

ETL big data with apache hadoop

Omid: A Transactional Framework for HBase

IoT Crash Course Hadoop Summit SJ

Making the leap to BI on Hadoop by Mariani, dave @ atscale

Using Hadoop for Cognitive Analytics

Curb your insecurity with HDP

The Path to Wellness through Big Data

Combining Machine Learning frameworks with Apache Spark

What the #$* is a Business Catalog and why you need it

The Future of Apache Storm

HIPAA Compliance in the Cloud

Real Time Machine Learning Visualization with Spark

Open Source Ingredients for Interactive Data Analysis in Spark

Machine Learning for Any Size of Data, Any Type of Data

A New "Sparkitecture" for modernizing your data warehouse

Extreme Analytics @ eBay

Faster, Faster, Faster: The True Story of a Mobile Analytics Data Mart on Hive

Intro to Spark with Zeppelin Crash Course Hadoop Summit SJ

The Future of Apache Hadoop an Enterprise Architecture View

Accelerating Data Warehouse Modernization

Semelhante a Real Time BI with Hadoop

Strata Online_road_to_enterprise_data_2011Lynn Langit

Real Time Interactive Queries IN HADOOP: Big Data Warehousing MeetupCaserta

Understanding Big Data for policy professionalsAlex Jouravlev

Accelerating analytics in a new era of dataArnon Shimoni

5 Things that Make Hadoop a Game ChangerCaserta

New World Hadoop Architectures (& What Problems They Really Solve) for Oracle...Rittman Analytics

Big Data and BI Best PracticesYellowfin

Big Data 2.0: YARN Enablement for Distributed ETL & SQL with HadoopCaserta

Big data and bi best practices slidedeckActian Corporation

Unlocking the Power of the Data LakeArcadia Data

Blueprint for integrating big data analytics and biDataWorks Summit

Big data and mstr bridge the elephantKognitio

Make Life Suck Less (Building Scalable Systems)Bradford Stephens

Make Life Suck Less (Building Scalable Systems)guest0f8e278

Data Integration and Data Warehousing for Cloud, Big Data and IoT:  What’s Ne...Rittman Analytics

Big Data Strategy for the Relational World Andrew Brust

Architecting Your First Big Data ImplementationAdaryl "Bob" Wakefield, MBA

Retail & CPGTata Consultancy Services

Building a Business on Hadoop, HBase, and Open Source Distributed ComputingBradford Stephens

Big Data Technologies and Why They Matter To R UsersAdaryl "Bob" Wakefield, MBA

Semelhante a Real Time BI with Hadoop (20)

Strata Online_road_to_enterprise_data_2011

Real Time Interactive Queries IN HADOOP: Big Data Warehousing Meetup

Understanding Big Data for policy professionals

Accelerating analytics in a new era of data

5 Things that Make Hadoop a Game Changer

New World Hadoop Architectures (& What Problems They Really Solve) for Oracle...

Big Data and BI Best Practices

Big Data 2.0: YARN Enablement for Distributed ETL & SQL with Hadoop

Big data and bi best practices slidedeck

Unlocking the Power of the Data Lake

Blueprint for integrating big data analytics and bi

Big data and mstr bridge the elephant

Make Life Suck Less (Building Scalable Systems)

Data Integration and Data Warehousing for Cloud, Big Data and IoT:  What’s Ne...

Big Data Strategy for the Relational World

Architecting Your First Big Data Implementation

Retail & CPG

Building a Business on Hadoop, HBase, and Open Source Distributed Computing

Big Data Technologies and Why They Matter To R Users

Último

A Journey Into the Emotions of Software DevelopersNicole Novielli

Scale your database traffic with Read & Write split using MySQL RouterMydbops

Connecting the Dots for Information Discovery.pdfNeo4j

Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq

Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada

Time Series Foundation Models - current state and future directionsNathaniel Shimoni

How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes

So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda

DevEX - reference for building teams, processes, and platformsSergiu Bodiu

A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3

The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3

The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3

Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3

Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3

Take control of your SAP testing with UiPath Test SuiteDianaGray10

Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll

Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3

From Family Reminiscence to Scholarly Archive .Alan Dix

Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen

Real Time BI with Hadoop

1. Real-Time BI in Hadoop Bradford Stephens Lead Engineer, Visible Technologies Principal Consultant, Drawn to Scale Consulting

2. Topics • Scalability and BI • Costs and Abilities • Search as BI

6. What Is BI?

8. What is “Real-Time” • Understanding Latency • We aim for <5 secs.

10. Scalability in BI • Scalbility matters now • Social Media: Catalyst • All data is important • Data doesn’t scale with business size any more

11. Search as BI • Katta = Distributed Search on Haddoop • Bobo = Faceted Lucene

12.

13.

14.

15.

16.

17. Doing it Cheap • 100 TB, Structured and Unstructured • Oracle- $100,000,000 • “NewSQL” - $4,000,000 • Hadoop + Katta - $250,000

18. Why We Need Hadoop • Need to process high-latency data to get the “small stuff” fast • Robust Ecosystem • Need more than SQL. RDBMS not a Swiss- Army Knife

19. Aggregation is Real- Time • Distributed Search w/ Katta + Facets = Aggregation-Based BI • Sum, Count, Filter, Avg, Group

20. Protips: Review • Understand High vs. Low Latency data • Hadoop makes it cheap • Pre-aggregate w/ Hadoop, Explore w/ Katta + Faceted Search

21. The Future • Search/BI as a Platform: “Google my Data Warehouse” • Real-Time MR on HBase

Real Time BI with Hadoop

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (20)

Semelhante a Real Time BI with Hadoop

Semelhante a Real Time BI with Hadoop (20)

Último

Último (20)

Real Time BI with Hadoop