Usage case of HBase for real-time application

•

6 gostaram•2,793 visualizações

Edward Yoon

Tecnologia

A usage case of HBase
for real-time application

Edward J.Yoon

udanax.org
• URL shortener service
• Provides HTTP, JSON and RESTful API
• Used for some twitter clients and a
number of web sites
• Rows are increasing as almost 30 per
second
• Runs on 5 nodes HBase cluster

Block Diagram
Clients ...
Apache Web Server

Tomcat 1 Tomcat 2 ...
HBase
Hadoop HDFS

$HBase Table Schema row longURL: creator: clickLog: longURL: creator: clickLog:{referrer url} 3XqT <LONG_URL> <IP address> <IP address> ... longURL: 3xqS ... ... <LONG_URL> ... ...$

MR and Hama BSP

• Information-Flow Analysis
• Web Structure Mining
• User propensity Analysis

Hadoop Scheduler

• FIFO: Queue-based scheduler
• FSS: Fair share scheduler

fair share scheduler

• Group jobs into “pools”
• Assign each pool a guaranteed minimum
share
• Divide excess capacity evenly between
pools

Mais conteúdo relacionado

Mais procurados

HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBaseCloudera, Inc.

HBase: Extreme MakeoverHBaseCon

HBaseCon 2015: HBase at Scale in an Online and High-Demand EnvironmentHBaseCon

HBaseCon 2012 | Base Metrics: What They Mean to You - ClouderaCloudera, Inc.

HBase and HDFS: Understanding FileSystem Usage in HBaseenissoz

Harmonizing Multi-tenant HBase Clusters for Managing Workload DiversityHBaseCon

Meet HBase 1.0enissoz

HBaseCon 2015: State of HBase Docs and How to ContributeHBaseCon

hbaseconasia2017: HBase Disaster Recovery Solution at HuaweiHBaseCon

hbaseconasia2017: Large scale data near-line loading method and architectureHBaseCon

HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon

HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, ClouderaCloudera, Inc.

HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...Cloudera, Inc.

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed StorageCloudera, Inc.

HBaseCon 2013: Apache HBase ReplicationCloudera, Inc.

HBaseCon 2012 | HBase, the Use Case in eBay Cassini Cloudera, Inc.

HBase Read High Availability Using Timeline Consistent Region Replicasenissoz

Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsightHBaseCon

HBaseCon 2013:High-Throughput, Transactional Stream Processing on Apache HBase Cloudera, Inc.

Chicago Data Summit: Geo-based Content Processing Using HBaseCloudera, Inc.

Mais procurados (20)

HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase

HBase: Extreme Makeover

HBaseCon 2015: HBase at Scale in an Online and High-Demand Environment

HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera

HBase and HDFS: Understanding FileSystem Usage in HBase

Harmonizing Multi-tenant HBase Clusters for Managing Workload Diversity

Meet HBase 1.0

HBaseCon 2015: State of HBase Docs and How to Contribute

hbaseconasia2017: HBase Disaster Recovery Solution at Huawei

hbaseconasia2017: Large scale data near-line loading method and architecture

HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest

HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera

HBaseCon 2012 | Gap Inc Direct: Serving Apparel Catalog from HBase for Live W...

HBaseCon 2013: Apache HBase at Pinterest - Scaling Our Feed Storage

HBaseCon 2013: Apache HBase Replication

HBaseCon 2012 | HBase, the Use Case in eBay Cassini

HBase Read High Availability Using Timeline Consistent Region Replicas

Optimizing Apache HBase for Cloud Storage in Microsoft Azure HDInsight

HBaseCon 2013:High-Throughput, Transactional Stream Processing on Apache HBase

Chicago Data Summit: Geo-based Content Processing Using HBase

Semelhante a Usage case of HBase for real-time application

Rich Data Graphs for MapReduceScott Cinnamond

Sept 17 2013 - THUG - HBase a Technical IntroductionAdam Muise

CoAPing with the Internet of ThingsAnmol Sarma

haproxy-150423120602-conversion-gate01.pdfPawanVerma628806

HAProxy Arindam Nayak

Alexander Sibiryakov- FronteraPyData

Performance Analysis of HBASE and MONGODBKaushik Rajan

Facebook keynote-nicolas-qconYiwei Ma

支撑Facebook消息处理的h base存储系统yongboy

Facebook Messages & HBase强王

JagmohancrawlJag Mohan Singh

High Availability for HBase Tables - Past, Present, and FutureDataWorks Summit

HBase lon meetupMatteo Bertozzi

Whirr dev-up-puppetconf2011Puppet

Architectural Evolution Starting from HadoopSpagoWorld

Real time web appsSepehr Rasouli

Apache HBase: Where We've Been and What's Upcominghuguk

HDFS- What is New and FutureDataWorks Summit

Designing for ScaleWooga

Erlang factory 2011 londonPaolo Negri

Semelhante a Usage case of HBase for real-time application (20)

Rich Data Graphs for MapReduce

Sept 17 2013 - THUG - HBase a Technical Introduction

CoAPing with the Internet of Things

haproxy-150423120602-conversion-gate01.pdf

HAProxy

Alexander Sibiryakov- Frontera

Performance Analysis of HBASE and MONGODB

Facebook keynote-nicolas-qcon

支撑Facebook消息处理的h base存储系统

Facebook Messages & HBase

Jagmohancrawl

High Availability for HBase Tables - Past, Present, and Future

HBase lon meetup

Whirr dev-up-puppetconf2011

Architectural Evolution Starting from Hadoop

Real time web apps

Apache HBase: Where We've Been and What's Upcoming

HDFS- What is New and Future

Designing for Scale

Erlang factory 2011 london

Mais de Edward Yoon

(소스콘 2015 발표자료) Apache HORN, a large scale deep learningEdward Yoon

Introduction to apache horn (incubating)Edward Yoon

Apache Hama at Samsung Open Source ConferenceEdward Yoon

K means 알고리즘을 이용한 영화배우 클러스터링Edward Yoon

차세대하둡과 주목해야할 오픈소스Edward Yoon

Quick Understanding of NoSQLEdward Yoon

The evolution of web and big dataEdward Yoon

Apache hama @ Samsung SW AcademyEdward Yoon

Apache Hama 0.4Edward Yoon

Introduction of Apache Hama - 2011Edward Yoon

MongoDB introductionEdward Yoon

Monitoring and mining network traffic in cloudsEdward Yoon

Apache hama 0.2-userguideEdward Yoon

Apache HAMA: An Introduction toBulk Synchronization Parallel on HadoopEdward Yoon

Understand Of Linear AlgebraEdward Yoon

BigTable And HbaseEdward Yoon

Heart ProposalEdward Yoon

Mais de Edward Yoon (17)

(소스콘 2015 발표자료) Apache HORN, a large scale deep learning

Introduction to apache horn (incubating)

Apache Hama at Samsung Open Source Conference

K means 알고리즘을 이용한 영화배우 클러스터링

차세대하둡과 주목해야할 오픈소스

Quick Understanding of NoSQL

The evolution of web and big data

Apache hama @ Samsung SW Academy

Apache Hama 0.4

Introduction of Apache Hama - 2011

MongoDB introduction

Monitoring and mining network traffic in clouds

Apache hama 0.2-userguide

Apache HAMA: An Introduction toBulk Synchronization Parallel on Hadoop

Understand Of Linear Algebra

BigTable And Hbase

Heart Proposal

Último

Zeshan Sattar- Assessing the skill requirements and industry expectations for...itnewsafrica

Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3

A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3

Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple

The State of Passkeys with FIDO Alliance.pptxLoriGlavin3

Top 10 Hubspot Development Companies in 2024TopCSSGallery

Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen

Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada

Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA

Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica

Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3

Connecting the Dots for Information Discovery.pdfNeo4j

So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda

[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra

Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani

QCon London: Mastering long-running processes in modern architecturesBernd Ruecker

React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech

Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica

Usage case of HBase for real-time application

1. A usage case of HBase for real-time application Edward J.Yoon

2. udanax.org • URL shortener service • Provides HTTP, JSON and RESTful API • Used for some twitter clients and a number of web sites • Rows are increasing as almost 30 per second • Runs on 5 nodes HBase cluster

3. Block Diagram Clients ... Apache Web Server Tomcat 1 Tomcat 2 ... HBase Hadoop HDFS

4. HBase Table Schema row longURL: creator: clickLog: longURL: creator: clickLog:{referrer url} 3XqT <LONG_URL> <IP address> <IP address> ... longURL: 3xqS ... ... <LONG_URL> ... ...

5. MR and Hama BSP • Information-Flow Analysis • Web Structure Mining • User propensity Analysis

6. Hadoop Scheduler • FIFO: Queue-based scheduler • FSS: Fair share scheduler

7. fair share scheduler • Group jobs into “pools” • Assign each pool a guaranteed minimum share • Divide excess capacity evenly between pools

8. fair share scheduler

Usage case of HBase for real-time application

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Usage case of HBase for real-time application

Semelhante a Usage case of HBase for real-time application (20)

Mais de Edward Yoon

Mais de Edward Yoon (17)

Último

Último (20)

Usage case of HBase for real-time application