SlideShare uma empresa Scribd logo
1 de 37
HBase  原理及应用 刘景龙  [email_address] 2011-08-23
Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
HBase 实现原理  -  HBase 是什么? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
HBase feature  (1) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
HBase feature  (2) s(Cont.) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
HBase feature  (3) t.)
HBase feature  (4) ,[object Object],[object Object],[object Object],[object Object]
HBase 不是什么! ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
数据模型 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
数据模型 ,[object Object],[object Object],[object Object],[object Object],[object Object]
体系结构
体系结构 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
数据索引 ,[object Object],[object Object],[object Object],[object Object],-ROOT- ZK-znode 在 -ROOT- 和 .META. region 中记录大小约 1KB , 按 region 默认大小 256MB 计算, 意味着 ROOT region 能映射 2.6 x 10 5 个 .META. region , 依次能映射总的 6.9 x 10 10   个 user region , 意味着大约是 1.8 x 10 19  (2 64 ) 字节用户数据(即 2 14 PB )
数据物理存储 ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
HBase  性能 ,[object Object],[object Object],[object Object],[object Object],Experiment (client Buffer enable) # of Tablet Servers Region server: 4 Google(1) Google(50) 1 client 4 client Random Reads 1465 1848 1212 593 Random Reads (mem) Not yet implemented Not yet implemented 10811 3745 Random Writes 12646 4889 8850 3745 Sequential Reads 4374 3995 4425 2463 Sequential Writes 12682 7204 8547 3623 scan 18898 19979 15385 10526
HBase  性能 ,[object Object],[object Object],[object Object],[object Object],Experiment (client Buffer disable) # of Tablet Servers Region server: 4 Google(1) Google(50) 1 client 4 client Random Reads 1517 1822 1212 593 Random Reads (mem) Not yet implemented Not yet implemented 10811 3745 Random Writes 3929 1984 8850 3745 Sequential Reads 4789 4309 4425 2463 Sequential Writes 3495 2205 8547 3623 scan 24775 24855 15385 10526
HBase  性能 ,[object Object],[object Object],[object Object],[object Object],Experiment (client Buffer disable and batch operation) Region server:4 / 1 client / 1 thread row num per batch 5 10 15 20 25 30 45 100 Sequential Writes 5000 5882 10000 15384 13888 15000 21428 27272
[email_address] ,[object Object],[object Object],[object Object]
[object Object]
什么是 message
业务特点
架构
send message
receive message
存储模式
为什么选择  HBase ? ,[object Object],[object Object],[object Object],[object Object]
[object Object]
什么是 puma
什么是 puma ,[object Object],[object Object],[object Object],[object Object],[object Object]
Data Freeway System
业务特点
写流程
读流程
适合场景 ,[object Object],[object Object],[object Object],[object Object]
[object Object],[object Object]
Facebook metrics system (ODS)
[object Object]

Mais conteúdo relacionado

Mais procurados

Building the Production Ready EB level Storage Product from Ceph - Dongmao Zhang
Building the Production Ready EB level Storage Product from Ceph - Dongmao ZhangBuilding the Production Ready EB level Storage Product from Ceph - Dongmao Zhang
Building the Production Ready EB level Storage Product from Ceph - Dongmao ZhangCeph Community
 
Hadoop 設定與配置
Hadoop 設定與配置Hadoop 設定與配置
Hadoop 設定與配置鳥 藍
 
Hadoop 簡介 教師 許智威
Hadoop 簡介 教師 許智威Hadoop 簡介 教師 許智威
Hadoop 簡介 教師 許智威Awei Hsu
 
110412 kningsoft-mongo db-intro-usage-in-mercury
110412 kningsoft-mongo db-intro-usage-in-mercury110412 kningsoft-mongo db-intro-usage-in-mercury
110412 kningsoft-mongo db-intro-usage-in-mercuryZoom Quiet
 
Cephfs架构解读和测试分析
Cephfs架构解读和测试分析Cephfs架构解读和测试分析
Cephfs架构解读和测试分析Yang Guanjun
 
Google LevelDB Study Discuss
Google LevelDB Study DiscussGoogle LevelDB Study Discuss
Google LevelDB Study Discusseverestsun
 
NoSQL-MongoDB介紹
NoSQL-MongoDB介紹NoSQL-MongoDB介紹
NoSQL-MongoDB介紹國昭 張
 
Leveldb background
Leveldb backgroundLeveldb background
Leveldb background宗志 陈
 
Hadoop MapReduce Task Scheduler Introduction
Hadoop MapReduce Task Scheduler IntroductionHadoop MapReduce Task Scheduler Introduction
Hadoop MapReduce Task Scheduler IntroductionHanborq Inc.
 
Distributed Data Analytics at Taobao
Distributed Data Analytics at TaobaoDistributed Data Analytics at Taobao
Distributed Data Analytics at TaobaoMin Zhou
 
大规模数据处理
大规模数据处理大规模数据处理
大规模数据处理Kay Yan
 
淘宝Hadoop数据分析实践
淘宝Hadoop数据分析实践淘宝Hadoop数据分析实践
淘宝Hadoop数据分析实践Min Zhou
 
诗檀软件 Oracle开发优化基础
诗檀软件 Oracle开发优化基础 诗檀软件 Oracle开发优化基础
诗檀软件 Oracle开发优化基础 maclean liu
 
Hadoop Map Reduce 程式設計
Hadoop Map Reduce 程式設計Hadoop Map Reduce 程式設計
Hadoop Map Reduce 程式設計Wei-Yu Chen
 
Ted yu:h base and hoya
Ted yu:h base and hoyaTed yu:h base and hoya
Ted yu:h base and hoyahdhappy001
 
Mongo db 簡介
Mongo db 簡介Mongo db 簡介
Mongo db 簡介昱劭 劉
 
Redis replication dcshi
Redis replication dcshiRedis replication dcshi
Redis replication dcshidcshi
 

Mais procurados (20)

Hadoop hive
Hadoop hiveHadoop hive
Hadoop hive
 
Building the Production Ready EB level Storage Product from Ceph - Dongmao Zhang
Building the Production Ready EB level Storage Product from Ceph - Dongmao ZhangBuilding the Production Ready EB level Storage Product from Ceph - Dongmao Zhang
Building the Production Ready EB level Storage Product from Ceph - Dongmao Zhang
 
Hadoop 設定與配置
Hadoop 設定與配置Hadoop 設定與配置
Hadoop 設定與配置
 
Level db
Level dbLevel db
Level db
 
Hadoop 簡介 教師 許智威
Hadoop 簡介 教師 許智威Hadoop 簡介 教師 許智威
Hadoop 簡介 教師 許智威
 
Mongo db 特性
Mongo db 特性Mongo db 特性
Mongo db 特性
 
110412 kningsoft-mongo db-intro-usage-in-mercury
110412 kningsoft-mongo db-intro-usage-in-mercury110412 kningsoft-mongo db-intro-usage-in-mercury
110412 kningsoft-mongo db-intro-usage-in-mercury
 
Cephfs架构解读和测试分析
Cephfs架构解读和测试分析Cephfs架构解读和测试分析
Cephfs架构解读和测试分析
 
Google LevelDB Study Discuss
Google LevelDB Study DiscussGoogle LevelDB Study Discuss
Google LevelDB Study Discuss
 
NoSQL-MongoDB介紹
NoSQL-MongoDB介紹NoSQL-MongoDB介紹
NoSQL-MongoDB介紹
 
Leveldb background
Leveldb backgroundLeveldb background
Leveldb background
 
Hadoop MapReduce Task Scheduler Introduction
Hadoop MapReduce Task Scheduler IntroductionHadoop MapReduce Task Scheduler Introduction
Hadoop MapReduce Task Scheduler Introduction
 
Distributed Data Analytics at Taobao
Distributed Data Analytics at TaobaoDistributed Data Analytics at Taobao
Distributed Data Analytics at Taobao
 
大规模数据处理
大规模数据处理大规模数据处理
大规模数据处理
 
淘宝Hadoop数据分析实践
淘宝Hadoop数据分析实践淘宝Hadoop数据分析实践
淘宝Hadoop数据分析实践
 
诗檀软件 Oracle开发优化基础
诗檀软件 Oracle开发优化基础 诗檀软件 Oracle开发优化基础
诗檀软件 Oracle开发优化基础
 
Hadoop Map Reduce 程式設計
Hadoop Map Reduce 程式設計Hadoop Map Reduce 程式設計
Hadoop Map Reduce 程式設計
 
Ted yu:h base and hoya
Ted yu:h base and hoyaTed yu:h base and hoya
Ted yu:h base and hoya
 
Mongo db 簡介
Mongo db 簡介Mongo db 簡介
Mongo db 簡介
 
Redis replication dcshi
Redis replication dcshiRedis replication dcshi
Redis replication dcshi
 

Destaque

V2 digital collaboration
V2 digital collaborationV2 digital collaboration
V2 digital collaborationMary Jo Davis
 
Bsc FOOD PRODUCTION question and answer
Bsc FOOD PRODUCTION  question and answerBsc FOOD PRODUCTION  question and answer
Bsc FOOD PRODUCTION question and answerProfessor
 
CETS 2010, Dannette Nicastro & Christine O'Malley, Participant Guide for Crea...
CETS 2010, Dannette Nicastro & Christine O'Malley, Participant Guide for Crea...CETS 2010, Dannette Nicastro & Christine O'Malley, Participant Guide for Crea...
CETS 2010, Dannette Nicastro & Christine O'Malley, Participant Guide for Crea...Chicago eLearning & Technology Showcase
 
Course module biotech_1_it
Course module biotech_1_itCourse module biotech_1_it
Course module biotech_1_itrupalidhir
 
300 puzzles-%5 bwww.placementpapers.net%5d
300 puzzles-%5 bwww.placementpapers.net%5d300 puzzles-%5 bwww.placementpapers.net%5d
300 puzzles-%5 bwww.placementpapers.net%5dPrafulla Tekriwal
 
open research practice
open research practiceopen research practice
open research practiceFabiana Kubke
 
Callture Partner Presentation
Callture Partner PresentationCallture Partner Presentation
Callture Partner PresentationCallture Inc
 
About Five Q
About Five Q About Five Q
About Five Q Five Q
 
Open access stratus 2012
Open access stratus 2012Open access stratus 2012
Open access stratus 2012Fabiana Kubke
 
Kanski ..."Lens".1aim.Net
Kanski ..."Lens".1aim.NetKanski ..."Lens".1aim.Net
Kanski ..."Lens".1aim.NetZakaria Ibrahim
 
Virgo RT from Eclipse Summit Europe 2010
Virgo RT from Eclipse Summit Europe 2010Virgo RT from Eclipse Summit Europe 2010
Virgo RT from Eclipse Summit Europe 2010Christopher Frost
 
The 2010 Political Landscape and the Influence of Indian American on Politics
The 2010 Political Landscape and the Influence of Indian American on PoliticsThe 2010 Political Landscape and the Influence of Indian American on Politics
The 2010 Political Landscape and the Influence of Indian American on PoliticsKathy Kulkarni
 

Destaque (20)

Cets 2015 lucas color matters
Cets 2015 lucas color mattersCets 2015 lucas color matters
Cets 2015 lucas color matters
 
V2 digital collaboration
V2 digital collaborationV2 digital collaboration
V2 digital collaboration
 
Bsc FOOD PRODUCTION question and answer
Bsc FOOD PRODUCTION  question and answerBsc FOOD PRODUCTION  question and answer
Bsc FOOD PRODUCTION question and answer
 
CETS 2010, Dannette Nicastro & Christine O'Malley, Participant Guide for Crea...
CETS 2010, Dannette Nicastro & Christine O'Malley, Participant Guide for Crea...CETS 2010, Dannette Nicastro & Christine O'Malley, Participant Guide for Crea...
CETS 2010, Dannette Nicastro & Christine O'Malley, Participant Guide for Crea...
 
World Cyber Games Malaysia 2012
World Cyber Games Malaysia 2012World Cyber Games Malaysia 2012
World Cyber Games Malaysia 2012
 
Herramientas publicitarias de google
Herramientas publicitarias de googleHerramientas publicitarias de google
Herramientas publicitarias de google
 
Course module biotech_1_it
Course module biotech_1_itCourse module biotech_1_it
Course module biotech_1_it
 
Bgt2
Bgt2Bgt2
Bgt2
 
300 puzzles-%5 bwww.placementpapers.net%5d
300 puzzles-%5 bwww.placementpapers.net%5d300 puzzles-%5 bwww.placementpapers.net%5d
300 puzzles-%5 bwww.placementpapers.net%5d
 
open research practice
open research practiceopen research practice
open research practice
 
Callture Partner Presentation
Callture Partner PresentationCallture Partner Presentation
Callture Partner Presentation
 
Ihps 2011-brochure
Ihps 2011-brochureIhps 2011-brochure
Ihps 2011-brochure
 
About Five Q
About Five Q About Five Q
About Five Q
 
Open access stratus 2012
Open access stratus 2012Open access stratus 2012
Open access stratus 2012
 
Cets 2014 rosenheck using mobile technology
Cets 2014 rosenheck using mobile technologyCets 2014 rosenheck using mobile technology
Cets 2014 rosenheck using mobile technology
 
Kanski ..."Lens".1aim.Net
Kanski ..."Lens".1aim.NetKanski ..."Lens".1aim.Net
Kanski ..."Lens".1aim.Net
 
Cets 2014 osborn new competency model
Cets 2014 osborn new competency modelCets 2014 osborn new competency model
Cets 2014 osborn new competency model
 
Malaysia China ICT info exchange
Malaysia China ICT info exchangeMalaysia China ICT info exchange
Malaysia China ICT info exchange
 
Virgo RT from Eclipse Summit Europe 2010
Virgo RT from Eclipse Summit Europe 2010Virgo RT from Eclipse Summit Europe 2010
Virgo RT from Eclipse Summit Europe 2010
 
The 2010 Political Landscape and the Influence of Indian American on Politics
The 2010 Political Landscape and the Influence of Indian American on PoliticsThe 2010 Political Landscape and the Influence of Indian American on Politics
The 2010 Political Landscape and the Influence of Indian American on Politics
 

Semelhante a Hbase

基于hbase的实时计算框架prom(20111114)
基于hbase的实时计算框架prom(20111114)基于hbase的实时计算框架prom(20111114)
基于hbase的实时计算框架prom(20111114)yiihsia
 
构建基于Lamp的网站架构
构建基于Lamp的网站架构构建基于Lamp的网站架构
构建基于Lamp的网站架构Cosey Lee
 
Ocean base海量结构化数据存储系统 hadoop in china
Ocean base海量结构化数据存储系统 hadoop in chinaOcean base海量结构化数据存储系统 hadoop in china
Ocean base海量结构化数据存储系统 hadoop in chinaknuthocean
 
Hbase在淘宝的应用与优化 修改
Hbase在淘宝的应用与优化 修改Hbase在淘宝的应用与优化 修改
Hbase在淘宝的应用与优化 修改yp_fangdong
 
Hbase性能测试文档
Hbase性能测试文档Hbase性能测试文档
Hbase性能测试文档baggioss
 
Redis分享
Redis分享Redis分享
Redis分享yiihsia
 
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)涛 吴
 
Hacking Nginx at Taobao
Hacking Nginx at TaobaoHacking Nginx at Taobao
Hacking Nginx at TaobaoJoshua Zhu
 
Nosql三步曲
Nosql三步曲Nosql三步曲
Nosql三步曲84zhu
 
百度分布式数据库平台
百度分布式数据库平台百度分布式数据库平台
百度分布式数据库平台Wyatt Fang
 
大规模数据处理
大规模数据处理大规模数据处理
大规模数据处理airsex
 
百度分布式数据库 刘斌 Sacc2010
百度分布式数据库 刘斌 Sacc2010百度分布式数据库 刘斌 Sacc2010
百度分布式数据库 刘斌 Sacc2010Chuanying Du
 
Hbase使用hadoop分析
Hbase使用hadoop分析Hbase使用hadoop分析
Hbase使用hadoop分析baggioss
 
Alibaba数据库运维最佳实践
Alibaba数据库运维最佳实践Alibaba数据库运维最佳实践
Alibaba数据库运维最佳实践freezr
 
How to plan a hadoop cluster for testing and production environment
How to plan a hadoop cluster for testing and production environmentHow to plan a hadoop cluster for testing and production environment
How to plan a hadoop cluster for testing and production environmentAnna Yen
 
應用Ceph技術打造軟體定義儲存新局
應用Ceph技術打造軟體定義儲存新局應用Ceph技術打造軟體定義儲存新局
應用Ceph技術打造軟體定義儲存新局Alex Lau
 
Hadoop大数据实践经验
Hadoop大数据实践经验Hadoop大数据实践经验
Hadoop大数据实践经验Hanborq Inc.
 

Semelhante a Hbase (20)

基于hbase的实时计算框架prom(20111114)
基于hbase的实时计算框架prom(20111114)基于hbase的实时计算框架prom(20111114)
基于hbase的实时计算框架prom(20111114)
 
构建基于Lamp的网站架构
构建基于Lamp的网站架构构建基于Lamp的网站架构
构建基于Lamp的网站架构
 
Ocean base海量结构化数据存储系统 hadoop in china
Ocean base海量结构化数据存储系统 hadoop in chinaOcean base海量结构化数据存储系统 hadoop in china
Ocean base海量结构化数据存储系统 hadoop in china
 
Hbase在淘宝的应用与优化 修改
Hbase在淘宝的应用与优化 修改Hbase在淘宝的应用与优化 修改
Hbase在淘宝的应用与优化 修改
 
Hic2011
Hic2011Hic2011
Hic2011
 
Hbase性能测试文档
Hbase性能测试文档Hbase性能测试文档
Hbase性能测试文档
 
Redis分享
Redis分享Redis分享
Redis分享
 
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
Pegasus: Designing a Distributed Key Value System (Arch summit beijing-2016)
 
Hacking Nginx at Taobao
Hacking Nginx at TaobaoHacking Nginx at Taobao
Hacking Nginx at Taobao
 
MogileFS
MogileFSMogileFS
MogileFS
 
Nosql三步曲
Nosql三步曲Nosql三步曲
Nosql三步曲
 
百度分布式数据库平台
百度分布式数据库平台百度分布式数据库平台
百度分布式数据库平台
 
大规模数据处理
大规模数据处理大规模数据处理
大规模数据处理
 
百度分布式数据库 刘斌 Sacc2010
百度分布式数据库 刘斌 Sacc2010百度分布式数据库 刘斌 Sacc2010
百度分布式数据库 刘斌 Sacc2010
 
HBase
HBaseHBase
HBase
 
Hbase使用hadoop分析
Hbase使用hadoop分析Hbase使用hadoop分析
Hbase使用hadoop分析
 
Alibaba数据库运维最佳实践
Alibaba数据库运维最佳实践Alibaba数据库运维最佳实践
Alibaba数据库运维最佳实践
 
How to plan a hadoop cluster for testing and production environment
How to plan a hadoop cluster for testing and production environmentHow to plan a hadoop cluster for testing and production environment
How to plan a hadoop cluster for testing and production environment
 
應用Ceph技術打造軟體定義儲存新局
應用Ceph技術打造軟體定義儲存新局應用Ceph技術打造軟體定義儲存新局
應用Ceph技術打造軟體定義儲存新局
 
Hadoop大数据实践经验
Hadoop大数据实践经验Hadoop大数据实践经验
Hadoop大数据实践经验
 

Mais de baggioss

Hdfs写流程异常处理
Hdfs写流程异常处理Hdfs写流程异常处理
Hdfs写流程异常处理baggioss
 
Hadoop基线选定
Hadoop基线选定Hadoop基线选定
Hadoop基线选定baggioss
 
Hic 2011 realtime_analytics_at_facebook
Hic 2011 realtime_analytics_at_facebookHic 2011 realtime_analytics_at_facebook
Hic 2011 realtime_analytics_at_facebookbaggioss
 
[Hi c2011]building mission critical messaging system(guoqiang jerry)
[Hi c2011]building mission critical messaging system(guoqiang jerry)[Hi c2011]building mission critical messaging system(guoqiang jerry)
[Hi c2011]building mission critical messaging system(guoqiang jerry)baggioss
 
Hdfs introduction
Hdfs introductionHdfs introduction
Hdfs introductionbaggioss
 
Hdfs原理及实现
Hdfs原理及实现Hdfs原理及实现
Hdfs原理及实现baggioss
 

Mais de baggioss (8)

Hdfs写流程异常处理
Hdfs写流程异常处理Hdfs写流程异常处理
Hdfs写流程异常处理
 
Hadoop基线选定
Hadoop基线选定Hadoop基线选定
Hadoop基线选定
 
Hic 2011 realtime_analytics_at_facebook
Hic 2011 realtime_analytics_at_facebookHic 2011 realtime_analytics_at_facebook
Hic 2011 realtime_analytics_at_facebook
 
[Hi c2011]building mission critical messaging system(guoqiang jerry)
[Hi c2011]building mission critical messaging system(guoqiang jerry)[Hi c2011]building mission critical messaging system(guoqiang jerry)
[Hi c2011]building mission critical messaging system(guoqiang jerry)
 
Hdfs introduction
Hdfs introductionHdfs introduction
Hdfs introduction
 
Hdfs
HdfsHdfs
Hdfs
 
Hdfs
HdfsHdfs
Hdfs
 
Hdfs原理及实现
Hdfs原理及实现Hdfs原理及实现
Hdfs原理及实现
 

Hbase

Notas do Editor

  1. 概念视图: Table 稀疏:某个特定行关键字下的任意列可以没有值
  2. 物理视图: Region 、 Group by Column-family 为了实现数据存储的可扩展性, Table->Regions 数据不仅在 Table 一级进行分割,在 Region 级以 column-family 为单位进行物理分割。同一 column-family 的数据作为一组,在物理上是相近的,应用设计人员应该将这一特性作为 schema 设计的重要考虑
  3. HMaster 的职责: 分配 Region :在启动时分配 region ,在 region 创建、删除、增长、分裂时进行负载均衡 扫描 root/meta :确保 region 在线、删除没有引用的父 region 管理 schema 的修改、在线、离线 Admin :分发管理性的 close 、 flush 、 compact 消息 查看 ZK 中自己的 lease 和 regionserver ,以使自己清楚何时运行修复工作
  4. HMaster 的职责: 分配 Region :在启动时分配 region ,在 region 创建、删除、增长、分裂时进行负载均衡 扫描 root/meta :确保 region 在线、删除没有引用的父 region 管理 schema 的修改、在线、离线 Admin :分发管理性的 close 、 flush 、 compact 消息 查看 ZK 中自己的 lease 和 regionserver ,以使自己清楚何时运行修复工作