熊嘉男 Track 1: Internals https://open.mi.com/conference/hbasecon-asia-2019 THE COMMUNITY EVENT FOR APACHE HBASE™ July 20th, 2019 - Sheraton Hotel, Beijing, China https://hbase.apache.org/hbaseconasia-2019/

1. THE COMMUNITY EVENT FOR
APACHE HBASE™

2. BDS: A data synchronization
platform for HBase
熊嘉男（侧⽥田）
Ali-HBase 数据链路路负责⼈人

3. Requirement

4. • HBase support cross-version migration without downtime?
• HBase support data backup to OSS or other storage?
• HBase support replicate incremental data to MQ,ES,Solr?
• Replicate incremental data from RDS to HBase?
• HBase data can be archived to Spark cluster for offline analysis?
• HBase High Availability
…….

5. Challenges

6. • Table Structure transformation
• Real-time data replication
• Client double write
• HBase Replication
• Full data migration
• DataX
• CopyTable
• Create Snapshot & Export Snapshot
• Data consistency verification
HBase clusters Migration
• Cross-Version Migration
compatibility issues
• Impact on Business
• Lack of integrated solutions
Migration Step Defect

7. • Heterogeneous full data migration
• DataX
• Sqoop
• Heterogeneous Real-time Data Replication
• HBase Real-time Data export
• Custom Replication Endpoint
• Custom Replication Sink
Heterogeneous Data Transmission

8. BDS

9. &
&
• Master & Slave
• Stateless Slave
• Plugin-in mode
• Higher scalability and better performance
High-Level Architecture

10. Technical Detail

11. HBase full data migration
3
3
.
3
3
.
.
. 3
. 3
. 3
. 3
1 3
35
3 2 4 3
5
3 4 3
2

12. HBase full data migration
• Avoid the impact on business
• Only access HDFS
• Dynamic migration rate
• Decoupled from HBase
• One-click migration
• Create table automaticlly
• Perceive changes in region
• Perceive HFiles compaction
• Efficient
• 100MB/s (single node)
• Higher scalability

13. Data localization rate
DataNode1 DataNode2
HFile HFile
RegionServer
Region
Local read remote read
• Data migration takes the issue of
data localization rates into account
• Avoid low localization rate after data
migration

14. File split
HFile1
HFile2
HFile3
HFile4
HFile1
HFile2-1
HFile2-2
HFile3
HFile4
Split
Region1
Region2
Load • Migration will split HFiles
according to the partitions of the
original and target tables
• Increase the speed of bulkload

15. HBase Real-time Replication
&
&

16. Data pipeline
352 1162 5 3
4
• Using RingBuffer as a queue
• AckQueue maintains offset
• Write throughput support dynamic configuration

17. Impact on business
4
43
4
43
2 2
43
43
2
2
2 43 4 2 43 31
2 43 4 2 43 31
• Read and write affect data replication
HBASE Replication BDS
• Decoupled from HBase
• Only access HDFS
• Data Replication is not affected by HBase crash

18. Hotspot
4
43
4
43
2 2
43
43
2
2
2 43 4 2 43 31
2 43 4 2 43 31
2
3 2
1
1
1
HBASE Replication BDS
• Hotspot • Round robin scheduling

19. Replication backlog
2
3 2
1
1
1
BDS
• Add slave nodes
• Slave throughput support
dynamic configuration
增加Worker节点并发处理理⽇日志的数量量 增加AsyncWriter并发
Add Worker nodes

20. Operation and maintenance
•BDS
•Easy to expand
•Easy to upgrade
•monitor
•alarm mechanism
•HBase Replication
•Bug fix
•No alarm
•Configuration modification and
system upgrade requires RS to
restart

21. BDS in Ali-Cloud

22. Clusters Migration

23. High Availability
--

24. Data Backup

25. Archive data to Spark
1 0

26. RDS

27. About me

28. Thanks！