Eshcar Hillel and Anastasia Braginsky (Yahoo!)
Real-time HBase application performance depends critically on the amount of I/O in the datapath. Here we’ll describe an optimization of HBase for high-churn applications that frequently insert/update/delete the same keys, such as for high-speed queuing and e-commerce.
Apache HBase, Accelerated: In-Memory Flush and Compaction
1. HBase Accelerated:
In-Memory Flush and Compaction
E s h c a r H i l l e l , A n a s t a s i a B r a g i n s k y , E d w a r d B o r t n i k o v ⎪ H B a s e C o n , S a n
F r a n c i s c o , M a y 2 4 , 2 0 1 6
3. Motivation: Dynamic Content Processing on Top of HBase
3
Real-time content processing pipelines
› Store intermediate results in persistent map
› Notification mechanism is prevalent
Storage and notifications on the same platform
Sieve – Yahoo’s real-time content management platform
Crawl
Docpro
c
Link
Analysis Queue
Crawl
schedule
Content
Queue
Links
Serving
Apache Storm
Apache HBase
4. Notification Mechanism is Like a Sliding Window
4
Small working set but not necessarily FIFO queue
Short life-cycle delete message after processing it
High-churn workload message state can be updated
Frequent scans to consume message
5. HBase Accelerated: Mission Definition
5
Goal:
Real-time performance in persistent KV-stores
How:
Use less in-memory space less I/O
6. HBase Accelerated: Two Base Ideas
6
In-Memory Compaction
Exploit redundancies in the workload to eliminate duplicates in memory
Gain is proportional to the duplicate ratio
In-Memory Index Reduction
Reduce the index memory footprint, less overhead per cell
Gain is proportional to the cell size
Prolong in-memory lifetime, before flushing to disk
Reduce amount of new files
Reduce write amplification effect (overall I/O)
Reduce retrieval latencies
8. Random writes absorbed in active segment (Cm)
When active segment is full
› Becomes immutable segment (snapshot)
› A new mutable (active) segment serves writes
› Flushed to disk, truncate WAL
On-Disk compaction reads a few files, merge-sorts them, writes back new files
9
HBase Writes
C’m
flush
memory HDFS
Cm
prepare-for-flush
Cd
WAL
write
9. 10
HBase Reads
C’m
memory HDFS
Cm
Cd
Read
Random reads from Cm or C’m or Cd (Block Cache)
When data piles-up on disk
› Hit ratio drops
› Retrieval latency up
Compaction re-writes small files into fewer bigger files
› Causes replication-related network and disk IO
Block cache
11. New compaction pipeline
› Active segment flushed to pipeline
› Pipeline segments compacted in memory
› Flush to disk only when needed
13
New Design: In-Memory Flush and Compaction
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
Block
cache
memory
12. Trade read cache (BlockCache) for write cache (compaction pipeline)
14
New Design: In-Memory Flush and Compaction
write read
cache cache
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
Block
cache
memory
13. 15
New Design: In-Memory Flush and Compaction
CPU
IO
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
Block
cache
memory
Trade read cache (BlockCache) for write cache (compaction pipeline)
More CPU cycles for less I/O
27. Exploit the Immutability of Segment after Compaction
29
Current design
› Data stored in flat buffers, index is a skip-list
› All memory allocated on-heap
New Design: Flat layout for immutable segments index
› Less overhead per cell
› Manage (allocate, store, release) data buffers off-heap
Pros
› Locality in access to index
› Reduce memory fragmentation
› Significantly reduce GC work
› Better utilization of memory and CPU
29. 31
New Design: Effective in-memory representation
C’m
flush-to-disk
memory HDFS
Cm
in-memory-flush
Cd
prepare-for-flush
Compaction
pipeline
WAL
memory
Are there redundancies
to compact?
yesno
Flatten the index –
less overhead per
cell
Flatten the index &
compact – less cells
& less overhead per
cell
34. Summary
36
Feature intended for HBase 2.0.0
New design pros over default implementation
› Predictable retrieval latency by serving (mainly) from memory
› Less compaction on disk reduces write amplification effect
› Less disk I/O and network traffic reduces load on HDFS
› New space efficient index representation
We would like to thank the reviewers
› Michael Stack, Anoop Sam John, Ramkrishna s. Vasudevan, Ted Yu