Presented by: Elliott Clark, Cloudera

1. 1
Using Metrics
How to monitor and debug Apache HBase with
metrics

2. 2
Hi! I’m Elliott
eclark@apache.org @elliottnclark Engineer @ Cloudera

3. Rules of HBase Metrics
1. Collect Metrics
2. Collect Metrics
3. Know your steady state
4. Explore
5. Correlate
3

4. Examples
• User Error
• Region Skew
• GC Pressure
• Suboptimal HBase
• Balancer not Balancing
4

5. Region Skew
5 Photo From: http://www.flickr.com/photos/64618542@N00/

6. Region Skew
6

7. Region Skew Symptoms
• Long tail latencies
• Unbalanced WALs
• Crashing region servers
7

8. Region Skew Fixes
• Hash Row Key
• Add random salt to start of Row Key
• Re-Arrange The Row Key
• Add Caching
8

9. GC Pressure
9 Photo by: http://www.flickr.com/photos/wwarby/

10. GC Pressure
10

11. GC Pressure Symptoms
• Crashing region servers
• ZK Connection Issues
• Spiky latency
11

12. GC Pressure Fixes
• Switch map reduce to Bulk Load
• Pre-Aggregate Puts
• Reduce data returned per scan#next
12

13. Bad Balancer
13 Photo By: http://www.flickr.com/photos/jimmiehomeschoolmom/

14. Bad Balancer
14

15. Bad Balancer Symptoms
• Bad latency after balancer
• Bad cache hit on some servers
15

16. Coming Soon
• Better balancer
• More per region metrics
• Renamed Metrics
• Metrics2 based metrics
16

17. Metrics Options
Cloudera Manger OpenTSDB
17
Ganglia
Ganglia Image From:http://www.flickr.com/photos/hongiiv/

18. 18