The main topic of slides is building high availability high throughput system for receiveing and saving different kind of information with horizontal scalling possibility using HBase, Flume and Grizzly hosted on Amazon EC2 low cost instances. Talk describes HBase HA cluster setup process with useful hints and EC2 pitfalls, Flume setup process with providing comparasion between standalone and embedded Flume versions and show difference and usecases of both versions. A lot of attention payed to Flume2Hbase streaming features with tweaks and different approaches for speeding up this process.

1. 1
How to manage 10K+ request per second in HA
manner
● Moisieienko Valerii
● JeeConf 2015

2. 2
Initial System Requirements
● High throughput
(3K+ RPS 50/50 write/read)
● Scalability
(5-10...50 instances)
● High availability
(fall is unacceptable)
● Low cost
(~ 500$/month)

3. 3
Low Cost?!

4. 4
Solution
E
L
B
E
L
B
HTTPHTTP
Amazon EC2
FLUMEFLUME GRIZZLYGRIZZLY
HBASE

5. 5
EC2: Instances
HDFS cluster Endpoint cluster
vCPU 2 2
RAM 8 4
Storage type EBS (Magnetic) Instance-based
Count 3 2

6. 6
Let's go deep: HDFS
1200
1200
1500
3072
512
512Server1,Heap(MB)
1200
1200
3072
512
2100
Server2,Heap(MB)
1200
1500
3072
512
1800
Server3,Heap(MB)
HDFS Namenode (active)
HDFS Namenode (backup)
HDFS Datanode
Other/Free

7. 7
Let's go deep: HBase
1200
1200
1500
3072
512
512Server1,Heap(MB)
1200
1200
3072
512
2100
Server2,Heap(MB)
1200
1500
3072
512
1800
Server3,Heap(MB)
HBase Master (active)
HBase Master (backup)
HBase Regionserver
Zookeeper
Other/Free

8. 8
HBase: GC
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:CMSInitiatingOccupancyFraction=60
-XX:NewSize=64m
-XX:MaxNewSize=64m

9. 9
HBase: Compression
● Around 40% safe data occupied space
● create 'table_test',{NAME =>
'family_test',VERSIONS => '1',
COMPRESSION => 'SNAPPY'}
0
20
40
60
80
100
120
Without compression Snappy
GB

10. 10
Flume:Overview
SOURCESOURCECHANNEL
FLUMEFLUME
HBASE
NetcatMemoryAsyncHBaseSink
SINKSINK

11. 11
Flume: HBase Sink Configuration
agent.sinks.h1.type =
org.apache.flume.sink.hbase.AsyncHBaseSink
agent.sinks.h1.zookeeperQuorum =
server1,server2,server3
agent.sinks.h1.channel = m1
agent.sinks.h1.table = users
agent.sinks.h1.columnFamily = data
agent.sinks.h1.batchSize = 100
agent.sinks.h1.coalesceIncrements = true
agent.sinks.h1.enableWal = true
agent.sinks.h1.serializer = ? extends
org.apache.flume.sink.hbase.AsyncHbaseEventSeri
alizer

12. 12
Flume: AsyncHbaseEventSerializer
void initialize(byte[] table, byte[] cf);
void setEvent(Event event);
List<PutRequest> getActions();
List<AtomicIncrementRequest> getIncrements();
void cleanUp();

13. 13
AWS: VPC
● IP control
● Elastic IP permanency
● Internal DNS via Route 53
● Security

14. 14
AWS: Storage Type Options
Pro Contra
Instance-based
●Direct hardware attach
●No network costs
●Fast
●Fixed volume, based
on instance type
●Erase in case of
instance reboot
EBS
●Volume flexible
configuration
●Magnetic/SSD choice
●Persistent
●Instance independent
●Pay only for occupied
volume
●Network attach
●Pay for I/O requests
S3
●Fast via Amazon API
●Cheap
●Cloud storage
●Very slow when
instance attached

15. 15
AWS: EBS
Pro Contra
Instance-based
●Direct hardware attach
●No network costs
●Fast
●Fixed volume, based
on instance type
●Erase in case of
instance reboot
EBS
●Volume flexible
configuration
●Magnetic/SSD choice
●Persistent
●Instance independent
●Pay only for occupied
volume
●Network attach
●Pay for I/O requests
S3
●Fast via Amazon API
●Cheap
●Cloud storage
●Very slow when
instance attached

16. 16
Results 1
Avg Max
Write RPS 4K 6K
Read RPS 2K 3K
Read response time 15 ms 30 ms
Costs: ~400 $/month

17. 17
That's all?!

18. 18
Load
● Traffic through endpoint x2
● HBase data volume x5
● HBase request count x2

19. 19
Major Compaction
2015-03-28 11:11:15,894 INFO org.apache.hadoop.hbase.regionserver.Store:
Completed major compaction of 3 file(s) in data of users,4012ea33-46b9-
42cd-9094-f2a939991fbf,1421554452486.c9e5790f5b45f4306fe860a5ec5480d7. into
136b805bebbe44319f032b7d6d89a5dc, size=331.7 M; total size for store is
331.7 M
2015-03-28 11:11:15,894 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest:
completed compaction: regionName=users,4012ea33-46b9-42cd-9094-
f2a939991fbf,1421554452486.c9e5790f5b45f4306fe860a5ec5480d7.,
storeName=data, fileCount=3, fileSize=334.2 M, priority=4,
time=181415339541360; duration=17sec

20. 20
Major Compaction: jitter
● Default: 0.3-0.5 (depends on version)
● 14 HRegions on 3 servers → ~2-3
simultaneous major compactions per day
<property>
<name>
hbase.hregion.majorcompaction.jitter
</name>
<value>0.8</value>
</property>

21. 21
Major Compaction: IO
● ~ 40% CPU IO wait time during compactions

22. 22
Major Compaction: Magnetic vs GP2
Magnetic GP2(SSD)
IOPS ~200 ~1300
Cost 0,05$/GB 0,10$/GB

23. 23
HBase: Read Performance Tweaks
●
hbase-site.xml
<property>
<name>hbase.regionserver.handler.count</name>
<value>30</value>
</property>
<property>
<name>hbase.regionserver.global.memstore.lowerLimit</name>
<value>0.45</value>
</property>
<property>
<name>hbase.regionserver.global.memstore.upperLimit</name>
<value>0.55</value>
</property>
● hdfs-site.xml
<property>
<name>dfs.datanode.handler.count</name>
<value>30</value>
</property>

24. 24
Flume: Standalone
GRIZZLY
ENDPOINT HTTPHTTPPUT
OBJECT
PUT
OBJECT FLUMEFLUME JSONJSON
HBASE

25. 25
Flume: Go Embedded
GRIZZLY
ENDPOINT
DATA
OBJECT
DATA
OBJECT
EMBEDDED
FLUME
EMBEDDED
FLUME
PUT
OBJECT
PUT
OBJECT
HBASE
HTTPHTTP

26. 26
Flume: Standalone vs Embedded
Pro Contra
Flume Standalone
●No / few code
●Variety data sources
●Scalability
●Data transformation
●Network overhead
Flume Embedded
●No network overhead
●Less data
transformation
●Full process control
●A lot of code
●Complexity
●Own data source
handling

27. 27
Final Solution
E
L
B
E
L
B
HTTPHTTP
Amazon EC2
GRIZZLY+EMBEDDED FLUME
HBASE

28. 28
Results 2
Avg Max
Write RPS 7K 9K
Read RPS 3K 5K
Read response time 8 ms 27 ms
Costs: ~420 $/month

29. 29
Conclusions
●
AWS: VPC + EBS is a great deal for HDFS/HBase
cluster
●
Don't use big NewSize for HBase
●
Use Snappy
●
Use async batch puts where possible
●
Use SSD for HBase data
●
Remember about HBase compaction
●
Embedded Flume could be a good solution

30. 30
Thank you for attention
Questions?
valeramoiseenko@gmail.com