HBASE Performane Test

www.vmcd.org
HBASE Performance test by YCSB
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Bulid YCSB From github
We use thumbtack modify version https://github.com/thumbtack-technology/ycsb
Modify pom.xml – using CDH platform version:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 maven-v4_0_0.xsd">
<modelVersion>
4.0.0</<parent>
<groupId>
com.yahoo.ycsb</<artifactId>
root</<version>
0.1.4</</parent>
<artifactId>hbase-binding</artifactId>

www.vmcd.org
<name>
HBase DB Binding</<repositories>
<repository>
<id>
central</<name>
Maven Repository Switchboard</<url>http://repo1.maven.org/maven2</url>
</repository>
<repository>
<id>maven-hadoop</id>
<name>
Hadoop Releases</<url>https://repository.cloudera.com/content/repositories/releases/</url>
</repository>
<repository>
<id>cloudera-repos</id>
<name>
Cloudera Repos</<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>
org.apache.hbase</<artifactId>hbase-client</artifactId>
<version>0.96.1.1-cdh5.0.0</version>

www.vmcd.org
</dependency>
<dependency>
<groupId>
org.apache.hadoop</<artifactId>hadoop-hdfs</artifactId>
<version>2.3.0-cdh5.0.0</version>
</dependency>
<dependency>
<groupId>
com.yahoo.ycsb</<artifactId>
core</<version>${project.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>
org.apache.maven.plugins</<artifactId>maven-assembly-plugin</artifactId>
<version>${maven.assembly.version}</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<appendAssemblyId>
false</</configuration>
<executions>

www.vmcd.org
<execution>
<phase>
package</<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Run mvn clean package – build HBase DB Binding successfully
Copy core-site.xml hbase-site.xml hdfs-site.xml to /home/hadoop/ycsb/hbase/src/main/conf
Load 50 millions data to usertable:
YCSB Client 0.1
Command line: -db com.yahoo.ycsb.db.HBaseClient -P workloads/workloada -p columnfamily=family -p recordcount=50000000 -s -threads 10 -p measurementtype=timeseries -p timeseries.granularity=2000 –load
[OVERALL], Reconnections, 0.0
[OVERALL], RunTime(ms), 823765.0
[OVERALL], Operations, 4.9999968E7
[OVERALL], Throughput(ops/sec), 60696.883213052264

www.vmcd.org
Run test scripts:
$ls -l |grep .sh
for mode in a b c d e f g h i j k do for thread in `seq 1 128` do /home/hadoop/ycsb/bin/ycsb run hbase -P workloads/workload"$mode" -p columnfamily=f1 -p table=test2 -s -threads "$thread" |tee workload"$mode"_t"$thread" done sleep 60 done
-rw-rw-r-- 1 hadoop 58901 Sep 10 23:27 workloada_t1
…
-rw-rw-r-- 1 hadoop 58901 Sep 10 23:27 workloadk_t120
-rw-rw-r-- 1 hadoop 54592 Sep 11 09:43 load50m.data

www.vmcd.org
Workload Summary
Total 11 modes to run test script on 1~128 threads
Workload all (ops/sec a-k mode)
0
50000
100000
150000
200000
250000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
mode_a
mode_b
mode_c
mode_d
mode_e
mode_f
mode_g
mode_h
mode_i
mode_j
mode_k

www.vmcd.org
Workload A AverageLatency (us) : read50% update50%
0
5000
10000
15000
20000
25000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
read_50%
update_50%

www.vmcd.org
Workload B AverageLatency (us) : read95% update5%
0
5000
10000
15000
20000
25000
30000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
read_95%
update_5%

www.vmcd.org
Workload C AverageLatency (us) : read100%
0
5000
10000
15000
20000
25000
30000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
read100%
read100%

www.vmcd.org
Workload D AverageLatency (us) : read95% insert5%
0
1000
2000
3000
4000
5000
6000
7000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
read 95%
insert5%

www.vmcd.org
Workload E AverageLatency (us) : scan95% insert5%
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
scan95%
insert5%

www.vmcd.org
Workload F AverageLatency (us) : read50% readmodify50%
0
2000
4000
6000
8000
10000
12000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
read50%
readmodify50%

www.vmcd.org
Workload G AverageLatency (us) : read5% update95%
0
2000
4000
6000
8000
10000
12000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
read5%
update95%

www.vmcd.org
Workload H AverageLatency (us) : update50% insert50%
0
500
1000
1500
2000
2500
3000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
update50%
insert50%

www.vmcd.org
Workload I AverageLatency (us) : read30% insert70%
0
2000
4000
6000
8000
10000
12000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
read30%
insert70%

www.vmcd.org
Workload J AverageLatency (us) : read20% update30% insert50%
0
2000
4000
6000
8000
10000
12000
14000
16000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
read20%
update30%
insert50%

www.vmcd.org
Workload K AverageLatency (us) : update20% insert80%
0
1000
2000
3000
4000
5000
6000
7000
8000
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
105
109
113
117
121
125
update20%
insert80%

www.vmcd.org
HBASE ENV: 3 regionservers and 10 hdfs datanodes
#hbase version 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: HBase 0.96.1.1-cdh5.0.0 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: Subversion file: ///var/lib/jenkins/workspace/generic-binary-tarball-and-maven-deploy/CDH5.0.0-Packaging-HBase-2014-03-27_22-53-27/hbase-0.96.1.1-cdh5.0.0 -r Unknown 2014-09-03 22:46:34,973 INFO [main] util.VersionInfo: Compiled by jenkins on Thu Mar 27 23:02:13 PDT 2014
export HBASE_OPTS="-Xloggc:/home/hadoop/logs/gc -XX:+UseConcMarkSweepGC -XX:PermSize=96m -XX:MaxPermSize=256m -Xss1m" export HBASE_REGIONSERVER_OPTS="$OPTS -XX:+HeapDumpOnOutOfMemoryError -Xms32g -Xmx32g -Xmn16g -XX:SurvivorRatio=2 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80" #export HBASE_REGIONSERVER_OPTS="$OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC -Xms32g -Xmx32g -Xmn12g -XX:SurvivorRatio=10 -XX:+UseConcMarkSweepGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSClassUnloadingEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:CMSFullGCsBeforeCompaction=0 -XX:+DisableExplicitGC -Dsun.net.client.defaultConnectTimeout=10000 -Dsun.net.client.defaultReadTimeout=30000" export HBASE_MASTER_OPTS="-Xms4g -Xmx4g" export HBASE_LOG_DIR=/home/hadoop/logs/hbase export HBASE_MANAGES_ZK=false

www.vmcd.org
Datanode OS system information : #cat /etc/issue CentOS release 6.5 (Final)
#cat /proc/cpuinfo |grep processor wc -l 24 #cat /proc/meminfo |more MemTotal: 99035868 kB
Mount system info: /dev/sdb1 on data1 type ext4 (rw) /dev/sdc1 on data2 type ext4 (rw) /dev/sdd1 on data3 type ext4 (rw) /dev/sde1 on data4 type ext4 (rw) /dev/sdf1 on data5 type ext4 (rw) /dev/sdg1 on data6 type ext4 (rw) /dev/sdh1 on data7 type ext4 (rw) /dev/sdi1 on data8 type ext4 (rw) /dev/sdj1 on data9 type ext4 (rw) /dev/sdk1 on data10 type ext4 (rw) /dev/sdl1 on data11 type ext4 (rw) /dev/sdm1 on data12 type ext4 (rw)
#sg_vpd /dev/sde --page=0xb1

www.vmcd.org
Block device characteristics VPD page (SBC): Nominal rotation rate: 7200 rpm : 3.5 inch
=== START OF INFORMATION SECTION === Device Model: ST2000NM0033-9ZM175 Serial Number: Z1X1YV4Q LU WWN Device Id: 5 000c50 06672b0c2 Firmware Version: SN03 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ACS-2 (revision not indicated) Local Time is: Wed Sep 3 15:53:55 2014 CST SMART support is: Available - device has SMART capability. SMART support is: Enabled

HBASE Performane Test

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to HBASE Performane Test

Similar to HBASE Performane Test (20)

More from Louis liu

More from Louis liu (20)

Recently uploaded

Recently uploaded (20)

HBASE Performane Test