The Economics of Scaling Cassandra - By Alex Bordei, Techie Product Manager at Bigstep This presentation was made during the "Cassandra Summit 2014" Event, in London. We benchmarked Cassandra on a number of configurations and we show what's the scaling profile. We test Cassandra on Docker as well as Cassandra's In-memory feature. Follow Alex on Twitter: @alexandrubordei Bigstep on Twitter: @BigStepInc If you have any questions, let us know at hello@bigstep.com and we'll do our best to answer. Stay informed: http://blog.bigstep.com/

8. Benchmark setup
Centos 6.5, Datastax 4.5.3, jmeter with custom
sampler using the java driver
/dev/shm (100G), HP on, -XX:+UseLargePages,
Disabled THP, IRQBalance

9. Scaling Horizontally - Latency
19
18
16
10 10
9
7
15
7
8
14
7
25
20
15
10
5
0
INSERT AVG response time (us) SELECT AVG response time (us) UPDATE AVG response time (us)
Average response time (ms)
smaller is better
1 node 2 nodes 3 nodes 4 Nodes

10. Scaling Horizontally - Throughput
202 193
90 96
168
151
106
120
252
223
130
249
325
260
195
130
65
0
INSERT throughput (k) SELECT throughput (k) UPDATE throughput (k)
KReq/sec - Bigger is Better
1 node 2 nodes 3 nodes 4 Nodes

11. 52 53
70
90
96
171
180
135
90
45
0
INSERT SELECT UPDATE
kReq/s - bigger is better
FMCI 4.32 FMCI 20.192
Scaling Vertically
33
40
19 18 19
10
50
40
30
20
10
0
INSERT SELECT UPDATE
Average Response time (ms)
Smaller is Better
FMCI 4.32 FMCI 20.192

12. Scaling Economics - Math
The score was computed by comparing:
Response times against the slowest
Number of requests/second against the fastest

13. Datastax’s Scaling Economics
2.3
1.5
1.0
0.4
0.3
3.0
2.4
1.8
1.2
0.6
0.0
Price-to-performance-ratio (bigger
is better)
1 node -FMCI 4.32 1 node 2 nodes 3 nodes 4 Nodes

14. Why?
• Amdhal’s Law
• Hardware prices

15. Amdhal’s Law

16. 80000.
60000.
40000.
20000.
WHAT IT IS WHAT IT SHOULD BE
8,855 10,493
Performance relative to price
15,825 14,638
17,304
22,249 23,963
3,918
20,986
26,505
35,107
53,010
70,215
0.
1x E3-
1230v2
1x E5-
2630v2
2x E5-
2630v2
1x E5-
2670v2
1x E5-
2690v2
2x E5-
2670v2
2x E5-
2690v2
Performance (higher is better)
Configuration
Specs PRICE ($) CPUMARK Est.
CPUMARK
1x E3-1230v2 4 cores,
3.3Ghz
$230.00 8855 3918
1x E5-2630v2 6 cores,
2.6Ghz
$616.00 10493 10493
2x E5-2630v2 2x6 cores,
2.6Ghz
$1232.00 15825 20986
1x E5-2670v2 8 cores,
2.6Ghz
$1556.00 14638 26505
1x E5-2690v2 10 cores,
3Ghz
$2061.00 17304 35107
2x E5-2670v2 2x8 cores,
2.6Ghz
$3112.00 22249 53010
2x E5-2690v2 2x10 cores,
3Ghz
$4122.00 23963 70215
CPU Prices

17. Money spending efficiency

18. 3024000.000s
2592000.000s
2160000.000s
1728000.000s
1296000.000s
864000.000s
432000.000s
0.000s
Native Virtual
sysbench memory 1TB read (1M
bs), write total time
518400.000s
432000.000s
345600.000s
259200.000s
172800.000s
86400.000s
0.000s
Native Virtual
sysbench multi-threading
performance
Virtualisation vs Native

19. Virtual Memory
Source:VIRTUAL MEMORY SYSTEMS AND TLB STRUCTURES Univ. Maryland 2001

20. Memory address translation with and without a TLB
Virtual Address Virtual Address
Physical Address Physical Address
Source:VIRTUAL MEMORY SYSTEMS AND TLB STRUCTURES Univ. Maryland 2001

21. TLB: Translation Lookaside Buffers
• TLB: Translation Lookaside Buffers
• Memory pointers in OS = address in virtual memory not real memory, need an offset to get to the real
memory. Offset needs to be calculated (and this is very expensive) so it is cached in TLB.
• TLB miss normally=150 cycles
• Hardware assisted virtualisation makes normal translation faster in VMs but introduces high penalty on
TLB miss.

22. TLB Misses
Source: “Memory System Characterization of Big Data Workloads” by Martin Dimitrov et al. - Intel Corp. [2013]

23. Centos 6.5, Datastax 4.5.3, jmeter
docker run -m 16G -d --privileged=true
Docker setup

24. Docker vs Native - Latency
19 18
10
21
19
11
20
26
13
40
30
28
50
40
30
20
10
0
INSERT SELECT UPDATE
Average Response Time (ms) -
Smaller Is Better
1 Node native 1 Node Native 1 docker container 1 node native with 2 docker containers 1 native with 4 docker containers

25. Docker vs Native - Throughput
90
96
168
82
92
149
78
68
81
45
60 56
180
135
90
45
0
INSERT SELECT UPDATE
KReq/s - bigger is better
1 Node native 1 Node Native 1 docker container 1 node native with 2 docker containers 1 native with 4 docker containers