A discussion of the recent work to transition Cassandra from its naive 1-partition-per-node distribution, to a proper virtual nodes implementation.

1. #Cassandra13
Rethinking Topology in Cassandra
Cassandra Summit
June 11, 2013
Eric Evans
eevans@opennms.com
@jericevans

2. #Cassandra13
DHT 101

3. #Cassandra13
DHT 101
partitioning
AZ

4. #Cassandra13
DHT 101
partitioning
AZ
BY
C

5. #Cassandra13
DHT 101
partitioning
AZ
BY
C
Key = Aaa

6. #Cassandra13
DHT 101
replica placement
AZ
BY
C
Key = Aaa

7. #Cassandra13
DHT 101
consistency
Consistency
Availability
Partition tolerance

8. #Cassandra13
DHT 101
scenario: consistency level = one
A
?
?
W

9. #Cassandra13
DHT 101
scenario: consistency level = all
A
?
?
R

10. #Cassandra13
DHT 101
scenario: quorum write
A
B
?
W

11. #Cassandra13
DHT 101
scenario: quorum read
A
B
?
R

12. #Cassandra13
Awesome, yes?

13. #Cassandra13
Well...

14. #Cassandra13
Problem:
Poor request/stream distribution

15. #Cassandra13
Distribution
AZ
BY
C
M

16. #Cassandra13
Distribution
AZ
BY
C
M

17. #Cassandra13
Distribution
AZ
BY
C
M

18. #Cassandra13
Distribution
AZ
BY
C
M

19. #Cassandra13
Distribution
Z A
BY
C
M

20. #Cassandra13
Distribution
A
BY
C
M
A1Z

21. #Cassandra13
Distribution
A
BY
C
M
A1Z

22. #Cassandra13
Distribution
A
BY
C
M
A1Z

23. #Cassandra13
Problem:
Poor data distribution

24. #Cassandra13
Distribution
A
BD
C

25. #Cassandra13
Distribution
A
BD
C
E

26. #Cassandra13
Distribution
E
A
D B
C

27. #Cassandra13
Distribution
E
A
D B
C

28. #Cassandra13
Distribution
A
BD
C
H E
FG

29. #Cassandra13
Distribution
A
BD
C
H E
FG

30. #Cassandra13
Virtual Nodes

31. #Cassandra13
In a nutshell...

32. #Cassandra13
Benefits
● Operationally simpler (no token management)
● Better distribution of load
● Concurrent streaming (all hosts)
● Smaller partitions mean greater reliability
● Better supports heterogeneous hardware

33. #Cassandra13
Strategies
● Automatic sharding
● Fixed partition assignment
● Random token assignment

34. #Cassandra13
Strategy
automatic sharding
● Partitions are split when data exceeds a threshold
● Newly created partitions are relocated to a host with less data
● Similar to Bigtable, or Mongo auto-sharding

35. #Cassandra13
Strategy
fixed partition assignment
● Namespace divided into Q evenly-sized partitions
● Q/N partitions assigned per host (where N is number of hosts)
● Joining hosts “steal” partitions evenly from existing hosts
● Used by Dynamo/Voldemort (“strategy 3” in Dynamo paper)

36. #Cassandra13
Strategy
random token assignment
● Each host assigned T random tokens
● T random tokens generated for joining hosts; New tokens divide
existing ranges
● Similar to libketama; Identical to Classic Cassandra when T=1

37. #Cassandra13
Considerations
1.Number of partitions
2.Partition size
3.How 1 changes with more nodes and data
4.How 2 changes with more nodes and data

38. #Cassandra13
Evaluating
Strategy No. Partitions Partition size
Random O(N) O(B/N)
Fixed O(1) O(B)
Auto-sharding O(B) O(1)

39. #Cassandra13
Evaluating
Automatic sharding
● Partition size is constant (great)
● Number of partitions scales linearly with data size (bad)

40. #Cassandra13
Evaluating
Fixed partition assignment
● Number of partitions is constant (good)
● Partition size scales linearly with data size (bad)
● Greater operational complexity (bad)

41. #Cassandra13
Evaluating
Random token assignment
● Number of partitions scales linearly with number of hosts (OK)
● Partition size increases with more data; Decreases with more
hosts (good)

42. #Cassandra13
Evaluating
● Automatic sharding
● Fixed partition assignment
● Random token assignment

43. #Cassandra13
Cassandra

44. #Cassandra13
Configuration
conf/cassandra.yaml
# Comma separated list of tokens, (new
# installs only).
initial_token:<token>,<token>,<token>
or
# Number of tokens to generate.
num_tokens: 256

45. #Cassandra13
Configuration
nodetool info
Token : (invoke with -T/--tokens to see all 256 tokens)
ID : 6a8dc22c-1f37-473f-8f7e-47742f4b83a5
Gossip active : true
Thrift active : true
Load : 42.92 MB
Generation No : 1370016307
Uptime (seconds) : 221
Heap Memory (MB) : 998.72 / 1886.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : size 1128 (bytes), capacity 98566144 (bytes), 42 hits, 54 re...
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN ...

46. #Cassandra13
Configuration
nodetool ring
Datacenter: datacenter1
==========
Replicas: 0
Address Rack Status State Load Owns Token
3074457345618258602
127.0.0.1 rack1 Up Normal 42.92 MB 33.33% -9223372036854775808
127.0.0.1 rack1 Up Normal 42.92 MB 33.33% 3098476543630901247
127.0.0.1 rack1 Up Normal 42.92 MB 33.33% 3122495741643543892
127.0.0.1 rack1 Up Normal 42.92 MB 33.33% 3146514939656186537
127.0.0.1 rack1 Up Normal 42.92 MB 33.33% 3170534137668829183
127.0.0.1 rack1 Up Normal 42.92 MB 33.33% 3194553335681471828
127.0.0.1 rack1 Up Normal 42.92 MB 33.33% 321857253369411447
127.0.0.1 rack1 Up Normal 42.92 MB 33.33% 3242591731706757118
...

47. #Cassandra13
Configuration
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 42.92 MB 256 33.3% 6a8dc22c-1f37-473f-8f7e-47742f4b83a5 rack1
UN 127.0.0.2 60.17 MB 256 33.3% 26263a2b-768e-4a79-8d41-3624a14b13a8 rack1
UN 127.0.0.3 56.85 MB 256 33.3% 5b3e208f-6d36-4c7b-b2bb-b7c476a1af66 rack1

48. #Cassandra13
Configuration
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 42.92 MB 256 33.3% 6a8dc22c-1f37-473f-8f7e-47742f4b83a5 rack1
UN 127.0.0.2 60.17 MB 256 33.3% 26263a2b-768e-4a79-8d41-3624a14b13a8 rack1
UN 127.0.0.3 56.85 MB 256 33.3% 5b3e208f-6d36-4c7b-b2bb-b7c476a1af66 rack1

49. #Cassandra13
Configuration
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 127.0.0.1 42.92 MB 256 33.3% 6a8dc22c-1f37-473f-8f7e-47742f4b83a5 rack1
UN 127.0.0.2 60.17 MB 256 33.3% 26263a2b-768e-4a79-8d41-3624a14b13a8 rack1
UN 127.0.0.3 56.85 MB 256 33.3% 5b3e208f-6d36-4c7b-b2bb-b7c476a1af66 rack1

50. #Cassandra13
Migration
A
BD

51. #Cassandra13
Migration
edit conf/cassandra.yaml and restart
# Number of tokens to generate.
num_tokens: 256

52. #Cassandra13
Migration
convert to T contiguous tokens in existing ranges
B
AAAAAA
AAA
A
A
A
A
AAA
C
A
B

53. #Cassandra13
Migration
shuffle
B
AAAAAA
AAA
A
A
A
A
AAA
C
A
B

54. #Cassandra13
Shuffle
● Range transfers are queued on each host
● Hosts initiate transfer to self
● Pay attention to the logs!

55. #Cassandra13
Shuffle
Usage: shuffle [options] <sub-command>
Sub-commands:
create Initialize a new shuffle operation
ls List pending relocations
clear Clear pending relocations
en[able] Enable shuffling
dis[able] Disable shuffling
Options:
-dc, --only-dc Apply only to named DC (create only)
-u, --username JMX username
-tp, --thrift-port Thrift port number (Default: 9160)
-p, --port JMX port number (Default: 7199)
-tf, --thrift-framed Enable framed transport for Thrift (Default: false)
-en, --and-enable Immediately enable shuffling (create only)
-pw, --password JMX password
-H, --help Print help information
-h, --host JMX hostname or IP address (Default: localhost)
-th, --thrift-host Thrift hostname or IP address (Default: JMX host)

56. #Cassandra13
Performance

57. #Cassandra13
removenode
Cassandra 1.2 Cassandra 1.1
0
50
100
150
200
250
300
350
400
450

58. #Cassandra13
bootstrap
Cassandra 1.2 Cassandra 1.1
0
100
200
300
400
500
600

59. #Cassandra13
The End
● Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati,
Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall
and Werner Vogels “Dynamo: Amazon’s Highly Available Key-value Store” Web.
● Low, Richard. “Improving Cassandra's uptime with virtual nodes” Web.
● Overton, Sam. “Virtual Nodes Strategies.” Web.
● Overton, Sam. “Virtual Nodes: Performance Results.” Web.
● Jones, Richard. "libketama - a consistent hashing algo for memcache clients” Web.