At Spotify we run over 100 Cassandra clusters, from small 3 node clusters to clusters with up to 100 nodes. Many of them are multi-datacenter clusters. I will talk about the challenges of having so many clusters and what tools we are using and have built for managing them. There will also be some war stories of when we have failed

1. Scaling Cassandra
in all directions
Jimmy Mårdell
yarin@spotify.com

2. Contents
‣ Intro
‣ Cassandra at Spotify
‣ Our Cassandra toolbox
‣ Sometimes you fail

3. About me
5+ years at Spotify
Tech Product Owner
Cassandra contributor and
DataStaxApache Cassandra MVP
@tcyarin

4. About Spotify
75+ million active users
20+ million paying subscribers
30+ million songs
2+ billion playlists
58 active markets
3 billion dollars paid to rightsholders

5. About Cassandra
Open source distributed database (“NoSQL”)
Decentralized
Scales linearly
Fault-tolerant
Multi data center replication

6. Cassandra at
Spotify

7. Some numbers
100+ Cassandra clusters
1400+ nodes total
500+TB data (all replicas)
Largest cluster: 2 * 45 nodes
Most reads/second/cluster: >175,000
Most writes/second/cluster: >250,000

8. What do we store in Cassandra?
User data
Playlists
Friends and artist followers
Monitoring
Personal recommendations
… and much more

10. Micro service architecture
200+ micro services
75+ autonomous teams
Teams end-to-end responsible fortheir services
No central ops team

11. Database architecture
One database per backend service
Database abstracted away
No multi-tenant databases

12. Why Cassandra?
No single point of failure
Robust datamodel
Open source
Fits well with our service architecture
Easyto upgrade

13. Our Cassandra
toolbox

14. Why do we build our own tools?
Many existing tools are too low-level (e.g. nodetool)
Limited functionality in community OpsCenter
Need tools for developers who are not DBA’s

15. Hecuba
Automates Cassandra topology changes
‣ Create new cluster
‣ Add nodes to cluster
‣ Replace broken nodes
Not open source (yet!)

16. Cassandra Reaper
Advanced anti-entropy repair scheduler
‣ Self service
‣ Takes cluster load into account
‣ Repairs small chunks ofthe ring in parallel
Open source! (https://github.com/spotify/cassandra-reaper)

17. Data import/export
Analysis and reporting done using Hadoop (2000+ nodes)
Bulk copy data to/from HDFS
‣ Export using cass2hdfs
‣ Import using hdfs2cass

18. Sometimes you fail

19. Running out of disk
What was the symptom?
Alerts that disk was almost full on several
Cassandra hosts used by our monitoring
system

20. Running out of disk
What had happened?
Compactions require temporary space
Cluster received a magnitude more writes
Compactions didn’t keep up
Expansions didn’t keep up

21. Running out of disk
What did we learn?
Running out of disk is horribly painful
Detect request pattern changes early
Avoid multi tenancy
Don’t tryto expand clusterwhen under heavy load
Use SSD’s

22. Summary
Manytears but…
We have never ever lost data with Cassandra!

23. Questions?