How Teads scale with Apache Cassandra. Internet scale means tons of data, read heavy workload, massive data ingestion and low latency. The French AdTech company Teads uses Cassandra massively, a reliable and performant Open Source database. Spawning Cassandra nodes in AWS is a piece of cake with Terraform and Chef.

1. Cassandra
To Infinity And Beyond
Romain Hardouin
Senior Cloud Infrastructure Engineer
The Global Media Platform

2. Cassandra @ Teads
About Teads
Workload
Infrastructure
Ops
Monitoring & Alerting
Data model
Why a fork?

3. About Teads

5. Ad Slot Available
Ad Available
Video Start
User click
Video Complete

6. French AdTech
AWS / GCP
Scala / JS / Go
Machine learning
Docker / CoreOS
Terraform / Chef / Debian
Cassandra / Kafka / MySQL / Redis
Spark / Flink

7. Workload

8. Workload
Different kinds of workloads
but all of them are latency sensitive
Internet scale
Massive amount of data ingested from partners
We also create lots of data by ourselves
No more analytics
● Tons of business critical counters
● Time series
● TTL, TTL, TTL

9. 3 years ago
1 million qps
Write heavy
Analytics stack

10. Now
+2 millions qps
Read heavy…

11. Now
...but also lots of streamed SSTables
not counted as writes

12. Infrastructure

13. Gimme some figures!
250 nodes
Mostly ephemeral data
28 TB
100 billions keys
3 regions
Regional vs Worldwide
21 DCs

14. Does it scale?
Requests vs Nodes over the past year (linreg)

15. Our Stack

16. AWS instances
c5d.4xlarge
16 vCPU @ 3.00GHz
32 GB RAM
400 GB NVMe

17. Why not i3?
Production P99 read latency
i3.2xlarge

18. Why not i3?
Production P99 write latency
i3.2xlarge

19. Why not EBS?
No more EBS
Cheap storage, great for STCS
Snapshots (S3 backup)
No coupling between disks and CPU/RAM
High latency, high I/O wait
Throughput: 160 MB/s
Unsteady performances

20. Ops

21. Ops Workflow
More on https://medium.com/teads-engineering/easy-cassandra-scaling-with-terraform-chef-rundeck-9443e0375aa7
Scale out
Scale in
Any changes

22. Ops Workflow
apply

23. Ops Workflow
replace
apply
Unreachable instance
Replace

24. Repair
No incremental repair
Scheduled with Reaper
http://cassandra-reaper.io

25. Monitoring & Alerting

26. Monitoring
Overview dashboards
Advanced dashboards for troubleshooting
Alerting dashboards
Outliers
Compare a node to average
Compare 3 DCs (multi regions)

27. Monitoring
Ratio cross DCs/Clusters to grasp workloads
Examples:
● R/W Spread: ( (maxqps
-minqps
)/maxqps
)*100
● P99/95 jitter factor: ( P99 - P95 ) / P99
● Memory cached / disk ratio

28. Monitoring
YAML Configuration
- include:
bean_regex: org.apache.cassandra.metrics:type=ReadRepair,name=.*
attribute:
- Count
- include:
bean:
org.apache.cassandra.metrics:type=CommitLog,name=TotalCommitLogSize

29. Alerting
Down node
Exceptions
Commitlog size
High latency
High pendings tasks
Many hints
Clock out of sync
IO Wait
Disk space
...

30. Data model

31. KISS principle
No secondary index
No materialized views
No UDT/UDF
No SASI*

32. Tables settings
All* type of compactions are used:
STCS, LCS, TWCS
LZ4 @ 4KB almost everywhere

33. Back in 2016
Old analytics data model:
How do you repair 45TB of data
within gc grace period?

34. Why a fork?

35. Apache Cassandra version
Fork started with C*2.1
Now in C*3.0
Maybe C*4.0 directly

36. Fork motivation
1. Need to add a patch ASAP
High Blocked NTR CASSANDRA-11363
2. Why not backport interesting tickets?
3. Why not add small features/fixes?
Expose tasks queue length via JMX CASSANDRA-12758

37. Backport example
CASSANDRA-13418
Allow TWCS to ignore overlaps when
dropping fully expired sstables

38. Custom feature:
Securing a legacy cluster without any downtime
Allow to create roles prior to use
standard auth stack
● AllowAnyCredentialsAuthenticator
○ "User '{}' has been authenticated without password checking"
● UnsecureAutoLoginCassandraRoleManager
○ Allow any non-existing role to login: "Auto login of role '{}'"
○ If a role exists standard checking will be done.
● UnsecureCassandraAuthorizer
○ Authorize DML on any resource for any authenticated user.
○ Super users will be given all permissions i.e. DML, DDL and DCL.
● No spam logs

40. One-liner of the year 2016
CASSANDRA-12580: Fix Merkle tree size calculation
Paulo Motta: log2 vs ln

41. Fix Merkle tree size calculation
“How do you repair 45TB of data
within gc grace period?”
For a subset of tables:
● Before: 23 days
● With CASSANDRA-12580: 16 hours

42. Can’t upgrade?

43. Can’t upgrade?
Quick reaction
High Blocked NTR ticket has been released in 2.1 but you
need to wait after a release
Stability, reliability
Critical counters
e.g. CASSANDRA-14958

44. Questions?

45. We hire ;-)

46. Thank you!

47. References
cassandra.apache.org
thelastpickle.com
Buzz Lightyear is a character in the Toy Story franchise.
© Disney/Pixar
Back to the Future is a trademark and copyright of Universal
Studios and U-Drive Joint Venture. Licensed by Universal
Studios. All Rights Reserved.
Crown icon designed by Good Ware from Flaticon
www.chef.io
www.datadoghq.com
www.packer.io
www.rundeck.com
www.terraform.io