DeveloperWeek Management 2022 Conference Presentation https://www.developerweek.com/global/conference/management/schedule/ In the last decade, the development of modern horizontally scalable open-source Big Data technologies such as Apache Cassandra (for data storage), and Apache Kafka (for data streaming) enabled cost-effective, highly scalable, reliable, low-latency applications, and made these technologies increasingly ubiquitous. To enable reliable horizontal scalability, both Cassandra and Kafka utilize partitioning (for concurrency) and replication (for reliability and availability) across clustered servers. But building scalable applications isn’t as easy as just throwing more servers at the clusters, and unexpected speed humps are common. Consequently, you also need to understand the performance impact of partitions, replication, and clusters; monitor the correct metrics to have an end-to-end view of applications and clusters; conduct careful benchmarking, and scale and tune iteratively to take into account performance insights and optimizations. In this presentation, I will explore some of the performance goals, challenges, solutions, and results I discovered over the last 5 years building multiple realistic demonstration applications. The examples will include trade-offs with elastic Cassandra auto-scaling, scaling a Cassandra and Kafka anomaly detection application to 19 Billion checks per day, and building low-latency streaming data pipelines using Kafka Connect for multiple heterogeneous source and sink systems.

1. Scaling Open Source Big
Data Cloud Applications is
Easy/Hard
Paul Brebner
Instaclustr—Technology Evangelist
©Instaclustr Pty Limited, 2022
DeveloperWeek 10 May 2022

2. Who am I?
• Previously
• R&D in distributed systems and performance engineering.
• Last 5 years
• Technology Evangelist for Instaclustr (soon NetApp)
• 100+ Blogs, demo applications, talks
• Open Source technologies including
• Apache Cassandra, Spark, Kafka, Zookeeper
• and Redis, OpenSearch, PostgreSQL, Kubernetes,
Prometheus, OpenTracing, etc

3. Cloud Platform for Big Data
Open Source Technologies
Latest addition is
Workflows with Uber’s
Instaclustr Managed Platform
©Instaclustr Pty Limited, 2021

4. Cloud Platform for Big Data
Open Source Technologies
Latest addition is
Workflows with Uber’s
This talk focuses on Cassandra and Kafka
©Instaclustr Pty Limited, 2021

5. Scaling is Easy! Cassandra and Kafka
Homogeneous distributed clusters à horizontally scalable
www.cassandra.apache.org/_/cassandra-basics.html

6. But actually lots of moving parts
(source: http://trumpetb.net/loco/rodsf.html)

7. Complications – DCs, Racks, Nodes, Partitions,
Replication Factor, Time (for auto-scaling)
Rows have a
partition key
and are
stored in
different
partitions

8. Example 1 – Cassandra Auto-Scaling
©Instaclustr Pty Limited, 2021

9. Two Ways of Resizing Clusters
1 - Horizontal Scaling
• Add nodes, no interruption
• But scale up only (not down)
• Takes time, puts extra load on cluster as data streams to extra nodes
2 - Vertical Scaling
• Replace nodes with bigger (or smaller) node types (more/less cores)
• Scale up and down
• Takes time, temporary reduction in capacity
• Choice of how many nodes are replaced concurrently – by “node” (1 node at a
time) or by “rack” (all nodes in a rack) , or in-between

10. Cluster resizing time – by node vs. by
rack – by rack is faster but …?
Cluster = 6 nodes, 3 racks, 2 nodes per rack
By node (concurrency 1)
By rack (concurrency 2)

11. Resizing by node – capacity reduced by 1/6 total
nodes each resize operation (simplified model)

12. Resizing by rack – capacity reduced by 2/6
nodes each resize operation

13. Comparison – resize by rack faster but has
bigger capacity hit during resize

14. Observations
• If the capacity during resize is exceeded latencies will increase
• Made worse by Cassandra load balancing which assumes equal sized
nodes
• By node, more nodes in the Cluster reduces the impact of reduced cluster
capacity during resizing (some clusters have 100s of nodes) – but will take
longer
• Many of our clusters have <= 6 nodes

15. Auto-scaling model - increasing load à linear
regression over 1 hour extrapolated to future
We predict the cluster will reach
100% capacity around the 280
minute mark (220 minutes in the
future)
Extrapolated
Measured

16. Resize by Rack vs. Node - initiated in time to
prevent overloading during resize operation
Resize by rack must be initiated sooner c.f. resize by node, even thought it’s faster to resize, as it has less capacity
during resize (67% c.f. 83% of initial capacity)
By
Rack
By Node

17. Auto-scaling POC – worked!
Monitoring API
Linear Regression +
Rules
Provisioning API
Rules generalized to allow for
• scaling up and down
• resizing by any number of nodes concurrently, up to rack size

18. Example 2 – Anomaly Detection
©Instaclustr Pty Limited, 2021
JoAnn Morgan Apollo 11 Mission Control

19. Multiple technologies: Kafka,
Cassandra, Kubernetes

20. Massively Scalable Anomaly Detection
– Tuning knobs (Orange h/w, yellow s/w)
Scaling is (too) Easy!
Initially just increased h/w resources

21. But scalability not great
0
1
2
3
4
5
6
7
8
0 100 200 300 400 500 600 700
Billions
checks/day
Total Cores
Total Cores vs. Billions of checks/day (pre-tuning)

22. Tuning required! Scalability Post-tuning
0
2
4
6
8
10
12
14
16
18
20
0 100 200 300 400 500 600 700
Billions
checks/day
Total Cores
Total Cores vs. Billions of checks/day (pre-tuning)
Billions of checks/day (pre-tuning) Billions of checks/day (post-tuning)

23. Tuning – Optimize s/w resources
(red arrows)
1
2
3
1. Minimize Kafka Consumers (thread pool 1)
2. Minimize Cassandra Connections
3. Maximize Cassandra client concurrency (thread
pool 2)

24. Example 3 – What’s really going on -
behind the Kafka partitions?
©Instaclustr Pty Limited, 2021
©Instaclustr Pty
Limited 2019,
2021, 2022

25. Kafka topic partitions enable
consumer concurrency
partitions >= consumers
Partition n
Topic “Parties”
Partition 1
Producer
Partition 2
Consumer Group
Consumer
Consumer
Consumers share
work within groups
Consumer

26. High consumer/partition fan out
Can be caused by:
1 Design – many topics and/or many consumers
2 Slow consumers à need more consumers to increase
throughput

27. Kafka write architecture – partition
replication

28. Benchmarking revealed that partitions
and replication factor are the culprit
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
1 10 100 1000 10000
TPS
Partitions
Kafka Partitions vs. Throughput
Cluster: 3 nodes x 4 cores = 12 cores total
Replication Factor 3 (TPS) Replication Factor 1 (TPS)

29. Implications?
• Bigger Cluster (more nodes, bigger nodes)
• Design to minimize topics and consumers
• Optimize consumers for minimum time
• Always benchmark with many partitions
• Blame the Apache Zookeeper?
• Responsible for Kafka control
• From version 3.0 it’s being replaced by native KRaft protocol
• Not yet production ready
• May enable more partitions (but may not impact throughput)

30. Scaling is Mostly Easy!
§ Using Scalable Open Source Big Data Technologies
§ Hosted by suitable Cloud providers
§ With suitable monitoring, understanding of autoscaling
and how different software “knobs” interact, and by
scaling incrementally
© Instaclustr Pty Limited, 2022

31. www.instaclustr.com
info@instaclustr.com
@instaclustr
THANK
YOU!
For further Information see blogs www.instaclustr.com/paul-brebner/