1. Don’t Let Kafka Be A Cluster:
Kafka Chaos Experimentation
Justin Fetherolf

2. Welcome!
VERICA | CONTINUOUS VERIFICATION
● What is Chaos Engineering?
● Why should we care?
● Experiment design
● Experiment results
● What’s next?

3. What is Chaos
Engineering?
VERICA | CONTINUOUS VERIFICATION

4. What does it all mean?
● Chaos Engineering
○ “the facilitation of experiments to uncover systemic weaknesses.”
● Experiment
○ “an operation or procedure carried out under controlled conditions in
order to discover an unknown effect, to test or establish a hypothesis, or
to illustrate a known law”
● Use experiments to create new knowledge
○ Tests make assertions about known properties
● Experiments verify behavior; not validate
VERICA | CONTINUOUS VERIFICATION

5. Chaos Engineering Principals
● Define “steady state”
● Form a hypothesis
● Introduce variables
● Attempt to disprove hypothesis
VERICA | CONTINUOUS VERIFICATION

6. Advanced Principles
● Build hypothesis around steady-state behavior
● Vary real-world events
● Run experiments in production
● Automate experiments to run continuously
● Minimize blast radius
VERICA | CONTINUOUS VERIFICATION

7. Why should
we care about
Chaos Engineering?
VERICA | CONTINUOUS VERIFICATION

8. Complex Systems
● Businesses require capabilities/properties/features
● Requires complexity from systems
● Can’t avoid complexity
● Embrace and navigate complexity
● As complexity increases, can’t maintain mental
model
VERICA | CONTINUOUS VERIFICATION

9. ● Kafka sits at the core of our businesses
● Kafka is a complex system
● More complex systems built on top of Kafka
● Cloud infrastructure isn’t always what we expect
● Know the safety margins of our systems
VERICA | CONTINUOUS VERIFICATION
Chaofka?

10. Our Kafka
Experiment
VERICA | CONTINUOUS VERIFICATION

11. Steady State
● Cluster
○ 5-node EKS w/ 1 broker per node - 5-broker Kafka cluster
○ t2.xlarge instance types
■ “moderate” network - 83.4 - 107.3 MiB/s
■ 20 GB “gp2” EBS volumes - 128 MiB/s
○ Metrics to Prometheus/Grafana
● Batch style workload; ~3 min @ 2.5 MiB/s every 5 min
○ ~5 million messages produced and consumed
○ 3 partitions, 3 replicas
VERICA | CONTINUOUS VERIFICATION

12. Steady State Metrics
VERICA | CONTINUOUS VERIFICATION

13. Hypothesis
“As the load on the Kafka cluster increases, the standard workload
can continue to successfully process each batch of messages
before the next batch begins.”
● How do we measure this?
○ Monitoring
■ Message/data rates
■ CPU/Memory/Net/Disk usage
○ Application status
VERICA | CONTINUOUS VERIFICATION

14. Introducing Variables
● How do we increase load? Enter Horus!
○ Scalable & configurable
○ Safety features
■ Halting could be triggered by
● Cluster or client metrics
● Other conditions
● Manual intervention
VERICA | CONTINUOUS VERIFICATION

15. Load Scaling Configuration
VERICA | CONTINUOUS VERIFICATION
● 4 distinct, increasing client sets; 15 minutes each
● 5 partition, 5 replica topic
● 10 - 40 producers; 10 step
○ 500 msg/s; 1024 byte/msg
● 7 consumer groups; 3 consumers each
● 4.88 - 19.5 MiB/s total production traffic
● 34.18 - 136.72 MiB/s total consumer traffic
● Increased replication traffic

16. Experiment Results
VERICA | CONTINUOUS VERIFICATION

17. What’s Next?
VERICA | CONTINUOUS VERIFICATION

18. The Future!
● Context sensitive
○ One size does not fit all
● Start small
● Start in non-production environment
● Minimize blast radius
● Unleash the Chaos!
VERICA | CONTINUOUS VERIFICATION

19. References and Resources
● Rosenthal, Casey and Jones, Nora. Chaos Engineering: System Resiliency in
Practice. 1st ed., O’Reilly, 2020.
● Hausmann, Steffen. “Best practices for right-sizing your Apache Kafka
clusters to optimize performance and cost.” AWS Big Data Blog, 17 Mar. 2022,
https://aws.amazon.com/blogs/big-data/best-practices-for-right-sizing-your-
apache-kafka-clusters-to-optimize-performance-and-cost/
● https://principlesofchaos.org/
● https://www.verica.io
● https://www.thevoid.community/
VERICA | CONTINUOUS VERIFICATION

20. Justin Fetherolf
Sr. Software Engineer
https://www.verica.io