(Yaniv Ranen, ZipRecruiter) Kafka Summit SF 2018 At ZipRecruiter, we connect employers with jobseekers. To do that, we record and process millions of events every day using Apache Kafka. Kafka availability is critical for our business. If Kafka is down—we are losing money. In this talk, we aim to explain the path we’ve gone through after suffering from a production incident that occurred during a rolling upgrade. This incident led us to perform chaos engineering to get our team more acquainted with Kafka’s internals and how to deal with incidents. We will share some best practices of chaos engineering Kafka and learn how to overcome failures when they appear. We will also share some of our strategies on how to perform a cluster update—rolling upgrade vs. a blue green vs. active-passive approach, including pros and cons for each method.

1. 1
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Kafka Chaos Engineering
Shlomi Hassan & Yaniv Ranen

2. 2
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
It’s a lovely day, let’s
upgrade the
production cluster

3. 3
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3
We didn’t even get a
chance to say
goodbye

4. 4
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Where did we go wrong?

5. 5
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Chaos Engineering is the discipline of experimenting on a system in
order to build confidence in the system’s capability to withstand
turbulent conditions in production.
Chaos engineering

6. 6
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Hello!
Yaniv Ranen
Shlomi Hassan

7. 7
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 77
ZipRecruiter’s mission
We actively connect
people to their next
great opportunity.

8. 8
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Logging infrastructure scheme

9. 9
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Logging Kafka cluster
Cluster spec
1) Kafka
a) 8 EC2 - m4.4xlarge
b) EBS - Io1
2) Zookeeper
a) 5 EC2 - m4.large
b) EBS - Io1
3) Cluster volume - 3TB/day
4) Broker data spread on
Multi AZ
KafkaZookeeper

10. 10
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Basic Kafka terminology

11. 11
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Basic Kafka terminology
P0
R1
P1
R1
broker1
P2
R1
P3
R1
P0
R2
P1
R2
broker2
P2
R2
P3
R2
broker3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry Replica y of partition x
Leader Replica is in bold
Active controller is in bold

12. 12
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.

13. 13
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Chaos engineering method
＋ Ask a question
＋ Write it down as a scenario in a
collaborative document
＋ Check the behaviour in a
controlled environment
＋ Document the exact commands
and output

14. 14
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 1414
Scenario #1
＋ What happens if we kill one broker?
＋ Will Kafka self-heal?

15. 15
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #1: Stopping a broker
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
P0
R2
P1
R2
Broker 2
P2
R2
P3
R2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
0Under replicated

16. 16
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #1: Stopping a broker
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
P0
R2
P1
R2
Broker 2
P2
R2
P3
R2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
54 Under replicated

17. 17
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #1: Stopping a broker
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
P0
R2
P1
R2
Broker 2
P2
R2
P3
R2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes

18. 18
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #1: Stopping a broker
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
P0
R2
P1
R2
Broker 2
P2
R2
P3
R2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes

19. 19
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #1: Stopping a broker
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
P0
R2
P1
R2
Broker 2
P2
R2
P3
R2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
0Under replicatedReassign partitions

20. 20
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 2020
Demonstration

21. 21
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 2121
Scenario #1: Stopping a broker
＋ Kafka is not self-healing
＋ Manual reassign partitions is needed

22. 22
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2
＋ What can we do to revive an
offline partition?

23. 23
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2: Reviving an offline partition
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
0offline partitions

24. 24
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2: Reviving an offline partition
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
4offline partitions

25. 25
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2: Reviving an offline partition
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
4offline partitions
Reassign partitions

26. 26
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2: Reviving an offline partition
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
4offline partitions
Reassign partitions

27. 27
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2: Reviving an offline partition
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
4offline partitions

28. 28
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2: Reviving an offline partition
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
4offline partitions

29. 29
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2: Reviving an offline partition
Broker 1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
4offline partitions

30. 30
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #2: Reviving an offline partition
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
0offline partitions
New topic
Data loss

31. 31
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3131
Scenario #2:
Reviving an offline partition
＋ Restarting broker revives the offline partition
＋ Partition reassignment doesn’t work when it
is offline
＋ Recreation of a broker
○ Data loss
○ Inconsistent state

32. 32
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #3
＋ How can we recover data
from an offline partition?

33. 33
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #3: Recovering offline partitions without data loss
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
Cluster health
0offline partitions
P0
R2
P1
R2
P2
R2
P3
R2

34. 34
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #3: Recovering offline partitions without data loss
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
P0
R2
P1
R2
P2
R2
P3
R2
Cluster health
4offline partitions

35. 35
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #3: Recovering offline partitions without data loss
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
P0
R2
P1
R2
P2
R2
P3
R2
Cluster health
4offline partitions

36. 36
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #3: Recovering offline partitions without data loss
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
"Broker 1”
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
P0
R2
P1
R2
P2
R2
P3
R2
Cluster health
4offline partitions
P0
-
P1
-
P2
-
P3
-

37. 37
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Scenario #3: Recovering offline partitions without data loss
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
“Broker 1”
ZooKeeper
Kafka cluster
Producer Consumer
Px
Ry
Replica y of partition x
EBS
volumes
P0
R2
P1
R2
P2
R2
P3
R2
Cluster health
4under replicated
P0
R2
P1
R2
P2
R2
P3
R2

38. 38
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3838
Scenario #3: How can we
recover from a lost AZ?
＋ Spawning a replacement node using the
same EBS volumes
＋ Maintain the same broker ID
＋ Consumer group offsets are kept

39. 39
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved. 3939
Our Conclusions
＋ Chaos engineering helps in gaining knowledge
＋ Kafka is not self-healing
＋ Offline partitions can be brought back using
EBS volumes
＋ Problems with the health check
＋ Using different versions of Kafka might introduce lag
＋ consider freezing the old version protocol
Inter.broker.protocol.version,log.message.format.version
＋ Upgrade active controller last

40. 40
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Deployment strategies
＋ In-place deployment might
prove risky

41. 41
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Blue green deployment
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Old cluster
P0
R2
P1
R2
P2
R2
P3
R2
Producers
Consumers

42. 42
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Disconnecting producers and consumers
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Old cluster
P0
R2
P1
R2
P2
R2
P3
R2
Producers
Consumers
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
New cluster
P0
R2
P1
R2
P2
R2
P3
R2

43. 43
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Draining the old cluster into the new one
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Old cluster
P0
R2
P1
R2
P2
R2
P3
R2
Producers
Consumers
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
New cluster
P0
R2
P1
R2
P2
R2
P3
R2
Replication
tool

44. 44
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Reconnecting the producers and consumers
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Old cluster
P0
R2
P1
R2
P2
R2
P3
R2
Producers
Consumers
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
New cluster
P0
R2
P1
R2
P2
R2
P3
R2
Replication
tool
Producers
Consumers

45. 45
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
The new cluster becomes the production one
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
New cluster
P0
R2
P1
R2
P2
R2
P3
R2
Producers
Consumers

46. 46
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Replication = data + offsets metadata

47. 47
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Replication tools
Tools name Pros Cons
Mirror Maker Open source
Relatively easy to use
Not a real mirror
UReplicator Creates a real mirror
Scalable
Open source
Maintained mainly/only by Uber
Kafka connect - Replicator Supports smart replication
Creates a real mirror
Based on Kafka Connect ecosystem
Paid solution

48. 48
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Blue green as a failover solution
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Old cluster
P0
R2
P1
R2
P2
R2
P3
R2
Producers
Consumers
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Passive cluster
P0
R2
P1
R2
P2
R2
P3
R2
Replication
tool
Producers
Consumers

49. 49
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Blue green as a failover solution
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Old cluster
P0
R2
P1
R2
P2
R2
P3
R2
Producers
Consumers
P0
R1
P1
R1
Broker 1
P2
R1
P3
R1
Broker 2
Broker 3
ZooKeeper
Passive cluster
P0
R2
P1
R2
P2
R2
P3
R2
Replication
tool
Producers
Consumers

50. 50
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Kafka Pro Tips
＋ Be part of the community
○ Join confluent Slack team
○ follow /suggest new KIP (Kafka Improvement)
○ Contribute fixes to Kafka and it’s ecosystem
＋ Use smart metrics (like health check) for better visibility
＋ Try chaos engineering at home use our Github repo
＋ Don’t stay behind
○ Use updated Kafka consumers/producers
○ Update your cluster regularly
＋ Follow confluent white papers
＋ Kafka Health check repo

51. 51
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Q & A

52. 52
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
ZipRecruiter, Inc. Proprietary and Confidential.
Copyright © 2018 ZipRecruiter, Inc. All Rights Reserved.
Thank you
https://www.linkedin.com/in/yaniv-ranen-284b003/
https://www.linkedin.com/in/shlomihassan/