Watch this talk here: https://www.confluent.io/online-talks/everything-you-always-wanted-to-know-about-kafkas-rebalance-protocol-but-were-afraid-to-ask-on-demand Apache Kafka® is a scalable streaming platform with built-in dynamic client scaling. The elastic scale-in/scale-out feature leverages Kafka’s “rebalance protocol” that was designed in the 0.9 release and improved ever since then. The original design aims for on-prem deployments of stateless clients. However, it does not always align with modern deployment tools like Kubernetes and stateful stream processing clients, like Kafka Streams. Those shortcomings lead to two major recent improvement proposals, namely static group membership and incremental rebalancing. This talk provides a deep dive into the details of the rebalance protocol, starting from its original design in version 0.9 up to the latest improvements and future work. We discuss internal technical details, pros and cons of the existing approaches, and explain how you configure your client correctly for your use case. Additionally, we discuss configuration tradeoffs for stateless, stateful, on-prem, and containerized deployments.

1. 1
1
Everything you always wanted to know
about Kafka’s rebalance protocol but
you were afraid to ask
Matthias J. Sax | Software Engineer
@MatthiasJSax

2. What is rebalancing
about?
● Group membership
● Resource assignment
● Example: KafkaConsumer
○ Consumer group
○ Partition ownership
@MatthiasJSax
2

3. 3
3
3
@MatthiasJSax
Design Decisions
● Broker side: membership
○ JoinGroup
○ Heartbeat
○ LeaveGroup
● Client side: assignment
○ SyncGroup
○ Leader

4. 4
4
4
Rebalancing happens if
(a) a member joins/leaves the group
(b) resources need to be reassigned
@MatthiasJSax

5. 5
Let’s Rebalance
GroupCoordinator
(broker side)
heartbeat ok heartbeat ok
session.timeout.ms
heartbeat.interval.ms
C1
C2
C3
@MatthiasJSax 5

6. 6
Let’s Rebalance
GroupCoordinator
(broker side)
C1
C2
C3
C4
JoinGroup
(subscription)
rebalanceheartbeat
@MatthiasJSax
synchronization barrier
JoinGroup
(subscription)
JoinResponse
max.poll.interval.ms (consumer)
rebalance.timeout.ms (connect)
C1
C2
C3
C4
6

7. 7
Let’s Rebalance
@MatthiasJSax
rebalance
synchronization barrier
JoinGroup
(subscription)
GroupCoordinator
(broker side)
Group Leader (receives all subscriptions)
Leader
Selection
C1
C2
C3
C4
SyncGroup
JoinResponse
SyncResponse
(assignment)
6

8. 8
Group-
Coordinator
Group-
Coordinator
Group-
Coordinator
Group-
Coordinator
Brokers:
- Maintain/monitor
groups
- Store group
metadata
AbstractCoordinator
WorkerCoordinator ConsumerCoordinator
Connect API Consumer / Streams API
Range RR Sticky custom
StreamsAssignor
Clients:
- Assign resources
__consumer_offsets
@MatthiasJSax 7

9. Kafka Connect
● Cluster of workers
● Single group
● Resources:
○ Tasks
○ Configuration
@MatthiasJSax
8

10. 10
10
Kafka Streams
● Application instances / threads
● Tasks plus Standbys
○ Stateful
○ Co-partitioning
● Interactive Queries
○ Endpoint metadata
@MatthiasJSax
9

11. 11
11
11
Issues
@MatthiasJSax
10

12. 12
12
Unnecessary Rebalance
group.id=“grp”
POD
Application
member.id=1
group.id=“grp”
POD
Application
member.id=2
group.id=“grp”
POD
Application
member.id=3
GroupCoordinator
(broker side)
group.id List of member.id
“grp” 1,2,3
@MatthiasJSax 11

13. 13
13
Unnecessary Rebalance
group.id=“grp”
POD
Application
member.id=2
group.id=“grp”
POD
Application
member.id=3
GroupCoordinator
(broker side)
group.id List of member.id
“grp” 2,3
@MatthiasJSax 11

14. 14
14
Unnecessary Rebalance
group.id=“grp”
POD
Application
member.id=4
group.id=“grp”
POD
Application
member.id=2
group.id=“grp”
POD
Application
member.id=3
GroupCoordinator
(broker side)
group.id List of member.id
“grp” 2,3,4
group.id List of member.id
“grp” 2,3,4
@MatthiasJSax 11

15. 15
15
Unnecessary Rebalance
group.id=“grp”
POD
Application
member.id=4
group.id=“grp”
POD
Application
member.id=2
group.id=“grp”
POD
Application
member.id=3
Why rebalance if we know that the application is
restarted anyway?
@MatthiasJSax 11

16. 16
16
Stop-the-World Effect
Application
Application
Application
Why stop processing if partitions are reassigned
anyway?
@MatthiasJSax 12

17. Latest Improvements
● Apache Kafka 2.3 /
Confluent Platform 5.3
○ static group membership
○ incremental rebalancing in
Kafka Connect
@MatthiasJSax
13

18. 18
18
Static Group Membership
group.id=“grp”
group.instance.id=“A”
POD
Application
member.id=1
group.id=“grp”
group.instance.id=“C”
POD
Application
member.id=3
group.id=“grp”
group.instance.id=“B”
POD
Application
member.id=2
group.instance.id member.id
A 1
B 2
C 3
GroupCoordinator
(broker side)
@MatthiasJSax 14

19. 19
19
Static Group Membership
group.instance.id member.id
A 1
B 2
C 3
GroupCoordinator
(broker side)
group.id=“grp”
group.instance.id=“A”
POD
Application
member.id=1
group.id=“grp”
group.instance.id=“C”
POD
Application
member.id=3
group.id=“grp”
group.instance.id=“B”
POD
Application
member.id=2
@MatthiasJSax 14

20. 20
20
Static Group Membership
group.instance.id member.id
A 1
B 2
C 3
GroupCoordinator
(broker side)
group.id=“grp”
group.instance.id=“C”
POD
Application
member.id=3
group.id=“grp”
group.instance.id=“B”
POD
Application
member.id=2
@MatthiasJSax 14

21. 21
21
Static Group Membership
group.instance.id member.id
A 1
B 2
C 3
GroupCoordinator
(broker side)
group.id=“grp”
group.instance.id=“A”
POD
Application
member.id=1
group.id=“grp”
group.instance.id=“C”
POD
Application
member.id=3
group.id=“grp”
group.instance.id=“B”
POD
Application
member.id=2
@MatthiasJSax 14

22. 22
22
22
Looking into the Future
● Work in progress
○ Incremental rebalancing
for Kafka Consumers
and Kafka Streams
● Future work
○ Smooth scale-out for
Kafka Streams
@MatthiasJSax
15

23. 23
23
Incremental Rebalancing
GroupCoordinator
(broker side)
C1
C2
C3
@MatthiasJSax
C1
C2
C3
JoinGroup
(subscription)
rebalanceheartbeat
JoinGroup
(subscription) JoinResponse
16

24. 24
24
Incremental Rebalancing
@MatthiasJSax
C1
C2
C3
GroupCoordinator
(broker side)
Group Leader
(received all subscriptions)
JoinResponse
SyncGroup
(intended
assignment)
SyncResponse
(enforce revoke)
16

25. 25
25
Incremental Rebalancing
@MatthiasJSax
C1
C2
C3
GroupCoordinator
(broker side)
Group Leader
(received all subscriptions) synchronization barrier
JoinResponse
SyncGroup
(intended
assignment)
SyncResponse
(enforce revoke)
JoinGroup
JoinResponse
16

26. 26
26
Incremental Rebalancing
@MatthiasJSax
C1
C2
C3
Group Leader
eived all subscriptions) synchronization barrier
GroupCoordinator
(broker side)
SyncGroup
(intended
assignment)
SyncResponse
(enforce revoke)
JoinGroup
Group Leader
(received all subscriptions)
JoinResponse
SyncGroup
SyncResponse
16

27. 27
27
27
@MatthiasJSax
Summary
● Deep dive into rebalance protocol
○ Powerful and flexible
○ Stop-the-world property
● Since AK 2.3 / CP 5.3
○ Static group membership
○ Incremental rebalancing (Connect)
● Work in progress for AK 2.4 / CP 5.4
17

28. 28
28
28
● heartbeat.interval.ms
● session.timeout.ms
vs
max.poll.interval.ms
● rebalance.timeout.ms
● Static group membership
○ group.instance.id
● Quick vs “considered” rebalance
Lessons Learned
@MatthiasJSax
18

29. 29@MatthiasJSax
Resources
• KIP-345: Introduce static membership protocol to reduce consumer rebalances (Apache Kafka 2.3 / Confluent Platform 5.3)
https://cwiki.apache.org/confluence/display/KAFKA/KIP-
345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
• Design doc: Incremental Cooperative Rebalancing
https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
• KIP-415: Incremental Cooperator Rebalancing in Kafka Connect (Apache Kafka 2.3 / Confluent Platform 5.3)
https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect
• KIP-429: Kafka Consumer Incremental Rebalance Protocol (accepted)
https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol
• KIP-441: Smooth Scaling Out for Kafka Streams (under discussion)
https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams
• "The Magical Rebalance Protocol of Apache Kafka" by Gwen Shapira (Strange Loop Talk, Sep 2018)
https://www.youtube.com/watch?v=MmLezWRI3Ys&t=8s
• Thanks to Jason Gustafson and Guozhang Wang
19

30. 30
30
We are hiring!
@MatthiasJSax
matthias@confluent.io | mjsax@apache.org