1. Restoring Restoration’s Reputation
in Kafka Streams
Bruno Cadonna & Lucas Brutschy

2. About the Presenters
2
Bruno Cadonna
Senior Software Engineer @ Confluent
Committer & PMC member @ Apache Kafka
Lucas Brutschy
Senior Software Engineer @ Confluent
Contributor @ Apache Kafka

3. 3
Outline
The basics
Why does restoration have a poor reputation?
Restoring restoration’s reputation
Experiments
Broader vision & future work
Take aways

4. The basics
4

5. Kafka Streams
5
Stateful processors use local state stores
1234 2345 9876 2846 xxxx xxxx xxxx 2846
4
3
4
stateless processor
stateful processor
stream

6. Subtopologies in Kafka Streams
6
stateless processor
stateful processor
stream
A topology consists of subtopologies connected by Kafka topics
Subtopology A Subtopology B
repartitioning
define new key

7. Tasks in Kafka Streams
7
A task is an instance of a subtopology that processes records from one topic partition
task A0
task A1
task A2
task B0
task B1
task B2
stateless processor
stateful processor
stream

8. Task assignment in Kafka Streams
8
During failover Kafka Streams moves tasks to the running clients
task A0
task B0
task A1
task A2
task B2
task B1
Streams client 1 Streams client 2 Streams client 3
Kafka Streams distribute tasks over the Streams clients
Running clients do not have local state for newly assigned tasks

9. Restoration in Kafka Streams
9
When tasks are moved to a different Streams client their states are restored from Kafka
Streams client 1 Streams client 3
task B1
task A0
task A0 task B1
task B1
task A0
Streams client 2
Restoration
task B1
task A0
Replication
Kafka Streams replicate the content of state stores to changelog topics in Kafka

10. Standby tasks
10
● Hot local replicas of a local state store
● Assigned to Streams clients that DO NOT own the task whose state is
replicated
● Updates local state store changelog topic
● No processing
● Avoid restoration when tasks are moved to other Streams clients

11. How to see restoration in your applications
11
Verify progress of restoration:
[2023-04-26 13:43:48,739] INFO stream-thread [<client ID>-StreamThread-1] Restoration in progress for
10 partitions. {benchmark-store-changelog-5: position=26324, end=252519, totalRestored=26272}
{benchmark-store-changelog-4: position=26360, end=252296, totalRestored=26336}
{benchmark-store-changelog-3: position=26394, end=252304, totalRestored=26351}
{benchmark-store-changelog-2: position=26279, end=252514, totalRestored=26240}
{benchmark-store-changelog-9: position=13270, end=252490, totalRestored=13239}
{benchmark-store-changelog-8: position=13277, end=252495, totalRestored=13247}
{benchmark-store-changelog-7: position=13273, end=252527, totalRestored=13234}
{benchmark-store-changelog-6: position=13266, end=252508, totalRestored=13245}
{benchmark-store-changelog-1: position=13298, end=252710, totalRestored=13278}
{benchmark-store-changelog-0: position=13275, end=252705, totalRestored=13256}
(org.apache.kafka.streams.processor.internals.StoreChangelogReader:575)
Verify whether restoration finished:
2023-04-26 13:49:05,441] INFO stream-thread [<client ID>-StreamThread-1] Restoration took 189227 ms for all
tasks [1_0, 0_9, 0_8, 1_9, 0_7, 1_8, 0_6, 1_7, 0_5, 1_6, 0_4, 1_5, 0_3, 1_4, 0_2, 1_3, 0_1, 1_2, 0_0, 1_1]
(org.apache.kafka.streams.processor.internals.StreamThread:902)

12. Why does restoration have a poor
reputation?
12

13. Phases of a Stream thread
Poll Restore Poll Restore Poll Restore Poll Restore
Process Process
“Restoration Phase” “Processing Phase”
Active
Tasks
Standby
Tasks
Active
Tasks
Standby
Tasks
Active
Tasks
Standby
Tasks
Active
Tasks
Standby
Tasks
Rebalance
Time
All active tasks are blocked from processing until
restoration is complete
Poll Re
stream thread
All active tasks have
caught up

14. Checkpointing offsets of states
No checkpoints!
Time If restoration is interrupted, it has to
restart from the beginning
✔ taking checkpoints
Poll Restore Poll Restore Poll Restore Poll Restore
Process Process
Active
Tasks
Standby
Tasks
Active
Tasks
Standby
Tasks
Active
Tasks
Standby
Tasks
Active
Tasks
Standby
Tasks
Rebalance Poll Re
✔ ✔
stream thread

15. More disadvantages of coupling restoration / processing
15
➔ If restoration is slow, we risk falling out of the consumer group
➔ If restoration is slow and exactly-once semantics are enabled, we
risk timing out transactions
➔ Both trigger rebalance and restoration

16. Restoring restoration’s reputation
16

17. Introducing the state updater
Time
Restoration and processing happen in parallel
task A0
task B1
Active Tasks
Poll Process
Rebalance Poll Poll Poll
Process Process
Process Poll Process
Restore
Standby Tasks
task B1
task A0
Active Tasks
17
state updater
stream thread

18. Restore
Standby Tasks
task B1
task A0
Active Tasks
task A0
task B1
Active Tasks
Poll Process
Rebalance Poll Poll Poll
Process Process
Process Poll Process
Checkpointing offsets during restoration
Time
Checkpoints are taken during restoration and
processing
✔ ✔
✔
✔ ✔ ✔ ✔
✔ taking checkpoints
18
stream thread
state updater
✔ ✔

19. Communication between stream thread and state updater
Rebalance Poll Process
19
Restore
Restored
tasks
Tasks to add
and remove
Removed
tasks
Exceptions
and tasks
stream thread
state updater

20. New Metrics
20
active-restoring-tasks The number of active tasks currently undergoing restoration
standby-updating-tasks The number of standby tasks currently undergoing state update
idle-ratio The fraction of time the thread spent on idling
active-restore-ratio .. on restoring active tasks
standby-update-ratio .. on updating standby tasks
checkpoint-ratio .. on checkpointing tasks restored progress
KIP-869 (Guozhang Wang)

21. Run it!
21
See how tasks are passed to the state updater:
[2023-04-26 17:12:29,915] INFO state-updater [<client ID>-StateUpdater-1] Stateful active task 1_4 was added
to the state updater (org.apache.kafka.streams.processor.internals.DefaultStateUpdater:370)
and complete restoration:
[2023-04-26 17:17:00,936] INFO state-updater [<client ID>-StateUpdater-1] Stateful active task 1_4 completed
restoration (org.apache.kafka.streams.processor.internals.DefaultStateUpdater:449)
State Updater is available in Kafka 3.5 but gated by an internal flag
Try it out by setting __state.updater.enabled__ streams config
(use at your own risk)

22. Experiments
22

23. Setup
23
input
output
repartition
topic
repartition
topic
State
Process without
restoration
Process only after
restoration

24. Getting back into the business sooner…
24
w/o state updater
w/ state updater
● 1 single Streams client
● Topics with 10 partitions
● Runtime: 20 min
● After 10 min the client is restarted and
local state cleaned up
● Streams client needs to restore all state

25. CPU load
25
w/o
state updater
w/
state updater
stream thread
state updater

26. Broader vision & future work
26

27. Stream Thread
Old model
27
Restore
Consumer
Main
Consumer
Poll/Restore/Process
Loop
Producer

28. Multi-core scenario
28
Stream Thread
Restore
Consumer
Main
Consumer
Poll/Restore/Process
Loop
Producer
Stream Thread
Restore
Consumer
Main
Consumer
Poll/Restore/Process
Loop
Producer
Stream Thread
Restore
Consumer
Main
Consumer
Poll/Restore/Process
Loop
Producer
Stream Thread
Restore
Consumer
Main
Consumer
Poll/Restore/Process
Loop
Producer

29. …
The vision
29
State Updater
Poll Thread
Restore
Consumer
Main
Consumer
Poll
Loop
Restore
Loop
Producer
Thread
Producer
Processing Thread 1
Processing Thread 2
Processing Thread N

30. Restoration with exactly-once semantics
30
- When a Streams client fails, the local state may have been
updated since the last commit
- We cannot “roll back” the state
- State has to be wiped, triggering full restoration
KIP-844: Transactional State Stores
KIP-892: Transactional Semantics for State Stores
●

31. Take aways
31

32. Take aways
32
• Restoration decoupled from processing
• Checkpointing during restoration
• Part of larger change in threading model
Joint work with Guozhang Wang

33. Thank you!
33

34. Challenges
34

35. 35
Poll
The Stream thread loop
Restore Process

36. 36
Poll
One more detail: check pointing
Restore Process
Checkpoint
File
✔
Keeps the latest offset of changelog topic
Answers the question “is our state up-to-date?” after restart

37. State Updater
Stream Thread
Model with the state updater
37
Restore
Consumer
Main
Consumer
Poll/Process
Loop
Producer
Restore
Loop

38. 38
Cycles saved
Cycles gained

39. 39
Cycles saved
Cycles gained
We save cycles on restoration!

40. 40
Cycles saved
Cycles gained
We gain cycles on
during processing?
* Throughput 15% lower!

41. Causing a slowdown, even when barely running at all?
41
Stream
Thread
State Updater
Thread

42. Icons
42
Central Nervous
System
Early Production
Streaming
Stream
Designer
Data
Everywhere
Kafka Cluster
Database Databases Data Lake DB Warehouse Data Center Cloud Cloud to Cloud Hybrid Cloud Cloud Dev Equal Cloud Cloud
Management
Server On-premise Serverless Replicator Operator Kafka KSQL Rocket ksqlDB KSQL Circle Connector Microservices Schema Registry
Streams Event Streams Number of
Data Sources
IOT Cluster Partition
Rebalancing
Stream Processing
Cookbook
Data
Governance
Apps Service Apps Custom Apps Logs Data Stacks Stack Overflow Storage Platform Data In Data Out Data Add
Branch
Processing Real-time Aggregate Data Frameworks CLI Dev Scale Combine Join Architect # of Producers

43. Icons
43
Webinar Developer Onboard Offboard Filter
Globe Infinity Settings Monitoring Anomaly
Detection
Analytics Real-time
Analytics
Real-time Processing Process Data Upload Download
Computer Devices Computer /
DB / Cloud
Status Open Source Web Confirmed RSS MQTT Message Quotes Interview # of Topics
Person People People
Manager
Career
Enablement
Roadmap Search Solution
Send
Features Company
Policies
Docs Invoice Blog Podcast Video Book Table Email Print
Continuous
Learning
Lock Key Warning Hacker Bug GDPR CCPA Shield Shield Open Machine
Learning
Eye

44. Icons
44
Shirt Food Catalyst Box Sparkly New
Manufacturing Venue Government Business Marketplace Ecommerce Sale Money Telecom Support Gaming Healthcare
Computer Love Partner Hand Arm Benefit Thumbs Up Swipe Select Promote Awareness Target
Car Truck Puzzle Lightening Star Question Check
Workday
Speed Time Coming Soon Time / Money ROI TCO Data in Terabytes
Per Day
# of Events
Per Day
Calendar Payday Docker
Transfer Expand / Shrink Add Balance Rest Trophy Certificate Badge