Deep Dive Into Kafka Tiered Storage With Satish Duggana | Current 2022 KIP-405 introduced tiered storage in Apache Kafka. The proposed design introduces the separation of compute and storage which benefits the brokers to largely focus on serving producer or consume requests and not manage the storage beyond local disks. But the important caveat here is that it should still maintain the same consistency semantics and lineage of data as in the local storage. This talk dives into the internals of tiered storage in how we achieve those semantics covering scenarios like new brokers bootstrapped, or brokers having hard failures, or other out-of-sync brokers becoming leaders etc. We will also talk about how topic deletion lifecycle management is done without leaking any segments in tiered storage based on the retention policies or while deleting a topic or a partition.

1. Deep dive into Kafka Tiered
Storage
Satish D / Kamal

2. Introduction
Goals
● Scalability
● Efficiency
○ Operational
○ Cost
● Elasticity
Non Goals
● It does not support compact topics
● It does not support JBOD feature
● Tiered storage does not replace ETL pipelines

3. Features
● Provides tiering in storage layer beyond local drives
○ Memory/PageCache
○ Local drive
○ Remote storage support (including cloud stores like S3/GCS/Azure)
■ consistency and ordering semantics as local storage
● Improves efficiency
○ operational
○ cost
● Isolation of reading latest and old data
● Easy tuning and provisioning of clusters
● No changes required from clients

4. Local and Remote Log Segments
6 7 8 9 Active
1 2 3 4 6 7
Local Log Segments
Remote Log Segments
5

5. High Level Architecture

6. Follower Fetch
● There are two main states
○ Fetch
■ Fetch the messages from the leader and append it to its log segment
○ Truncating
■ Truncate the existing data to make sure its log segments follow the
same log lineage of the leader

7. Follower Fetch
1. Leader copies log segments with the auxiliary
state(includes leader epoch cache and producer-id
snapshots) to remote storage.

8. Follower Fetch
1. Leader copies log segments with the auxiliary
state(includes leader epoch cache and producer-id
snapshots) to remote storage.
2. Leader publishes remote log segment metadata about the
copied remote log segment.

9. Follower Fetch
1. Leader copies log segments with the auxiliary
state(includes leader epoch cache and producer-id
snapshots) to remote storage.
2. Leader publishes remote log segment metadata about the
copied remote log segment.
3. Follower tries to fetch the messages from the leader.

10. Follower Fetch
1. Leader copies log segments with the auxiliary
state(includes leader epoch cache and producer-id
snapshots) to remote storage.
2. Leader publishes remote log segment metadata about the
copied remote log segment.
3. Follower tries to fetch the messages from the leader.
4. Follower waits till it catches up consuming the required
remote log segment metadata.

11. Follower Fetch
1. Leader copies log segments with the auxiliary
state(includes leader epoch cache and producer-id
snapshots) to remote storage.
2. Leader publishes remote log segment metadata about the
copied remote log segment.
3. Follower tries to fetch the messages from the leader.
4. Follower waits till it catches up consuming the required
remote log segment metadata.
5. Follower fetches the respective remote log segment
metadata to build auxiliary state.

12. Follower Fetch
● Which offset should follower fetch from
○ Last tiered offset
○ Earliest local offset
2001 9000 Active
0 1000
Local Log Segments
Remote Log Segments
…….. 2001 9000
…….. 12000
……..
Earliest Local Offset
Last Tiered Offset

13. Follower Fetch
● Maintain the same log lineage
● Leader epochs
○ It is a representation of leader transitions, which is a monotonically increasing
number
○ Added with each message batch by the leader into the log segment
○ Maintains the leader epoch sequence file with epoch vs start-offset
■ Maintained by each replica
■ Enables maintaining the log lineage across the replicas

14. Leader
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
Leader epochs: (0,0)
Leader - A RemoteStorage

15. Leader
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
Leader epochs: (0,0)
Leader - A
seg 0-1, uuid-1
Leader epochs = (0, 0)
RemoteStorage

16. Leader
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-1
6: msg 6 LE-1
Leader epochs: (0,0), (1, 3)
Leader - B
seg 0-1, uuid-1
Leader epochs = (0, 0)
RemoteStorage

17. Leader
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-1
6: msg 6 LE-1
Leader epochs: (0,0), (1, 3)
Leader - B
seg 0-1, uuid-1
Leader epochs = (0, 0)
seg 2-5, uuid-2
Leader epochs = (0, 0),
(1.3)
RemoteStorage

18. Leader
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-1
6: msg 6 LE-1
7: msg 7 LE-1
Leader epochs: (0,0), (1, 3)
Leader - B
seg 0-1, uuid-1
Leader epochs = (0, 0)
seg 2-5, uuid-2
Leader epochs = (0, 0),
(1.3)
RemoteStorage

19. Follower Fetch - Empty Follower

20. Follower Fetch - Empty follower
1. Fetch from 0
a. Receives OMTS(OffsetMovedToTieredStorage)
2. Fetch ELO (Earliest Local Offset)
a. Receives ELO (leader epoch, offset)
3. Fetch remote segment info and build local leader
epoch sequence until ELO
a. Receives leader epoch sequence, producer-id snapshot
4. Fetch from ELO to HW
Leader - A
Follower - B

21. Follower Fetch - Empty follower
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1,3), (2, 5) (3, 7)
Leader - A
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1,3), (2, 5)
RemoteStorage

22. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Leader - A
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

23. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

24. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Fetch offset 0
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

25. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Fetch offset 0
Receives OMTS
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

26. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Fetch offset 0
Receives OMTS
Fetch ELO
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

27. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Fetch offset 0
Receives OMTS
Fetch ELO
Receives ELO (LE-1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

28. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Fetch offset 0
Receives OMTS
Fetch ELO
Receives ELO (LE-1, 3)
Fetch remote segment info and
build local leader epoch
sequence until ELO
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

29. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1,3), (2, 5) (3, 7)
Fetch offset 0
Receives OMTS
Fetch ELO
Receives ELO (LE-1, 3)
Fetch remote segment info and
build local leader epoch
sequence until ELO
Leader epochs = (0, 0)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0), (1,
3), (2, 5)
RemoteStorage

30. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Fetch offset 0
Receives OMTS
Fetch ELO
Receives ELO (LE-1, 3)
Fetch remote segment info and
build local leader epoch
sequence until ELO
Fetch from ELO to HW
Leader epochs = (0, 0), (1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

31. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Fetch offset 0
Receives OMTS
Fetch ELO
Receives ELO (LE-1, 3)
Fetch remote segment info and
build local leader epoch
sequence until ELO
Fetch from ELO to HW
Leader epochs = (0, 0), (1.3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
RemoteStorage

32. Follower Fetch - Empty follower
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
Leader epochs = (0, 0),
(1.3), (2, 5) (3, 7)
Fetch from ELO to HW
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
Leader epochs = (0, 0), (1,3),
(2,5)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1,3), (2, 5)
RemoteStorage

33. Follower Fetch - Empty follower - Summary
1. Fetch offset 0
a. Receives OMTS
2. Fetch EarliestLocalOffset (ELO)
a. Receives ELO (leader epoch, offset)
3. Fetch remote segment info and build local leader
epoch sequence until ELO
a. Receives leader epoch sequence
4. Fetch from ELO to HW
Leader - A
Follower - B

34. Follower Fetch - Out of sync follower

35. Follower Fetch - Out of sync follower
● Follower catching up with the leader
● Segments are copied to remote storage
○ Locally available
○ Locally not available

36. Follower Fetch - Out of sync follower
● Follower trying to catch up with the leader
● Segments are copied to remote storage
○ Locally available
■ Fetch from the leader like it does without tiered storage

37. Follower Fetch - Out of sync follower
● Follower trying to catch up with the leader
● Segments are copied to remote storage
○ Locally available
■ Fetch from the leader like it does without remote storage
○ Locally not available
■ Truncate the data on follower

38. Follower Fetch - Out of sync follower
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
8: msg 8 LE-3
9: msg 9 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
Leader epochs (0, 0), (1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 6-8: uuid-3
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

39. Follower Fetch - Out of sync follower
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
8: msg 8 LE-3
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
Leader epochs (0, 0), (1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 6-8: uuid-3
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

40. Follower Fetch - Out of sync follower
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
4: msg 4 LE-1
5: msg 5 LE-2
6: msg 6 LE-2
7: msg 7 LE-3
8: msg 8 LE-3
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
Leader epochs (0, 0), (1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 6-8: uuid-3
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

41. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
Leader epochs (0, 0), (1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 6-8: uuid-3
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

42. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
Fetch from leader LE-1, 4
Leader epochs (0, 0), (1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 6-8: uuid-3
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

43. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
Fetch from leader LE-1, 4
Receives OMTS, truncate local
segments.
Leader epochs (0, 0), (1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 6-8: uuid-3
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

44. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-1
Fetch from leader LE-1, 4
Receives OMTS, truncate local
segments.
Leader epochs (0, 0), (1, 3)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 6-8: uuid-3
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

45. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
Fetch from leader LE-1, 4
Receives OMTS, truncate local
segments.
[This is similar to empty broker]
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 6-8: uuid-3
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

46. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
Fetch from leader LE-1, 4
Receives OMTS, truncate local
segments.
Fetch ELO
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

47. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
Fetch from leader LE-1, 4
Receives OMTS, truncate local
segments.
Fetch ELO
Receives ELO 9, LE-3
Rebuilds leader epoch
sequence up to LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

48. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
Fetch ELO
Receives ELO 9, LE-3
Rebuilds leader epoch
sequence up to LE-3
Fetch 9, LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

49. Follower Fetch - Out of sync follower
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
Fetch ELO
Receives ELO 9, LE-3
Rebuilds leader epoch
sequence up to LE-3
Fetch 9, LE-3
9: msg 9 LE-3
10: msg 10 LE-3
11: msg 11 LE-3
Leader epochs
(0, 0), (1, 3), (2, 5), (3, 7)
Leader - A Follower - B
seg 0-2, uuid-1
Leader epochs = (0, 0)
seg 3-5, uuid-2
Leader epochs = (0, 0),
(1.3), (2, 5)
seg 9-10: uuid-4
Leader epochs = (0, 0),
(1.3), (2, 5), (3,7)
RemoteStorage

50. Follower Fetch - Unclean leader election

51. Follower Fetch - Unclean leader election
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-0 (HW)
Leader epochs : (0, 0)
0: msg 0 LE-0
Leader epochs : (0, 0)
Leader - A Follower - B
seg 0-2: uuid-1
log:
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
Leader epochs : (0, 0)
RemoteStorage

52. Follower Fetch - Unclean leader election
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-0 (HW)
Leader epochs : (0, 0)
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0,0), (1, 1)
Leader - A
Stopped
Leader - B
seg 0-2: uuid-1
log:
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
Leader epochs : (0, 0)
seg 0-1: uuid-2
log:
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0, 0),
(1,1)
RemoteStorage

53. Follower Fetch - Unclean leader election
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
3: msg 3 LE-0 (HW)
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0, 0),
(1,1)
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0,0), (1, 1)
Follower-A Leader - B
seg 0-2: uuid-1
log:
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
Leader epochs : (0, 0)
seg 0-1: uuid-2
log:
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0, 0),
(1,1)
RemoteStorage

54. Follower Fetch - Unclean leader election
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0, 0),
(1,1)
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0,0), (1, 1)
Follower-A Leader - B
seg 0-2: uuid-1
log:
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
Leader epochs : (0, 0)
seg 0-1: uuid-2
log:
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0,0), (1,1)
RemoteStorage
RLMM maintains the indexes of epochs, offsets and the respective
segments
- For a given leader-epoch and offset it returns any segment that
contains the respective message
seg-0-2, uuid-1
segment epochs
(0, 0)
seg-0-1, uuid-2
segment epochs
(0, 0), (1, 1)
RemoteMetadata
Storage

55. Follower Fetch - Unclean leader election
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0, 0),
(1,1)
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0,0), (1, 1)
Follower-A Leader - B
seg 0-2: uuid-1
log:
0: msg 0 LE-0
1: msg 1 LE-0
2: msg 2 LE-0
Leader epochs : (0,0)
seg 0-2: uuid-2
log:
0: msg 0 LE-0
1: msg 4 LE-1
2: msg 5 LE-1
Leader epochs : (0,0), (1,1)
RemoteStorage
Consumer fetch: LE:0, offset:0
- Either of the segments is chosen by RLMM
seg-0-2, uuid-1
segment epochs
(0, 0)
seg-0-1, uuid-2
segment epochs
(0, 0), (1, 1)
RemoteLogMeta
data Storage
Consumer fetch: LE:1, offset:1
- msg 4 from seg 0-1

56. Thanks

57. Q & A