This document summarizes the key changes to Kafka's message format over different versions from 0.7 to 0.10. It discusses how the message format changes helped address bottlenecks and shifting from predominantly network-bound to CPU-bound. It also describes how LinkedIn migrated their mirroring pipelines to continue using the efficient 0.7 approach with newer Kafka versions.

1. ​ Joel Koshy
​ Sr. Staff Software Engineer
​ LinkedIn

2. Outline
Background
A look back at Kafka 0.7
Kafka 0.8, 0.9 and shifting bottlenecks
Kafka 0.10 message format changes
Mirroring – the 0.7 way
Q&A

4. Kafka at LinkedIn

5. Kafka 0.7

6. Kafka 0.7 message format (uncompressed)

7. Kafka 0.7 message format (compressed)

8. Kafka 0.7 message format
Message offset is
the physical offset

9. Kafka 0.7 message format
Internal messages of compressed
message sets are not addressable
via a scalar offset

10. Kafka 0.7 message format
Consumer checkpoints offset M
for this message

11. • Tricky to checkpoint within
compressed message set
• Hard to rewind by N
messages
• Unsuitable for features
such as log compaction
Drawbacks of 0.7
message format

12. B U T ! ! !
Very efficient
(broker did not need to modify messages)

13. Predominantly network in 0.7
S H I F T I N G B O T T L E N E C K S O V E R T I M E

14. Kafka 0.8
Replication

15. Replication in Kafka 0.8

16. Replication in Kafka 0.8

17. S H I F T I N G B O T T L E N E C K S O V E R T I M E
Predominantly network in 0.7
Still network in 0.8, gradually tilting toward storage

18. Kafka 0.8 message format (uncompressed)
NOT a
physical offset!

19. Kafka 0.8 message format (compressed)
Envelope offset is the
largest offset in the set

20. Broker now needs to assign logical offsets
even to internal messages

21. Handling produce requests

22. Handling produce requests

23. Handling produce requests

24. Handling produce requests

25. Kafka 0.9
Security (and many other features)

26. SSL
• Forego zero-copy optimization
• CPU overhead to decrypt/encrypt
• Minor impact
• (Used only on our mirroring pipelines
at the time)
Kafka 0.9

27. Mirroring topology and storage policy changes

28. Mirroring topology and storage policy changes

29. Mirroring topology and storage policy changes
• Reduced retention period
across the board
• File system tuning

30. 2 0 1 5 - 2 0 1 6
New hardware
10Gbps NICs, bigger disks, XFS

31. S H I F T I N G B O T T L E N E C K S O V E R T I M E
Predominantly network in 0.7
Storage and network in 0.8, 0.9 (1Gbps NICs)
Increasingly CPU in 0.9 (10Gbps NICs)

32. Kafka 0.9 broker profile

33. Kafka 0.9 broker profile
Cluster expansion

34. Kafka 0.10
New message format

35. Kafka 0.10 message format (uncompressed)

36. Kafka 0.10 message format (compressed + append time)

37. Kafka 0.10 message format (compressed + create time)

38. Handling produce requests

39. Handling produce requests

40. Handling produce requests

41. Handling produce requests

42. Handling produce requests

43. Handling produce requests

44. Handling produce requests

45. Handling produce requests

46. Handling fetch requests

47. Handling fetch requests

48. Handling fetch requests

49. Handling fetch requests

50. Handling fetch requests

51. Migrate clients before switching to 0.10 message format
Ideal Less ideal Worse Worst
Majority
producer version
0.10 0.9 0.10 0.9
Majority
consumer version
0.10 0.10 0.9 0.9

52. Migrating to the new message format

53. Migrating to the new message format

54. Migrating to the new message format

55. Migrating to the new message format

56. Migrating to the new message format

57. Migrating to the new message format

58. Migrating to the new message format

59. Migrating to the new message format

60. Migrating to the new message format

61. Migrating to the new message format

62. Migrating to the new message format

63. Migrating to the new message format

64. Migrating to the new message format

65. Migrating to the new message format

66. U S E C A U T I O N ! !
Severe performance degradation with older clients
and there is no roll-back after switching

67. So know your clients!
• Useful to have a shepherding
system in your service infra
• EOL older libraries
• Check API versions in public
access logs
• Add API version metrics to the
Kafka broker
PRODUCERS CONSUMERS

68. Kafka 0.10 broker profile

69. Impact of new message format

70. • Fine-grained time-based offset
lookup
• Facilitates correctness of time-
based retention
Other benefits of
0.10 format

71. “Time” under 0.9 message format
Message time is mtime of
containing segment so time
granularity is at segment-level

72. “Time” under 0.9 message format
Time resets on
partition reassignment

73. Time in 0.10 message format
Time indexes built from
timestamps in messages

74. Time in 0.10 message format

75. Mirroring pipelines
(embracing the 0.7 way)

76. 0.7 mirror maker was a dumb fast pipe

77. 0.7 mirror maker was a dumb fast pipe

78. 0.7 mirror maker was a dumb fast pipe

79. 0.7 mirror maker was a dumb fast pipe

80. … limited only by network
0 . 7 M I R R O R M A K E R

81. 0.8+ mirror maker
• Needs to preserve order of
keyed messages

82. 0.8+ mirror maker
• Needs to preserve order of
keyed messages

83. 0.8+ mirror maker
• Needs to preserve order of
keyed messages

84. 0.8+ mirror maker
• Needs to preserve order of
keyed messages
• 0.8+ consumers do not support
shallow iteration (KAFKA-732)
• 0.8+ producers do not support
pass-through mode

85. Kafka 0.8+ mirror maker profile

86. Handling keyed messages in pass-through mode
• Need to preserve order of keyed messages…
but pass-through mirror maker cannot
repartition

87. Handling keyed messages in pass-through mode
• Need to preserve order of keyed messages…
but pass-through mirror maker cannot
repartition
• Work around is to require identical partition
counts across all clusters and do identity
partitioning
• i.e., Pinput= Poutput

88. • Restore shallow iteration in consumer
(KAFKA-1895)
• “Todd’s trick” – introduce an identity
compression codec in producer
• Uniform partition counts across clusters
0.10 pass-through
mirroring how-to

89. • Restore shallow iteration in consumer
(KAFKA-1895)
• “Todd’s trick” – introduce an identity
compression codec in producer
• Uniform partition counts across clusters
• … and a few more subtleties (future talk)
0.10 pass-through
mirroring how-to

90. • Jiangjie Qin (KIP-3[1,2,3])
• Todd Palino (pass-through mirroring)
• Kafka open source community
Acknowledgments

91. +