Trilha BigData - Escalabilidade, Performance e Arquitetura do Apache Kafka para Event-Driven Architectures

1. @alexandregamma
Apache Kafka

2. Alexandre Gama
- Senior Software Engineer
- Team Leader
- Teacher
- Speaker / Writer

3. @alexandregamma
Agenda
Apache Kafka 101
Kafka High Level Architecture
Kafka Low Level Architecture

4. @alexandregamma
Apache
Kafka 101
- Where is Used ?
- What is Apache Kafka ?
- For What is Used ?
- Why is so Special?

5. Companies Using Kafka
- 1⁄3 of all Fortune 500

6. Companies Using Kafka
- 1⁄3 of all Fortune 500
- Top 10 travel companies

7. Companies Using Kafka
- 1⁄3 of all Fortune 500
- Top 10 travel companies
- 7 of top 10 banks

8. Companies Using Kafka
- 1⁄3 of all Fortune 500
- Top 10 travel companies
- 7 of top 10 banks
- 8 of top 10 insurance companies

9. Companies Using Kafka
- 1⁄3 of all Fortune 500
- Top 10 travel companies
- 7 of top 10 banks
- 8 of top 10 insurance companies
- 9 of top ten telecom companies

10. Companies Using Kafka
- 1⁄3 of all Fortune 500
- Top 10 travel companies
- 7 of top 10 banks
- 8 of top 10 insurance companies
- 9 of top ten telecom companies
- Linkedin

11. Companies Using Kafka
- 1⁄3 of all Fortune 500
- Top 10 travel companies
- 7 of top 10 banks
- 8 of top 10 insurance companies
- 9 of top ten telecom companies
- Linkedin
- Microsoft

12. Companies Using Kafka
- 1⁄3 of all Fortune 500
- Top 10 travel companies
- 7 of top 10 banks
- 8 of top 10 insurance companies
- 9 of top ten telecom companies
- Linkedin
- Microsoft
- Netflix

13. Companies Using Kafka
- 1⁄3 of all Fortune 500
- Top 10 travel companies
- 7 of top 10 banks
- 8 of top 10 insurance companies
- 9 of top ten telecom companies
- Linkedin
- Microsoft
- Netflix
- Elo7 (the best one :P)

14. What is Kafka?

15. What is Kafka?
- Distributed Streaming Platform

16. What is Kafka?
- Distributed Streaming Platform
- Publish and Subscribe

17. What is Kafka?
- Distributed Streaming Platform
- Publish and Subscribe
- React to Events in Real Time

18. Kafka Big Picture

19. For What it is Used?

20. For What?
- Real Time Streams

21. For What?
- Real Time Streams
- Collect Big Data

22. For What?
- Real Time Streams
- Collect Big Data
- 1,000,000,000,000

23. For What?
- Real Time Streams
- Collect Big Data
- 1,000,000,000,000
- Real Time Analysis

24. Can Work With?

25. Can work with?

26. Can work with?

27. Can work with?

28. Can work with?

29. Why is so Special?

30. Why is so special?
- Distributed Streaming Platform

31. - Distributed Streaming Platform
- High Throughput and Real Time
Why is so special?

32. Why is so special?
- Distributed Streaming Platform
- High Throughput and Real Time
- Reliability

33. Why is so special?
- Distributed Streaming Platform
- High Throughput and Real Time
- Reliability
- Awesome Replication

34. Why is so special?
- Distributed Streaming Platform
- High Throughput and Real Time
- Reliability
- Awesome Replication
- Fault Tolerance

35. @alexandregamma
High Level
Architecture
- Brokers
- Topic Architecture
- Producer Architecture
- Consumer Architecture

36. Big Picture Again

37. Kafka Broker

38. Kafka Cluster

39. Producer -> Broker

40. Broker -> Consumer

41. Topic Architecture

43. Topic Architecture
- Kafka stores Topics in logs

44. Topic Architecture
- Kafka stores Topics in logs
- Topic log is broken up into Partitions

45. Topic Architecture
- Kafka stores Topics in logs
- Topic log is broken up into Partitions
- Partitions for Speed, Scalability, and Size

46. Topic Architecture
- Kafka stores Topics in logs
- Topic log is broken up into Partitions
- Partitions for Speed, Scalability, and Size
- Maintains record Order only in a single Partition

47. Topic Architecture
- Kafka stores Topics in logs
- Topic log is broken up into Partitions
- Partitions for Speed, Scalability, and Size
- Maintains record Order only in a single Partition
- Topic Partitions are a unit of Parallelism

48. Producer Architecture

49. Producer Architecture
- Load Balancing

50. Producer Architecture
- Load Balancing
- Async Messages

51. Producer Architecture
- Load Balancing
- Async Messages
- Durability

52. Load Balancing: Route Tier?

53. Wrong Broker ?

54. Wrong Broker - Load Balancing

55. Strategy

56. Async Messages

57. Async Messages

58. Async Messages

59. Batch Messages

60. Batch Messages

61. Curiosity..
Buffer Memory on Producer

62. Curiosity 2..
Producer Retries on Errors

63. Curiosity 3..
Messages in Flight

64. Producer Durability - But Before...

65. Producer Durability - But Before...

66. Producer Durability - But Before...

67. Producer Durability - But Before...

68. Producer Durability - But Before...

69. Producer Durability

70. Producer Durability

71. Producer Durability

72. Producer Durability

73. Consumer Architecture

74. Consumer Architecture
- Big Picture

75. Consumer Architecture
- Big Picture
- Consumer Group

76. Consumer Architecture
- Big Picture
- Consumer Group
- ZooKeeper

77. Consuming Messages

78. Consuming Messages

79. Pull vs Push

80. Pull Strategy

81. Offset

82. Consumer Group

83. Consumer Group

84. Consumer Group

85. Consumer Group

86. Consumer Group

87. @alexandregamma
Low Level
Architecture
- Persistence
- Efficiency

88. Persistence

89. Persistence
Data
● Filesystem vs Memory

90. Persistence
Data
● Filesystem vs Memory
● Disk is slow?

91. Data
● Filesystem vs Memory
● Disk is slow?
● Linear Writes vs Random Writes
Persistence

92. Data
● Filesystem vs Memory
● Disk is slow?
● Linear Writes vs Random Writes
● JVM Memory
Persistence

93. Persistence
Data
● Filesystem vs Memory
● Disk is slow?
● Linear Writes vs Random Writes
● JVM Memory
● Compact Byte Structure

94. Persistence
Data
● Filesystem vs Memory
● Disk is slow?
● Linear Writes vs Random Writes
● JVM Memory
● Compact Byte Structure
Cache
● Flush vs Persisted Log

95. Persistence
Data
● Filesystem vs Memory
● Disk is slow?
● Linear Writes vs Random Writes
● JVM Memory
● Compact Byte Structure
Cache
● Flush vs Persisted Log
● Kernel Pagecache

96. Persistence
Data
● Filesystem vs Memory
● Disk is slow?
● Linear Writes vs Random Writes
● JVM Memory
● Compact Byte Structure
Cache
● Flush vs Persisted Log
● Kernel Pagecache
● O(1) -> Data Size

97. Persistence
Data
● Filesystem vs Memory
● Disk is slow?
● Linear Writes vs Random Writes
● JVM Memory
● Compact Byte Structure
Cache
● Flush vs Persisted Log
● Kernel Pagecache
● O(1) -> Data Size
● OS -> Read Ahead

98. Persistence
Data
● Filesystem vs Memory
● Disk is slow?
● Linear Writes vs Random Writes
● JVM Memory
● Compact Byte Structure
Cache
● Flush vs Persisted Log
● Kernel Pagecache
● O(1) -> Data Size
● OS -> Read Ahead
● OS -> Write Behind

99. Efficiency

100. Required!

101. Required!

102. Required!

103. Required!

104. Required!

105. Required!

106. Required!

107. Required!

108. Required!

109. Required!

110. Required!

111. Required!

112. Required!

113. Required!

114. Zero Copy

115. Zero Copy

116. Zero Copy

117. Zero Copy

118. Thank You!