Learn about how Netflix achieved 5 9s guarentees on KAfka's largest workload.

2. ●
○
○
●
○
○
○

3. ●
●
●

4. A NETFLIX ORIGINAL SERVICE

5. Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Consumer
Kafka
Management
HTTP
PROXY

6. Fronting Kafka Clusters Consumer Kafka Clusters
Number of clusters 24 15
Total number of instances 1700+ 1100+
Instance type d2.2xl i2.2xl
Replication factor 2 2
Retention period 8 to 24 hours 2 to 4 hours

7. ●
○
○
○
●
○

8. ●
○
●

9. ●
○
●
●
●
○
○

10. ●
●
●
0.1% 0.5% 1% 5% Percent loss

11. ●
●
○
○
●

12. ●
●
●

13. ●
○
○
● …

15. ●
○ →

16. ●
●
○
○
○ →
●

18. ●
●
○
○
○

19. ●
○
○
○
●
●

20. ●
○ →
●
○ →
●
○ →

21. ●
●
○
●
○

22. Rack 0 Rack 1
0
Broker 0 Broker 1 Broker 2 Broker 3
3 0 1 1 2 2 3
N = Partition N for a topic with 2 replicas
0 ← Off line partition

23. Rack 0 Rack 1
0
Broker 0 Broker 1 Broker 2 Broker 3
3 1 2 0 1 2 3
N = Partition N for a topic with 2 replicas
No offline partition

24. ●
●
○
●
●

25. ●
○
○
○
●
○
○

26. Event
Producer
Kafka
Buffer exhausted
and message
drop Slow replication
Broker with
networking
problem
Disk read
causes slow
responses
X
X
X

28. 28

29. 29

30. ●
●
○
○
○

31. ●
○
○
○

32. ●
○
○
●
●
●

34. ●
●
●
●

35. RouterFronting
Kafka
Event
Producer
X
Consumer
Kafka
Copy topic metadata
Consumer

36. ●
Fully
Automated