In this talk, Peter will cover his experience using Spark, Cassandra & Kafka to build a real time analytics platform that processed billions events a day. He will cover the challenges in how to turn all those raw events into actionable insights. He will also cover scaling the platform across multiple regions, as well as across multiple cloud environments.

1. Zero to Insights
Real time analytics with Kafka, C*, and Spark
Peter Bakas

2. Peter Bakas | @peter_bakas
@ Netflix : Cloud Platform Engineering - Event and Data Pipelines
@ Ooyala : Analytics, Discovery, Platform Engineering & Infrastructure
@ Yahoo : Display Advertising, Behavioral Targeting, Payments
@ PayPal : Site Engineering and Architecture
@ Play : Advisor to various Startups (Data, Security, Containers)
Who is this guy?

3. Let’s get down to business

4. Netflix is a logging company

5. that occasionally streams video

6. ● 450 billion events per day
● 8 million events & 17 GB per second during
peak
● Hundreds of event types
By the Numbers

8. Publish, Collect, Process, Aggregate & Move Data

9. @ Cloud Scale

10. What does it take to run @ Cloud Scale?

11. How did we get here?

12. EMR
Event
Producer
Chukwa

13. What are we supposed to do?

14. Event
Producer
Druid
EMR
Stream
Consumers
Kafka
Router
Suro
Event
Producer

15. Event
Producer
Druid
EMR
Stream
Consumers
Kafka
Router
Suro
Event
Producer

16. Where are we going?

17. Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone

18. Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone

19. Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone

20. Routing Service
++

21. Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone

22. Consumer
Kafka
Custom Apps
Real time processing

23. Consumer
Kafka
Custom Apps
Real time processing

24. Consumer
Kafka
Custom Apps
Real time processing

25. Consumer
Kafka
Custom Apps
Real time processing

26. Consumer
Kafka
Custom Apps
Real time processing
Fronting
Kafka

27. Ooyala’s experience

28. About Ooyala
Powering personalized
video experiences
across all screens.

29. ● 5 billion events per day
● 1 billion videos per month
● 200 million unique users per month
● 130 countries
● 25% of U.S. online viewers watch video powered by Ooyala
By the Numbers

30. Where did it all start?

32. Precomputed Aggregates

33. What if we need more dynamic queries?

34. Why not just use C*?

35. What were the options?
100% Precomputation 100% Dynamic

36. Where we wanted to be
100% Precomputation 100% Dynamic
Partly dynamic

37. Our solution

38. API
loggersloggersloggersloggersloggers
loggersloggersloggersloggersingest
loggersloggersloggersloggersjob server
Delphi - Real time Analytics
Kafka

39. Challenges

40. ● Hiring
● Rapidly evolving ecosystem
● Enterprise Service for Enterprise Software
Challenges

41. Q&A time!

42. Obligatory...
Everyone is hiring
pbakas@netflix.com