Presentation at Spark Summit 2015

1. Spark and Spark
Streaming @ Netflix
Kedar Sadekar & Monal Daxini

2. Mission
• Enable rapid pace of innovation for
Algorithm Engineers
• Business Value – More A/B tests

3. Experiments
Users with plays Feature selection Large sample size
Turn back time Multiple ideas

4. Use Cases
Feature
Selection
Feature
Generation
Model
Training
Metric
Evaluation

5. Use Cases
More
Users
Faster
Iteration
Interactive
Turn back
time

6. Solution – Netflix BDAS
Netflix BDAS
Notebooks
• Zeppelin /
iPython
• Inline Graphs /
REPL
Prana Sidecar
• Netflix
Ecosystem
Berkeley BDAS
• Faster Compute
• Scale Users
• Access to S3 / Hive

7. Netflix BDAS - Features
• Simplicity
– Individual cluster
• Prana - Netflix ecosystem
– Automatic configuration
– Classloader isolation
– Discovery & healthcheck

8. Netflix BDAS - Features
• Ad-hoc experimentation
• Time machine functionality
• Access to Hive data and micro services from single place
– Access to multiple AWS buckets (S3)

9. Netflix BDAS – Sample Deployment

10. Wins
• 8X the number of users
• 5x - 9x faster
• Interactive
• Turn back time

11. Learnings
• Easy to bring down an online system
• Almost killed 1000’s of ETL jobs
- hive metastore update
• Too many systems and configuration
• Playing catch up with libraries and tools
- Hadoop, iPython, Zeppelin
• Scala / Spark learning curve
• Debugging
- files open, no resources etc.

12. Increased Adoption
Adoption increasing amongst teams
• Multiple Algorithmic Eng. teams
• Personalization Infrastructure
• Marketing
• Security
• A/B Test Engineering

13. Looking Ahead
• Spark-R / Dataframes support
• Multi-tenancy
– Job specific configurations
• Debuggability
• Newer notebooks
• Spark Streaming
– Lambda Architecture
– Real-time algorithms (trending now)

14. Netflix Streaming
Event Data Pipeline
Monal Daxini

15. Netflix Event Data Pipeline
Event Streams
Stream processing

16. Publish
Collect
Move
Process
Events @ Cloud Scale

17. 450 Billion Events per Day
8 Million (17 GB) per sec peak

18. S3
EMR
Event
Producer
Fronting
Kafka
Consumer
Kafka
At least Once
Processing
Stream
Consumers
Mantis
450 Billion events / day

19. S3
EMR
Event
Producer
Fronting
Kafka
Consumer
Kafka
At least Once
Processing
Stream
Consumers
Mantis
450 Billion events / day

20. S3
EMR
Event
Producer
Fronting
Kafka
Consumer
Kafka
At least Once
Processing
Stream
Consumers
Mantis
450 Billion events / day

21. S3
EMR
Event
Producer
Fronting
Kafka
Consumer
Kafka
Stream
Consumers
Mantis
450 Billion events / day
At least Once
Processing

22. Backpressure
Direct API for Kafka
Improve Cloud Multi-tenancy
What’s Missing?
+
ì

23. Backpressure+
JobScheduler JobGenerator DStreamGraph
Driver
00 : 0000 : 0100 : 0200 : 0300 : 0400 : 0500 : 0600 : 0700 : 0800 : 0900 : 10
Unbounded …
Ops Queue

24. Backpressure
SPARK-7398 – Add backpressure to Spark streaming
SPARK-6691 – Add dynamic rate limiter to Spark streaming
+

25. Backpressure+
Backpressure implementation slated for
Spark 1.5 release

26. Direct API for Kafka
Spark 1.3
Kafka Integration
Spark 1.2
Receiver based
Kafka Integration
2x
Faster

27. Direct API for Kafka
Prefetch messages
Connection reuse (pooling)
Enhance

28. Cloud Multi-tenancyì
Cloud
Scheduler
Mesos Framework
Mesos Slave
Docker
Executor
Task Task
Docker
Executor
Task Task
Mesos Slave
Docker
Spark Driver
Improve

29. Next?
Measuring Spark Streaming Latencies
Spark Streaming Cloud Multi-tenancy