In this talk, we dive into the Netflix Data Science & Engineering architecture. Not just the what, but also the why. Some key topics include the big data technologies we leverage (Cassandra, Hadoop, Pig + Python, and Hive), our use of Amazon S3 as our central data hub, our use of multiple persistent Amazon Elastic MapReduce (EMR) clusters, how we leverage the elasticity of AWS, our data science as a service approach, how we make our hybrid AWS / data center setup work well, and more.

2. What is Netflix’s data warehouse?
a) Cassandra
b) Teradata
c) Hive
d) S3

4. DSE Platform

7. DSE Platform
Chukwa
S3

9. Aegisthus

11. DSE Platform
Chukwa
Aegisthus
S3

12. Sting

15. DSE Platform
Sting
Chukwa
Aegisthus
S3

16. What is Netflix’s data warehouse?
a) Cassandra
b) Teradata
c) Hive
d) S3

17. DSE Platform
Sting
Chukwa
Aegisthus
S3

18. S3

19. S3

20. 99.999999999%

22. S3

23. High SLA
S3
Query

24. HDFS ?

26. “Data Science as a Service”
• Execution Service / Genie
• Event Service
• Metadata Service

32. High SLA Cluster Job
High SLA
S3
Query Cluster Job
Query

33. High SLA
S3
Query Cluster Job
Query

34. High SLA Cluster Job
High SLA
S3
Query Cluster Job
Query

35. Super SLA Cluster Job
Super SLA
S3
High SLA Cluster Job
High SLA
Query Cluster Job
Query

36. Super SLA Cluster Job
High SLA Cluster Job
High SLA
S3
Query Cluster Job
Query

38. Questions?
http://jobs.netflix.com
kurtbrown@netflix.com