O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa Tangirala, Netflix) | Cassandra Summit 2016

4.559 visualizações

Publicada em

Learning is an analytic process of exploring the past in order to predict the future. Hence, being able to travel back in time to create features is critical for machine learning projects to be successful. To enable this, we built a time machine that computes features for any arbitrary time in the recent past for offline experimentation. We also built a real-time stream processing system to capture the interests of members during different times of the day and to quickly adapt to changes in the collective interests of members as it happens in case of real-world events.

Building the time machine for offline experimentation and the real-time infrastructure for online recommendations with Apache Spark (Streaming) and Apache Cassandra empowered us to both scale up the data size by an order of magnitude and train and validate the models in less time. We will delve into the architecture, use case details, data models used for cassandra and share our learnings.

About the Speakers

Prasanna Padmanabhan Engineering Manager, Netflix

Prasanna leads the Data Systems for Personalization team at Netflix. His primary focus is on building various big data infrastructure components that help their algorithmic engineers to innovate faster and improve personalization for Netflix members. In the past, he has built distributed data systems that leverages both batch and stream processing.

Roopa Tangirala Engineering Manager, Netflix

Roopa Tangirala is an experienced engineering leader with extensive background in databases, be they distributed or relational. She manages the database engineering team at Netflix responsible for operating cloud persistent and semipersistent runtime stores for Netflix, which includes Cassandra, Elasticsearch, Dynomite and MySQL databases, by ensuring data availability, durability, and scalability to meet the growing business needs.

Publicada em: Software
  • Download The Complete Lean Belly Breakthrough Program with Special Discount. ♣♣♣ https://tinyurl.com/bkfitness4u
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Is it possible to improve your memory? How can I improve my memory recall? more info... ◆◆◆ https://bit.ly/2GEWG9T
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui
  • Is Your Ex With a Man? Don't lose your Ex girlfriend! This weird trick will get her back! ♣♣♣ http://goo.gl/FXTq7P
       Responder 
    Tem certeza que deseja  Sim  Não
    Insira sua mensagem aqui

Netflix Recommendations Using Spark + Cassandra (Prasanna Padmanabhan & Roopa Tangirala, Netflix) | Cassandra Summit 2016

  1. 1. Netflix Recommendations using Spark + Cassandra Prasanna Padmanabhan Roopa Tangirala
  2. 2. Turn on Netflix and the absolute best content for you would automatically start playing
  3. 3. Netflix Recommendations
  4. 4. Netflix Recommendations
  5. 5. Ranking Everything is a Recommendation Rows Over 80% of what members watch comes from our recommendations Recommendations are driven by Machine Learning Algorithms
  6. 6. Data Driven Offline Experiment using Historical Data Online A/B Testing Rollout Feature to ALL members Success Success Fail Algorithmic Page Generation Trending Now
  7. 7. Offline Experimentation
  8. 8. Algorithmic Page Generation Personalizing the ordering of rows on the homepage
  9. 9. Algorithmic Page Generation Without Algorithmic Page Generation With Algorithmic Page Generation Diversity of the Page Affinity for specific rows Drawbacks
  10. 10. Algorithmic Page Generation Production
  11. 11. Algorithmic Page Generation Production Variant 1
  12. 12. Algorithmic Page Generation Production Variant 1 Variant 2 Row Distribution TV/Movie Ratio
  13. 13. Algorithmic Page Generation Production Variant 1 Variant 2 Evaluate best variant based on the plays Actual Plays:
  14. 14. Algorithmic Page Generation Production Variant 1 Variant 2 Evaluate best variant based on the plays Actual Plays:
  15. 15. Algorithmic Page Generation Production Variant 1 Variant 2 Evaluate best variant based on the plays Actual Plays:
  16. 16. Variant 2 Algorithmic Page Generation Production Variant 1 Evaluate best variant based on the plays Actual Plays:
  17. 17. Offline Experiment Architecture Member Selection Runs once a day Ratings Service S3 Snapshot Snapshot Store Snapshot Forklift Viewing History Service MyList Service Data Snapshots Evaluate Metrics Generate Pages … … A/B Test
  18. 18. Data Model - Requirements • Need for historical service data • Optimize for Batch Writes and Point Reads
  19. 19. Data Model 20161009_1001 20161009_1002 DATE_MEMBER_ID MyList BLOB MyList BLOB R O W S COLUMN COLUMN FAMILY: MYLIST
  20. 20. Data Model 20161009_1001 20161009_1002 DATE_MEMBER_ID ViewingData BLOB ViewingData BLOB R O W S COLUMN COLUMN FAMILY: VIEWING-HISTORY
  21. 21. Data Model 20161009_1001_0 20161009_1001_1 DATE_MEMBERID_IDX ViewingData BLOB ViewingData BLOB R O W S COLUMN 20161009_1001_2 ViewingData BLOB COLUMN FAMILY: VIEWING-HISTORY
  22. 22. Online A/B Testing
  23. 23. Trending Now Videos that are Trending and Personalized for you
  24. 24. Trending Now It’s 7 PM on a Monday
  25. 25. Trending Now It’s 10 PM on a Saturday
  26. 26. Trending Now Pokeman
  27. 27. Fast Feedback Loop
  28. 28. Trending Now - Data Infrastructure Impression Service Viewing History Service UI Online Services Trends Store Compute Trends Model Training Captures videos shown in view port Captures videos played by members Publish Models Viewing History Service Ratings. .. .
  29. 29. State Management in Cassandra Video Number of Plays Stranger Things 100 Narcos 200 Orange is the new Black 300
  30. 30. State Management in Cassandra Trends Store State Present ? Compute Trends Yes No Init State from Cassandra Load State Update State Read Events
  31. 31. Data Model - Requirements • Trending data is for a specific interval of time • Optimize for Batch Writes and Batch Reads
  32. 32. Data Model 101_METADATA 102_METADATA VIDEOID_METADATA Plays BLOB Plays BLOB R O W S COLUMNS 103_METADATA Plays BLOB COLUMN FAMILY: Interval 1, Interval 2 … Interval N Impressions BLOB Impressions BLOB Impressions BLOB
  33. 33. Roopa Tangirala Engineering Manager @ Netflix Twitter - @roopatangirala
  34. 34. FORKLIFTER
  35. 35. ARCHITECTURE SOURCE TARGET
  36. 36. USE CASES
  37. 37. APACHE THRIFT CQL
  38. 38. DEMO
  39. 39. WHY NOT DSE SPARK?
  40. 40. SCALABILITY
  41. 41. COST EFFECTIVENESS
  42. 42. LESSONS LEARNT
  43. 43. TTL HANDLING • TTL Reading And Writing is Asymmetric - CASSANDRA 12216 • Thrift Column TTL vs CQL Row TTL
  44. 44. 1 6 5 4 3 2 PARTITION DIFFERENCES 500000 200000 425k450k475k 200k175k150k125k 500k
  45. 45. TUNING • spark.cassandra.connection.keep_alive_m s • spark.cassandra.connection.timeout_ms • spark.driver.maxResultSize
  46. 46. OOM EXCEPTIONS Spark.executor.memory spark.cassandra.input.split.size_in_mb
  47. 47. WRITES SPEED SPARK • cassandra.output.batch.size.bytes • cassandra.output.batch.size.rows • cassandra.output.concurrent.writes • cassandra.output.throughput_mb_per_sec
  48. 48. Write Timeouts cassandra.output.throughput_mb_per_sec
  49. 49. QUESTIONS?

×