O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Consumer Science and Product Development at Netflix - OSCON 2012

6.026 visualizações

Publicada em

At the core of product innovation at Netflix is consumer science; Netflix constantly tests and iterates on new ideas for every aspect of the streaming service. This experiment-oriented culture uses data from actual customer usage of the product to quickly understand which ideas are great.

At the 2012 O'Reilly Open Source Convention (OSCON), Rochelle King and Matt Marenghi described the consumer science process in detail. They demonstrated how it can apply to everything from a new UI concept to a new back-end algorithm. They also discussed how the company culture and the product development approach inter-relate, and shared how this customer focus drives decision-making at every level of the company. They also discussed some of the limitations of consumer science, and how to make decisions where science can’t easily shine a light.

  • Entre para ver os comentários

Consumer Science and Product Development at Netflix - OSCON 2012

  1. 1. Consumer Science &Product Development
  2. 2. Rochelle KingVP, User Experience & Product Services Matt Marenghi VP, User Interface Engineering
  3. 3. Netflix Overview
  4. 4. TV & MovieEnjoyment Made Easy
  5. 5. ~26 Million Members
  6. 6. ~26 Million Members
  7. 7. Over800PartnerProducts
  8. 8. 8
  9. 9. 9
  10. 10. 10
  11. 11. 11
  12. 12. 12
  13. 13. 13
  14. 14. ? 13
  15. 15. Netflix & Consumer Science“If you want to increase your success rate, double your failure rate.” – Thomas Watson, Sr., founder of IBM
  16. 16. Goal:Customer Satisfaction 15
  17. 17. Measuring Success 16
  18. 18. 17
  19. 19. Consumer Science 17
  20. 20. Consumer Science 17
  21. 21. Consumer Science 17
  22. 22. Consumer Science 17
  23. 23. Consumer Science A/B testing 17
  24. 24. What “performs best”? 18
  25. 25. Choosing the Right Metrics 19
  26. 26. Choosing the Right Metrics Core Metric: Retention 19
  27. 27. Choosing the Right Metrics Core Metric: Retention 20
  28. 28. Choosing the Right Metrics Core Metric: RetentionProxy Metric: Hours Watched 20
  29. 29. Start with a Hypothesis... 21
  30. 30. Start with a Hypothesis... If we make ahuge “play” button, people will watch more. 21
  31. 31. Start with a Hypothesis... If we make ahuge “play” button, people will watch more. If we give people $1 every time they press “play”, retention will improve. 21
  32. 32. Start with a Hypothesis... If we make a huge “play” button, people will watch more. If we give people $1 every time they press “play”, retention will improve. Showing moremovies & TV shows will lead tomore streaming and improved retention. 21
  33. 33. Start with a Hypothesis... If we make a huge “play” button, people will watch more. If we give people $1 every time they press “play”, retention will improve. Showing moremovies & TV shows will lead tomore streaming and improved retention. 22
  34. 34. Determine the Variables Showing moremovies & TV shows will lead tomore streaming and improved retention. 23
  35. 35. Determine the Variables Showing moremovies & TV shows will lead tomore streaming and improved retention. 23
  36. 36. Determine the Variables 24
  37. 37. Determine the Variables more titles per rowmore rows 24
  38. 38. Determine the Variables more titles per row Depth vs.more rows Breadth 24
  39. 39. Design the Test Control 25 rows x 75 titles 25
  40. 40. Design the Test Control 25 rows x 75 titles 25
  41. 41. Design the Test Control 25 rows x 75 titles Cell 125 rows x150 titles 25
  42. 42. Design the Test Control 25 rows x 75 titles Cell 1 Cell 225 rows x 31 rows x150 titles 75 titles 25
  43. 43. Design the Test Control 25 rows x 75 titles Cell 1 Cell 2 Cell 325 rows x 31 rows x 31 rows x150 titles 75 titles 150 titles 25
  44. 44. Design the Test Control 25 rows x 75 titles Cell 1 Cell 3 Cell 225 rows x 31 rows x150 titles 31 rows x 150 titles 75 titles 25
  45. 45. 26
  46. 46. Level theplaying field 26
  47. 47. Level the Data fromplaying field real customers 26
  48. 48. Level the Data from Align to coreplaying field real customers metrics 26
  49. 49. Large scale concept testingcan provide general direction
  50. 50. Original PlayStation 3 UI
  51. 51. Original PlayStation 3 UIHow can we get our customers to watch more?
  52. 52. Cell 1:Browsing more titles using a flexible menu system and hierarchy will lead to more viewing
  53. 53. Cell 2:A simple, flat interface which focuses on content will lead to more viewing
  54. 54. Cell 3:Separating navigation from content will guide ourmembers to the content and lead to more viewing
  55. 55. Cell 4:A video-rich browsing experience will lead to more viewing
  56. 56. VOTE! Cell 1: Cell 2:Hierarchy Grid Cell 3: Cell 4:Separation Video
  57. 57. And the winner is... Cell 1: Cell 2: Control Grid Cell 3: Cell 4:Separation Video
  58. 58. Iterate...
  59. 59. Iterate...
  60. 60. Iterate...
  61. 61. Iterate...
  62. 62. Iterate...
  63. 63. Iterate...
  64. 64. Iterate...
  65. 65. Iterate...
  66. 66. Data can give you confidence in your decisions
  67. 67. HypothesisA cleaner UI which showcases the content will lead to more viewing.
  68. 68. Cell 0: Cell 1:Control Clean
  69. 69. larger boxes
  70. 70. no titles
  71. 71. on hover
  72. 72. ResultsCell 0: Cell 1:Control Clean 39
  73. 73. ResultsCell 0: Cell 1:Control Clean 39
  74. 74. Roll Out! 40
  75. 75. Roll Out! but... 40
  76. 76. Roll Out! but... “NO GOOD...it SucKs BIG TIME...plz change back” 40
  77. 77. Roll Out! but... “NO GOOD...it SucKs BIG TIME...plz change back” “I am hoping that at least oneperson at Netflix with authority will put down the crack pipe...and go back to the old interface” 40
  78. 78. Roll Out! but... “NO GOOD...it SucKs BIG TIME...plz change back” “I am hoping that at least oneperson at Netflix with authority will put down the crack pipe...and go back to the old interface” “I don’t like it, where is the sortable list? and I can’t stand the scroll it’s just wierd and stupid...” 40
  79. 79. Respond... 41
  80. 80. Today
  81. 81. Making Decisions
  82. 82. Dealing With Results 44
  83. 83. Dealing With ResultsRoll it out 44
  84. 84. Dealing With Results Roll it outWith consideration to thechange effect on users 44
  85. 85. Dealing With Results Roll it out Move OnWith consideration to thechange effect on users 44
  86. 86. Dealing With Results Roll it out Move OnWith consideration to the Polish won’t make it turnchange effect on users positive 44
  87. 87. When the World is Flat 45
  88. 88. When the World is FlatUnsure of Value Retest? 45
  89. 89. When the World is Flat Unsure of Value Retest? If there’s a specificconcern, address it and consider retesting 45
  90. 90. When the World is Flat Unsure of Value Value Add Feature Retest? Roll out? If there’s a specificconcern, address it and consider retesting 45
  91. 91. When the World is Flat Unsure of Value Value Add Feature Retest? Roll out? If there’s a specific but...concern, address it and - Ongoing tax consider retesting - Likely to constrain future innovation 45
  92. 92. Pitfalls 46
  93. 93. Pitfalls• A/B testing becomes a crutch for decision making 46
  94. 94. Pitfalls• A/B testing becomes a crutch for decision making• Not getting a clear signal 46
  95. 95. Pitfalls• A/B testing becomes a crutch for decision making• Not getting a clear signal• Too many variations 46
  96. 96. Pitfalls• A/B testing becomes a crutch for decision making• Not getting a clear signal• Too many variations• Local maximum problem 46
  97. 97. Pitfalls• A/B testing becomes a crutch for decision making• Not getting a clear signal• Too many variations• Local maximum problem• Declaring victory too soon 46
  98. 98. Pitfalls• A/B testing becomes a crutch for decision making• Not getting a clear signal• Too many variations• Local maximum problem• Declaring victory too soon• Not knowing when to end a test 46
  99. 99. Culture of Consumer Science
  100. 100. + 48
  101. 101. Approach + 48
  102. 102. Approach + People 48
  103. 103. 49
  104. 104. Fostering the Culture 49
  105. 105. Fostering the Culture• Universally embraced 49
  106. 106. Fostering the Culture• Universally embraced• Common vocabulary 49
  107. 107. Fostering the Culture• Universally embraced• Common vocabulary• Be disciplined 49
  108. 108. Fostering the Culture• Universally embraced• Common vocabulary• Be disciplined• Share results broadly 49
  109. 109. Fostering the Culture People Matter• Universally embraced• Common vocabulary• Be disciplined• Share results broadly 49
  110. 110. Fostering the Culture People Matter• Universally embraced • Humble• Common vocabulary• Be disciplined• Share results broadly 49
  111. 111. Fostering the Culture People Matter• Universally embraced • Humble• Common vocabulary • Focused• Be disciplined• Share results broadly 49
  112. 112. Fostering the Culture People Matter• Universally embraced • Humble• Common vocabulary • Focused• Be disciplined • Data-driven• Share results broadly 49
  113. 113. Fostering the Culture People Matter• Universally embraced • Humble• Common vocabulary • Focused• Be disciplined • Data-driven• Share results broadly • Curious about business 49
  114. 114. “We are what we repeatedly do.Excellence, then, is not an act but a habit.” – Aristotle
  115. 115. Questions? Rochelle King - roking@netflix.com Matt Marenghi - mmarenghi@netflix.comPS - Interested in learning more first hand? We’re hiring designers and engineers!
  116. 116. END

×