O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
From Idea to
Execution: Spotify’s
Discover Weekly
Chris Johnson :: @MrChrisJohnson
Edward Newett :: @scaladaze
DataEngConf...
Who are We??
Chris Johnson Edward Newett
Spotify in Numbers
• Started in 2006, now available in 58 markets
• 75+ Million active users, 20 Million paying subscriber...
Challenge: 30M songs… how do we recommend
music to users?
Discover
Radio
Related Artists
Discover Weekly
• Started in 2006, now available in 58 markets
• 75+ Million active users, 20 Million paying subscribers
•...
The Road to
Discover Weekly
2013 :: Discover Page v1.0
• Personalized News Feed of
recommendations
• Artists, Album Reviews, News
Articles, New Releas...
2014 :: Discover Page v2.0
• Recommendations grouped into
strips (a la Netflix)
• Limited to Albums and New
Releases
• Mor...
Insight: users spending more time on
editorial Browse playlists than Discover.
Idea: combine the
personalized experience
of Discover with the lean-
back ease of Browse
Meanwhile… 2014 Year In Music
Play it forward: Same content as the
Discover Page but.. a playlist
Lesson 1:
Be data driven from
start to finish
Slide from Dan McKinley - Etsy
2008 2012 2015
• Reach: How many users are you reaching
• Depth: For the users you reach, what is the
depth of reach.
• Retention: For th...
• Reach: DW WAU / Spotify WAU
• Depth: DW Time Spent / Spotify WAU
• Retention: DW week-over-week retention
Discover Weekl...
2008 2012 2015
Slide from Dan McKinley - Etsy
Step 1: Prototype (employee test)
Step 1: Prototype (employee test)
Results of Employee Test were very positive!
2008 2012 2015
Slide from Dan McKinley - Etsy
Step 2: Release AB Test to 1% of Users
Google Form 1% Results
Personalized image resulted in 10% lift in WAU
• Initial 0.5% user test
• 1% Spaceman image
• 1% Personalized
image
Lesson 2:
Reuse existing
infrastructure in creative
ways
Discover Weekly Data Flow
Recommendation
Models
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
•Aggregate all (user, trac...
1 0 0 0 1 0 0 1
0 0 1 0 0 1 0 0
1 0 1 0 0 0 1 1
0 1 0 0 0 1 0 0
0 0 1 0 0 1 0 0
1 0 0 0 1 0 0 1
•Aggregate all (user, trac...
NLP Models on News and Blogs
Playlist itself is a
document
Songs in
playlist are
words
NLP Models work great on Playlists!
[3] http://benanne.github.io/2014/08/05/spotify-cnns.html
Deep Learning on Audio
•normalized item-vectors
Songs in a Latent Space representation
•user-vector in same space
Songs in a Latent Space representation
Lesson 3:
Don’t scale until
you need to
Scaling to 100%: Rollout Challenges
‣Create and publish 75M playlists every week
‣Downloading and processing Facebook imag...
Scaling to 100%: Weekly refresh
‣Time sensitive updates
‣Refresh 75M playlists every Sunday night
‣Take timezones into acc...
Discover Weekly publishing flow
What’s next?
Iterating on content
quality and interface
enhancements
Iterating on quality and adding a feedback loop.
DW feedback comes at the expense of presentation bias.
Lesson 4:
Users know best. In the
end, AB Test everything!
Lesson 5 (final lesson!):
Empower bottom-up
innovation in your org and
amazing things will happen.
Thank You!
(btw, we’re hiring Machine Learning and
Data Engineers, come chat with us!)
• Reach: How many users
• Reach: How many users
• Reach: How many users
Próximos SlideShares
Carregando em…5
×
211.515 visualizações

Publicada em

• Reach: How many users are you reaching
• Depth: For the users you reach, what is the
depth of reach.
• Retention: For the users you reach, how many
do you retain?
Define success metrics BEFORE you
release your test

Publicada em: Software, Tecnologia

×