Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a LTV prediction - How to save millions with ML
Semelhante a LTV prediction - How to save millions with ML (20)
Último
Último (20)
LTV prediction - How to save millions with ML
- 1. LTV prediction HOW TO SAVE MILLIONS WITH ML
- 2. Practical info Slides and recording will be available after the talk Please use Slido 2020-05-28 2
- 3. Who am I? Senior Data Scientist at SuperScale 4 years of experience with data in gaming Started in Pixel Federation • Objective feedback to production team • Innovative statistical methods to improve the games Studied applied mathematics 2020-05-28 3
- 4. SuperScale Growth partner for developers and publishers 2020-05-28 4
- 5. SuperScale Growth partner for developers and publishers 2020-05-28 5
- 7. How does F2P business work? You can play for free Game is monetized with • In-app purchases (only 2%) • Ads
- 8. We need a lot of players to be successful How does F2P business work? Game Organic searchMarketing
- 9. Cost per player (CPI) Revenue per player - customer lifetime value (LTV) Goal of marketing is to grow while being profitable LTV > CPI How does F2P business work?
- 10. LTV > CPI CPI is easily calculated But what about LTV? • You must wait months or years to calculate it • But that is not the only problem… How does F2P business work?
- 11. How much can we spend? Optimization between scale and profitability new players CPI LTV PROFIT profitperplayer
- 12. How much can we spend? Optimization between scale and profitability new players CPI LTV PROFIT
- 13. How much can we spend? Optimization between scale and profitability new players CPI LTV MAXIMUM PROFIT
- 14. Spend optimization Optimization between scale and profitability • Spend too slow lose profit • Spend too fast lose profit When spending millions you want to know your LTV
- 15. Spend optimization It is not just one optimization problem You have several marketing channels • Facebook • Google • Instagram • And many more… Granularity can go much deeper • This leads to small samples and high variance
- 16. Business summary LTV can save us literally millions of dollars
- 18. Google cloud infrastructure Takes care of automatic scaling Game raw data Marketing data Data warehouse BigQuery Batch preprocessing Dataflow Machine Learning ML Engine AI Platform Model storage Cloud Storage Scheduler Cloud Functions Analytics & BI Data Studio Colab
- 19. Define the prediction problem The most important step in modelling • What specifically are we predicting? • What are the training/testing data? • How do we evaluate the predictions? To answer these, we must specify the final product
- 20. Define the prediction problem Important questions • How will the model be used? • Will the predictions change anything? What? Answers determine • What kind of model should be used • How should the model be tuned and evaluated
- 21. Possible use-cases for LTV prediction Company financial planning • Simple aggregated model based on averages is enough
- 22. Possible use-cases for LTV prediction Company financial planning Prioritize customer support • Identification of VIP players as soon as possible • Classification is more appropriate than regression
- 23. Possible use-cases for LTV prediction Company financial planning Prioritize customer support Budget allocation between marketing channels • Absolute values are not as important as relative comparisons • Variation is more important than bias
- 24. Possible use-cases for LTV prediction Company financial planning Prioritize customer support Budget allocation between marketing channels Optimize scale of marketing • Spend more or not? • Absolute error is important since it is compared to costs
- 25. Optimize scale of marketing We start a campaign in US on Facebook How is the campaign performing? • Should we scale up? • Or cut back on spending?
- 26. Optimize scale of marketing Check data • Each date has different distribution of players • Not very useful
- 27. Optimize scale of marketing Look at it by install date cohort
- 28. Optimize scale of marketing Look at it by install date cohort
- 29. Optimize scale of marketing Look at it by install date cohort
- 30. Optimize scale of marketing Look at it by install date cohort
- 31. Optimize scale of marketing Look at it by install date cohort • Distribution of players is stable • Better for making prediction
- 32. Define the prediction problem Predict cohort revenue using the first days of data • Prediction horizon is usually 1 year
- 33. Define the prediction problem Will it end here? Something like this
- 34. Define the prediction problem Here?
- 35. Define the prediction problem Or here?
- 36. How can we predict this? Best case scenario • We have more than 1 year of data • The game has stable influx of similar players We can estimate the curve using old cohorts and fit it to new players
- 37. Best case scenario We can estimate the curve using old cohorts and fit it to new players
- 38. Best case scenario Add confidence intervals and we are…
- 39. Best case scenario Add confidence intervals and we are… done!
- 40. Easy, right? Well, usually not so much • We do not have enough data ?
- 41. Easy, right? Well, usually not so much • We do not have enough data
- 42. Easy, right? Well, usually not so much • We do not have enough data • The curve changes in time
- 43. Easy, right? Well, usually not so much • We do not have enough data • The curve changes in time • There is huge variance in cohorts
- 44. The model works but You need big cohorts to counter the variance You need time (data points) to get stable prediction We want to do better!
- 45. How to improve? Variance will always be there unless we sacrifice granularity But we did not use all available data yet • We have detailed behavioral data for each player ML models can be useful here
- 46. How to train useful ML model Before you dump everything in a deep NN… Think about the prediction problem and its use case • Predict 1-year revenue using first days (e.g. 7 days) Straightforward approach • One cohort – one row in dataset • All data from 7 days are columns of feature matrix • 1-year revenue is the target • Train the model • ??? • Profit!
- 47. How to train useful ML model Even if you get a good model, there is a problem You had to exclude all cohorts from last year Gaming industry is changing all the time Try explaining that to the end user What now?
- 48. How to train useful ML model One possible solution • Change the target to shorter horizon • Add the prediction to your original data and fit the curve again • This can improve your early prediction performance data ML model result Improved fitted curve
- 49. How to communicate results You have a great model, but you must convince others • Different stakeholders have different interests How did we convince EA that our model is better?
- 50. How to communicate results Definitely not like this Electronics Arts SuperScale RMSE 195 036 28 487
- 51. How to communicate results We visualize the usefulness leading to better business decisions 30% change compared to day 7 prediction 10% change compared to day 7 prediction
- 52. How to communicate results Accuracy is not enough in real applications • Bias • Variance • Stability in time
- 53. Final notes We simplified a lot • There are a lot questions, uncertainties and practical difficulties • You must make a lot of impactful decisions along the whole process This leads to numerous variations of the final model
- 54. How we look at it There are • a lot of different models with different advantages • a lot different games with different monetization One model cannot incorporate everything Our approach to LTV modelling • Create an ensemble of models that works on any game • Use the models to make automatic recommendations for marketing optimization
- 55. Conclusion LTV predictions can save you a lot of money Basic predictions are easy, but it is hard to use the full potential Making useful ML model is never straightforward Think first about how the final model will be used • This leads to appropriate model definition and evaluation strategy