Whole way of developing and maintaining an LTV model for Crazy Panda game starting from the very rough extrapolation models at the soft launch to more accurate user-based Machine Learning models for mature products. Moreover, we will peek into the main obstacles on our way and how to overcome them. How is LTV calculation different for new games at soft launch phase vs mature products? - Presentation run during on of GameCamp webinars; http://www.gamecamp.io/events/understanding-prediction-ltv/ - All GameCamp webinars: http://www.gamecamp.io/events/

1. DEVELOPING AN EFFECTIVE
LTV MODEL AT SOFT LAUNCH
AND KEEPING IT VALID
FURTHER BEYOND
Ivan Kozyev

2. IVAN KOZYEV
Head of Analytics at Crazy Panda
IVAN KOZYEV
Head of Analytics at Crazy Panda

3. CRAZY PANDA PRODUCTS
World Poker Club
Casual Poker
74M registrations
Stellar Age
MMO Strategy
2.7M registrations
The Household
Social Farm
30M registrations
Pirate Tales
Party Battler
1.5M registrations

4. Math behind LTV
Basic definitions
LTV (also Life-Time Value, CLTV, CLV or LCV) is an amount of the profit
attributed to the entire relationship with a user.
Can be individual or aggregated (average) for a group of users
Individual: LTV = profit from user
Cohort average: LTV = total profit ÷ number of users
Can be actual or predicted

5. Predicting LTV
Game stages
Soft launch
Characterized by:
Small number of users and limited data
Unknown life-time length
Uniform users (in terms of data)
Suggested LTV approach:
General LTV models
Interpolate and extrapolate appropriate curves
Be ready to apply some heuristics

6. Predicting LTV
Game stages
Some time after global launch
Characterized by:
Large amounts of data
Settled user behaviour
Big user diversity
Suggested LTV approach:
Specific LTV models for different use cases
Combination of different approaches to LTV modelling
Machine Learning as a way to increase efficiency

7. Predicting LTV
Characterized by:
Huge amounts of data
Some or most of that data cannot be used for model training
Challenging to keep models up-to-date with new features
Suggested LTV approach:
Having more than one model, working on different sets of features
Regular validation of LTV models predictions
The “slicing” technique
Game stages
Game maturity

8. Game stage: the soft launch
Characterized by
Small number of users and
limited data
Unknown life-time length
Uniform users (in terms of
data)

9. Game stage: the soft launch
Suggested approaches
Interpolate or extrapolate
depending on data
Choose correct metric to
approximate
Work around edge cases:
LTV = F(t)
LTV(tn+1) ≥ LTV(tn)
LTV(t→∞) = LTVmax

10. Game stage: the soft launch
Extra notes
Product knowledge is crucial:
Monetization limits
User behavior
Other options:
LTV = ∫(Ret(t) * ARPU)dt
LTV = ∫(Ret(t) * ARPU(t))dt

11. Model validation
The most important step in LTV model development
Always have a validation sample
Your validation sample must be representative
Do not overfit on validation sample: use it only once per model
Forecast validation - verifying and determining the predictive power of
model forecasts and predictions

12. Some time after global launch
Improving the model
Shortening the confidence
interval for prediction
Lowering the sample size for
reliable results
Reducing the amount of
time/data needed for a
model
$9.5 ± $2.1 → $9.3 ± $0.9
890 users → 214 users
7D of data → 2D of data

13. Some time after global launch
Improving the model
Different LTV models for
different country groups
Different LTV models for
different sources
Different LTV models for
different monetization types
Tier 1 vs Tier 2 vs Tier 3
based on conversions
Google Ads vs video networks
based on optimization
In-apps vs ad-based
based on monetization

14. Some time after global launch
Machine Learning approach
Advantages: Disadvantages:
Can solve and take into
account all dependencies at
once
Can identify very complex
relations
Can give very accurate results
or even per-user predictions
Needs a lot of data for training
Takes time and skill to develop
The LTV itself as a metric is a
very complex target

15. Some time after global launch
Machine Learning approach
Predict shorter periods; for example, 7-, 14- or 28-day ARPU
Use along with existing LTV models
Predict proxy metrics other than LTV: payers, number of
payments, etc
How can we use Machine Learning to build better LTV models?

16. Model improvement
Summary
Set target model metric for improvement: decrease in sample size,
data needed for a prediction, or just general accuracy
Try different approaches: predicting proxy things like payer
conversion or classifying payers by their type
Always think about how the LTV model will be used and what
needs it will be covering
Mind the validation

17. Mature game
Games change with every update, while LTV models tend to become
more and more complex. This complexity plus sudden changes in
product features introduce a whole new challenge: keeping LTV models
valid and up-to-date.
Update is bad → we are losing money due to incorrect
predictions
Update is good → we are losing money due to not scaling our
development or marketing
What could possibly go wrong?

18. Mature game
Dealing with issues
Build a quick new model with rough “soft launch techniques” for
quick validation
Retrain Machine Learning models as soon as possible. It is quite
handy to have a few models using very limited amount of data
Do an A/B test, if possible
Use product knowledge to measure impact on monetization and
changes in user behavior

19. Mature game
Case: Live-Ops events
We started to do Live-Ops
events in our game and have
already done 3 or 4 of them. We
want to understand their
impact on LTV, but for classical
approach we would need 10
times more.
What can we do apart from
waiting?

20. Mature game
The “slicing” technique
Slice the LTV curve
into smaller periods
For each slice, define
a validation cohort
Calculate the impact
on LTV over that period
Normalize those impacts
and calculate the final
improvement
Do not forget about the
novelty effect

21. Mature game
The “slicing” technique
Slice the LTV curve
into smaller periods
For each slice, define
a validation cohort
Calculate the impact
on LTV over that period
Normalize those impacts
and calculate the final
improvement
Do not forget about the
novelty effect

22. THANK YOU
FOR YOUR ATTENTION!
ANY QUESTIONS?
IVAN KOZYEV
Head of Analytics at Crazy Panda
i.kozyev@crazypanda.ru
telegram: @IvanKozyev
https://www.linkedin.com/in/ivan-kozyev/