Improving LTV with Personalized Live Ops Offers: Hill Climb Racing 2 Case Study | Jakub Marek

1. Improving LTV with Personalized LiveOps Offers
Approaches & Hill Climb Racing 2 Case Study
Jakub Marek

2. London | 28-30 May 2019
Hello!
I am Jakub Marek
Head of Big Data & LiveOps @ SuperScale
jakub.marek@superscale.com

3. “We’re forming growth partnerships with world’s top developers
to scale their games to maximum potential.”

4. London | 28-30 May 2019
● 60+ World Class Experts In-House
○ Creatives
○ UA Campaign Management
○ Analytics & Data Science
○ Business Intelligence
○ LiveOps Optimization
○ Game & Monetization Design
● Founded in 2016
● Bratislava, London, Berlin, Helsinki &
Prague offices
First Light Games
Our partners
Games

5. London | 28-30 May 2019
LiveOps Optimization
Motivation

6. London | 28-30 May 2019
How can we improve monetization in games?
Huge topic - lots of angles
● Core Game design
● Meta game - retention systems
● Better onboarding
● Economy Balancing
● New content pipeline
...
● LiveOps Offers - focus of this talk

7. London | 28-30 May 2019
Each player gets either the same price and content in an offer or gets to
choose from 3-5 similar offers.
There is no consideration of player’s:
● payment potential
● spending habits
● in-game behavior
This results in lower conversion
& lower satisfaction of players
Offers in most games are not maximizing revenue.
= missed opportunity => lower revenue.

8. London | 28-30 May 2019
The Question
How can we improve the monetization through special offers
● using already existing content in the game?
● without impacting existing monetization systems?
=> by delivering relevant content at relevant time for a relevant price
How to actually do it?
● utilize segmentation - targeting players on a personal level

9. London | 28-30 May 2019
Approach: better monetization of existing game content through
personalised pricing, content and delivery
+52%
LTV Uplift from baseline
Sneak peek: results from Hill Climb Racing 2

10. London | 28-30 May 2019
LiveOps Optimization
Approach

11. London | 28-30 May 2019
Base model for LiveOps offers
STP Model
Well describes the process in
these questions:
● WHO?
● WHAT?
● HOW MUCH?
● HOW OFTEN?

12. London | 28-30 May 2019
Segmentation - Naive approach
Simple rule-based
segmentation based on LTV
Advantages:
● Easy to understand
● Most widely used
● Enables basic targeting

13. London | 28-30 May 2019
Recency, Frequency and Monetary value
● Directly models purchasing behavior
● Great for predicting purchase effectiveness
● Easy to understand and apply in content
creation process
● Basis for targeted offers
● Still no answer to WHAT?
RFM segmentation
Segmentation - Business Analytics Approach (Purchasing Behavior)

14. London | 28-30 May 2019
Segmentation - Business Analytics Approach (Gameplay Interaction)
Gameplay Interaction Segmentation
Players in the game are not the same -
● They unlock different content
● Progress to different points in the story
● Different split between game modes
● Different spending habits

15. London | 28-30 May 2019
Best results = Combining both segmentation approaches
Purchasing Behavior Segmentation Gameplay Interaction Segmentation

16. London | 28-30 May 2019
Case Study: Hill Climb Racing 2
Game intro

17. London | 28-30 May 2019
Case study: Hill Climb Racing 2
● Fingersoft
● Released Q4 2016
● iOS & Android
● Multiplayer racing game
● 150+ million players
.. and how we delivered +52%
true LTV uplift.

18. London | 28-30 May 2019
Case study: Hill Climb Racing 2 - core game

19. London | 28-30 May 2019
Case study: Hill Climb Racing 2 - car collection & upgrading

20. London | 28-30 May 2019
Case study: Hill Climb Racing 2 - gacha (tuning parts)

21. London | 28-30 May 2019
Case Study: Hill Climb Racing 2
Overview

22. London | 28-30 May 2019
Pre-personalization monetization in HCR2
● direct gem pack purchases
● rank-based offers (tied to progression)
● special offers (offers with skin and good value, rotating in shop)
● seasonal offers (Halloween, Christmas, Back to school)

23. London | 28-30 May 2019
Goals
1. Increase revenue through personalization
2. Deliver to players relevant content at relevant time for relevant price
3. Do not harm user retention
4. Minimize “cannibalization” of future profits

24. London | 28-30 May 2019
What are we actually aiming for?
Minimize
additional value
(value multiplier)
Increase revenue
per user
Price
Content distribution
Value
Availability
Offer sets and their
sequence
Visual aspects
Optimization
=> The value is in the personalization!

25. London | 28-30 May 2019
Business analytics
(BQ + Periscope + Python/R)
BI analyses
(monetization and gameplay
data)
Machine learning models
(probability modeling +
XGBoost + others)
Exporting segments
(offer config + user ID
pairings in BigQuery)
Analytics server LiveOps server Game client
Downloading offer config +
user ID from analytics
server
Distributing correct offer
configs to users
Displaying personalized
popup offers
Sending analytics data to
BigQuery
(popup + IAP data)
Evaluation of results
Automatic setup of popup
offer definitions
Defining target payer
segments & offer contents
LiveOps Optimization - Infrastructure

26. London | 28-30 May 2019
LiveOps Optimization - Modules
Players profile
(custom pipeline)
Vehicles progression
(mining or more diverse, frequency,
recency ..)
Models Offers sequence
method/models
Pricing model maximizes
revenue per impression
● Adjust conversion per price point
● Weights impression and purchases
over time
● Calculates revenue coefficient for
each player
● Adapts over time
● Automatic upselling
Predefined
sequence
Based on player’s
preference
(best converting sequence of offers
based on particular player
preference)
Behavioral preference model
● Takes into account all behavioral
data which defines future behavioral
● Creates set of personalized features
to choose from to increase variability
of offers
Upgrading preferences
(vehicle, amount, recency.. )
Tuning parts usage
(frequency,amount coins and scraps,
vehicle ..)
Purchasing behavior
(products distribution, price,frequence,
recency, amount.. )
Currency spending
(both soft and hard currency)
Rank progression
(rank, seasonal rank, teams)
Automatic value definition
model
● Automatic value definition (needed for
deeper personalization)
Modes preference
(challenges, adventures, events ..)
A/B testing tools
Behavioral based
(vehicle chests distribution,
amount of hard currency ..)
Monetary based
(waterfall, upselling ..)
Support tools
Automatic evaluation of
iterations
(ARPU, ARPPU, conversion, revenue,
impressions ..)
Subsegment definition
● Resources
(hard and soft currencies)
● Visuals
(backgrounds, titles, colors,
value multipliers)
User interface
based
(different value multiplier,
discount..)
Automatic offer
generation

27. London | 28-30 May 2019
Progression defines needs of players
Players progress through various flows - for example:
Step 1
Vehicle purchase
with soft currency
Vehicle upgrading until all
soft currency is depleted
Step 2
Step 3
Joining event and seeing what
vehicles winners are playing
Step 4
Opening race chests
and seeing rare card
Purchase
Player purchases bundle with
new vehicle from event, skin
and additional vehicle chest
Can we find these patterns and learn from our existing data?

28. London | 28-30 May 2019
Top challenges when designing offers
1. Understanding player behavior
○ What are player’s needs/preferences at the moment? What about their payment potential?
2. Creation of offers
○ How to keep things fresh and rotate through similar offers over and over again?
3. Evaluation, AB testing and continuous improvement
○ How to work towards long-term success?
4. Avoiding “cannibalisation” of future profit
○ How much additional value players should get?

29. London | 28-30 May 2019
Top challenges when designing offers
1. Understanding player behavior
○ What are player’s needs/preferences at the moment? What about their payment potential?
2. Creation of offers
○ How to keep things fresh and rotate through similar offers over and over again?
3. Evaluation, AB testing and continuous improvement
○ How to work towards long-term success?
4. Avoiding “cannibalisation” of future profit
○ How much additional value players should get?

30. London | 28-30 May 2019
First personalization model - rule based
RFM
(recency + LTV + avg. purchase)
Behavioral segmentation
(using only 2 major features - most
played & upgraded vehicle)
In first iterations we used data to understand player behavior, distributions and created rule
based segments which led to 40% improvement over the baseline during offer weekends.

31. London | 28-30 May 2019
First machine learning implementation
Monetary segmentation
(Modified RFM + highest previous
purchase)
Behavioral segmentation
(Agglomerative hierarchical clustering)
Prediction of chest type
(Random forest - feature importance)
Understanding monetary possibilities
of players
Understanding players preferences
in soft currency spending
Additional personalization of
chest content
Introduction of machine learning models led to even higher improvement at 108% improvement over
the baseline during offer weekends.

32. London | 28-30 May 2019
Current data science model (+52% true LTV uplift)
Content definition based on
behavior in a game
Extreme gradient boosting (XGBoost)
Combination of soft and hard currency
spending with upgrading behavior, tuning
behavior and purchasing behavior
RFM
BI method
Probability modelling
Price conversion modelling with adaptation
and revenue maximization function
Understanding payment
potential through monetary
parameters of player
Understand which price points and
sequence of price points maximizes
revenue from each player individually.

33. London | 28-30 May 2019
Takeaways (segmentation)
1. LTV & frequency of purchasing are more important than recency.
2. Setting price of offer based on player’s previous highest purchase is a good starting
point.
3. Finding a way how to offer player content he uses/interacts with the most without
breaking your existing systems is essential (and should be your starting point as well).

34. London | 28-30 May 2019
Top challenges when designing offers
1. Understanding player behavior
○ What are player’s needs/preferences at the moment? What about their payment potential?
2. Creation of offers
○ How to keep things fresh and rotate through similar offers over and over again?
3. Evaluation, AB testing and continuous improvement
○ How to work towards long-term success?
4. Avoiding “cannibalisation” of future profit
○ How much additional value players should get?

35. London | 28-30 May 2019
Elements of offers to optimize
Amount of
resources
Additional value
Offer price
Availability
Type of chests
Visuals & copy

36. London | 28-30 May 2019
Takeaways (creation of offers)
1. Changing the “theme” of offer (gacha focus, currency focus, variety of offered items) to keep offers
from feeling stale (on pictures are offers from the same iteration)
2. Varying the value multiplier (while overall high value multiplier introduces cannibalization there is a
segment of players which reacts only to high-X-value offers)

37. London | 28-30 May 2019
Top challenges when designing offers
1. Understanding player behavior
○ What are player’s needs/preferences at the moment? What about their payment potential?
2. Creation of offers
○ How to keep things fresh and rotate through similar offers over and over again?
3. Evaluation, AB testing and continuous improvement
○ How to work towards long-term success?
4. Avoiding “cannibalisation” of future profit
○ How much additional value players should get?

38. London | 28-30 May 2019
Iterative improvement of performance - AB testing
AB test categories
● Pricing
○ New $80 price point introduced (top payers LTV +36%)
● Additional value multiplier optimization
○ Better personalization + lower value = kept good performance
● Segment definition
○ Waterfall model testing (moving players between price points)
○ Adaptation of personalization to unique play-styles (e.g. current vehicle or
game mode preference)

39. London | 28-30 May 2019
Iterative improvement of performance - ad hoc analyses
Continuous improvement that does not negatively affect other aspects of the game.
Monitoring cannibalization
development over time on different
level of detail
(segment/subsegment/offer etc.)
Monitoring community
discussions on different platforms
(e.g. Discord)
Monitoring performance over time,
comparing iterations of offers with
different content distribution per
segment/subsegment

40. London | 28-30 May 2019
Takeaways (evaluation & continuous improvement)
1. Ask the right questions from data
if the answer doesn’t change what you will do ask a different question
2. AB tests are expensive - test for major things first
new segmentation method is more important than testing new background
3. Your community is probably active somewhere - try to find those players and see
how they perceive offers they are getting.

41. London | 28-30 May 2019
Top challenges when designing offers
1. Understanding player behavior
○ What are player’s needs/preferences at the moment? What about their payment potential?
2. Creation of offers
○ How to keep things fresh and rotate through similar offers over and over again?
3. Evaluation, AB testing and continuous improvement
○ How to work towards long-term success?
4. Avoiding “cannibalisation” of future profit
○ How much additional value players should get?

42. London | 28-30 May 2019
Result after 33 iterations
+41%
Total revenue growth
But how much have we impacted the baseline?
Is there some cannibalization?

43. London | 28-30 May 2019
Evaluation - True LTV uplift calculation
● True LTV uplift = LTV uplift after consideration (and deduction) of baseline cannibalization
→ For the correct calculation of the LTV uplift we need to establish long-term control
group(s) (i.e. to run a long-term AB test)
● Preceding every iteration we split new payers 90/10
=> 90% of the new payers start receiving personalized offers
=> 10% will never receive any personalized offer, but they will still be receiving all other offers
(seasonal, rank-up offers etc.)
● We do the same for non-payers
before each iteration we find all new players who joined the game and split them 90/10 as well

44. London | 28-30 May 2019
Evaluation - Payers - 49% true LTV uplift
LTV calculation
LTV = cumulative revenue
of a group / cumulative
number of distinct players
+ 49% true
LTV UPLIFT
21% Baseline
cannibalisation
(payers w/o offers out
of control revenue)
+84%
LTV uplift

45. London | 28-30 May 2019
Evaluation - Non-payers (new players) - 58% true LTV uplift
+ 58% true LTV UPLIFT
+ 25% baseline LTV over time
Hypothesis
1. Personalized offers increase
engagement of players over time
2. Coins spending analysis supports this,
overall upgrading and tuning of vehicles
increased for players who are purchasing
personalized offers
LTV calculation
LTV = cumulative revenue of a group /
cumulative number of distinct players

46. London | 28-30 May 2019
Takeaways (evaluation & cannibalization)
1. Cannibalization is not something you can ignore
test it through proper AB testing.
2. Offering hard currency in offers can introduce huge cannibalization of the existing
monetization systems
proceed carefully or drop all together
3. Making players happy with their first purchase might cause zero cannibalization and
even increase LTV further.

47. London | 28-30 May 2019
Thank you for attention.
Questions?
jakub.marek@superscale.com