These are the slides from Bill Grosso's Keynote at the Conference on Artificial Intelligence in Creative Industries (nucl.ai). The thesis of the talk is that "analytics" has gone from -- "Reporting" which gives information about past events -- to "Business Intelligence" which reframes the reporting in terms of business diagnostics (but is still about past events) -- to "Predictive Analytics" which attempts to predict future events. -- to ... frameworks that can actually alter system behavior in response to predicted outcomes. Scientific Revenue is one of the first of the frameworks. It marries BI and Predictive Analytics to an adaptive system that can alter pricing in order to maximize lifetime revenue. In this talk, which is somewhat technical, Bill discusses the design and architecture of adapative systems, using Scientific Revenue as an example.

1. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 1
Dynamic Pricing In Mobile Games
July 22, 2015

2. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 2
About Me
• CEO and Founder,
Scientific Revenue
• Previously CTO / SVP
Product for Live Gamer
(Payments Aggregator
focused on F2P Gaming )
• Long history in data and
artificial intelligence

3. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 3
The Last AI Conference I Spoke At ….

4. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 4
About Scientific Revenue
Scientific Revenue provides a dynamic price management solution for mobile
games that boosts in-app purchase revenue. We match the right prices with
the right players at the right times, to keep players engaged, increase
conversion, and grow profits for game publishers.

5. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 5
What Heather Said

6. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 6
Earlier Today ….
• The heavy lifting going on around knowing what
players are doing has to do with prediction and
classification
• Classic territory
• Systems today are more strongly on detection and
diagnosis, not action side
• We’re starting to get solid predictive analytics
• Scientific Revenue is about a control framework
• Well, that and some pretty nice machine learning.
• This talk is mainly about control frameworks.

7. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 7
What is a Price

8. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 8
An Offer To Sell a Good or Service for “Real Money”

9. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 9
An Offer To Sell a Good or Service for “Real Money”

10. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 10
Lots of Decisions
Different:
• Prices
• Coin amounts
• (volume discounts)
• Framing text and cues
• Default selection
• Different bonus types
Same:
• Call to action

11. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 11
These are Also “Pricing Decisions”

12. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 12
Not Just “How Much for How Much”
• Pricing decisions are also
• What additional inducements do you offer?
• When do you make the offer?
• What channels you make the offer in?
• What messages accompany the offer?
• How long the user has to act on the offer?
• ….

13. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 13
The Problem

14. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 14
Increase LTR (“R” = “Revenue”)
• Pricing optimization is a tool to revenue maximization
• Without causing adverse reactions
• NOT Looking at things at the “individual transaction
level”
• For games with very high churn and very short
retention times, these approaches overlap
• But if you’re keeping your users around and hoping for
more revenue later (second purchases, advertising, …)
then there are other aspects to consider

15. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 15
LTR
• 20% of all purchases occurred on day 1
• All spending was done by day 40
• 27% of all first purchases occur on day 1
• 80% of all first purchases occur in week 1
• 49% of purchasers bought a second time
• Half of all purchases occurred in week 1
• Second purchases were the same size as first
purchases

16. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 16
The Settled Science isn’t Very Useful
• Classical Economics involves pricing to the demand
curve (and, maybe, estimating demand curves using
multi-armed bandits)

17. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 17
Reality is Actually …. Kind of Unsettled
• Training effects?
• Framing Effects?
• Volume discounts?
• Churn Impacts?
• Community Impact?
• Moral Hazard?

18. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 18
Moral Hazard
This is absolutely disgusting. I'll be sure to tell everyone
about this creepy, exploitive tracking of players.

19. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 19
Our Predecessors Made Many Simplifying Assumptions ….
• Non-negotiated pricing
• Flexible return policy
• Segmentable market demand
• Highly competitive markets / little or no vendor
loyalty
• Publically available ratecards
• Pre-existing anchoring on pricing and rates
• Infrequent, large-dollar amount purchases
• Customers return months or years later
• Low variable costs
• Fixed capacity
• Inventory can be changed from one product to
another
• Perishable inventory

20. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 20
The Architecture

21. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 21
In Block Form

22. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 22
Three Distinct Requirements
• Data Collection
• This isn’t an algorithm problem. It’s a modeling and feature problem
and it requires a well designed data set informed by the machine
learning goals
• Control Framework
• The point is to change prices. By itself, that’s actually pretty hard to
do (in the “lot of code” sense)
• Asynchronous Evaluation Framework
• Most of our model building and training is done asynchronously

23. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 23
Evaluation Framework: Global Cycle
Calibrate
Measure
Diagnose
Propose
Promote Test

24. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 24
Evaluation Framework: Calibration
• Solving “Cold Start”
• Have a canned set of 70+ segments (that are “known”
to exhibit pricing and behavioral differences).
• Have a predefined set of 250+ additional features
• Have a diagnostic framework that can exhaustively
measure a large number of metrics and which can
evaluate features for predictive power

25. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 25
Example: Part of Day / Day of Week
• Left: Number of new users by date and hour, lighter = more
• Right: Number of purchasers by date and hour, lighter = more
• People who join at noon are 4 times more likely to spend than people who join at 5 AM (in this
game)

26. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 26
Evaluation Framework: Calibration
• Run for three weeks to train the models against the
initial segments and get baseline performance data
• Compare the initial segments to each other to get an
idea of variation and benchmark good performance
• Spot “underperformers” and “overperformers”
• Automated diagnostics to explore reasons why

27. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 27
Directional Metrics
• Traditional KPIs can indicate issues,
but don’t help much with action
• ARPU dropping? What is the
automated, or partially automated,
outcome?
• Part of the power of our approach
comes from putting features against
finer-grained behavioral metrics
• And then automating the sifting
• Example:
• Purchase Index
• Default Acceptance Ratio
• Upsell %
• Downsell %

28. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 28
Evaluation Framework: Segmentation
• Predictive Analytics
• Churn Prediction, Likely To Purchase, Potential Whale, …
• Custom Models
• Disposable Income, Affluence, Gamerness, Mobile Native Ness, …
• “Possibly Important” Features
• Lots of these
• Propose Segments and Pricing Policies
• Based on important features, create segments and compare

29. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 29
Evaluation Framework: User Lifecycle
(start) Join
Initial
Profile
Baseline
Modeling
Baseline
Prediction
Observe Adjust
Initial
Pricing Set
Reprice

30. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 30
Dealing With Intuitive Ideas

31. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 31
Intuitive Physics

32. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 32
Intuitive Economics

33. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 33
What is Stickershock
A feeling of surprise and disappointment caused by
learning that something you want to buy is very
expensive.
Astonishment and dismay experienced on being
informed of a product’s unexpectedly high price.

34. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 34
Formalizing Stickershock
Before:
• D0 to D3 timeframe
• A user visits a payment wall (or purchase opportunity) early in their lifecycle (and unusually
early)
• We have other signals that they are likely to buy (usually behavior-oriented)
During:
• They don’t buy
• They abandon relatively quickly
After:
• They don’t come back to the payment wall
• They either grind or leave the game entirely
Supporting Evidence:
• Low affluence signals

35. © 2015. Company Confidential and Not for Redistribution. info@scientificrevenue.com 07/22/15 35
Options for Dealing with Stickershock
• Give out more currency early
• Initial Framing Offer
• Targeted Intervention
• Reorder baseline prices and reset default
• Different set of Baseline Prices
• Different set of Baseline Prices with windowing