2. Understanding and
improving games through
machine learning
Natasha Latysheva
Data Scientist
Jagex Games Studio

3. This talk:
1. Introduction to Jagex
2. Recap of main machine learning
fields + examples
3. Fun ML in video games in general
4. ML projects at Jagex Data Science
• Identifying player personas
• Deep learning player lifecycle
sequences
• Automatic bot and abuse
detection
• Trading card deck analysis
• Quest recommender systems

4. Jagex Game Studios
• Video game development company based in
Cambridge
• Best known for Runescape, the world's
largest free-to-play massively multiplayer
online role-playing game (MMORPG). First
released in 2001!
• Extremely community oriented game
• Old School Runescape was released in 2013

5. RuneScape
Main..

6. .. and
RuneScape
Old School

7. TF2 meets
Minecraft!

12. A quick machine
learning recap

13. Machine learning recap
Essentially learning patterns from data.
Branches of ML:
• 1. Supervised learning.
• Most familiar names, e.g. linear regression,
logistic regression, decision trees, random
forests, SVMs, many neural nets.
• Algorithm is presented with input data and
explicit labels on that input. Learn signals
associating input data with labels.
• Everything from good ole iris classification to
detecting diseased patients from complex
medical images.

14. Machine learning recap
2. Unsupervised learning.
• No explicitly labelled data, instead often seeking
to find natural sub-groups of similar data points
within the data.
• Describing the structure of the dataset.
• Objective evaluation of accuracy often
difficult/not possible
• Common methods:
• Clustering methods like k-means and
hierarchical clustering
• Dimensionality reduction methods like
principal components analysis
• Outlier/anomaly detection
• Some NNs, e.g. generative adversarial
networks (GANs)
Larsen et al. 2016

15. Machine learning recap
3. Reinforcement learning.
• Agent must maximise cumulative reward by
learning to optimally interact with an
environment by choosing ‘good’ actions
• E.g. robot must come up with correct limb
positions and joint torque to move
efficiently/quickly
• Manufacturing program must maximise total
units of product produced each day by
controlling/tuning sub-processes
• Video game player agent must process the
information from pixels on the screen to
operate virtual keyboard + mouse to choose
when to turn, apply brakes, gas, etc.
DuskDrive (OpenAI Universe, 2016)
Houthooft et al. 2017

16. Fun machine learning in
video games examples

17. 1. DeepMind
• Classic Atari games
• Learning from raw pixel feeds to
maximise future cumulative reward
(i.e. score), no previous information
encoded
• Ability to generalise skill playing
across games
• In some cases, outperforming human
players
• Deep Q-network approach
• Combining deep learning of the
value function Q(s, a) with RL +
experience replay
• Deep NN is Q(s, a) approximator,
trained with Bellman updates Compulsory DeepMind demo:
https://www.youtube.com/watch?v=V1eYniJ0Rnk
Mnih et al., 2015

18. DeepMind handling quite complex games

19. DeepMind handling quite complex games

20. • 2. Hitman: NPC locomotion learning with RL

21. • 3. Hearthstone (collectible card game)

22. • 3. Hearthstone (collectible card game)
Bursztein, 2014
5. Predicting outcome:
Domain experts (like casters) generate metrics (per turn +
cumulative) that seem important to winning a game:
• Mana advantage (delta mana spent in turn)
• Board advantage (delta # minions)
• Hand size advantage
• Board mana advantage (delta total mana spent), etc.
• No info on hero class or health

23. • 4. Candy Crush (King)

24. Recent and upcoming ML
projects at Jagex

25. 1. Identifying player
personas

26. .
Identifying player personas
• Understand different archetypes of players
in Runescape
• Ultimately we would like to have a better
sense of our customer base
• Helps games devs
• Helps us understand the motivations, churn
behaviours, and value of our customers

27. .
Identifying player personas
• Draw inspiration from Bartel types
• Generated ~280 features to quantify our player
behaviour
• Loosely based on Bartle Types (Killers, Achievers,
Socializers, Explorers)
• Quickly abandoned
• Example behavioural features
• Chat Rank Score
• Skill Heterogeneity
• Forum Interactions
• Economic Activity
• Social Clustering Coefficients
• Abilities Used During Play

28. .
Identifying player personas
• Data cleaning:
• Normalising by total and that year play time,
logging certain variables
• Dimensionality reduction using principal
components analysis
• Identifying optimal number of clusters using
silhouette plots or tSNE
Then… doing the clustering in PC space!

29. Life stages
• Differences in playtime both overall and recently are such a strong differentiating factor between
players that you first need to cluster on life cycle stages

30. Life stages
Name % Median
Lifetime
Playtime
Median
Yearly
Playtime
Little Activity
Short Lifetime Playtime
30.7% 3h 3h
Little Activity
Long Lifetime Playtime
22.1% 590h 4h
Active
Mid Lifetime Playtime
22.2% 75h 35h
Extremely Active
Extreme Lifetime
Playtime
25% 2130h 320h
• The extremely dedicated players make up a huge proportion of total revenue.

31. Playstyles interacting with lifecycle stages
• Can observe which play styles emerge
or dominate at which lifecycle stages
• Can characterise transitions between
the different personas in consecutive
years
• Importantly: Quantify which types of
players tend to abandon the game

32. 2. Deep learning player
lifecycle sequences

33. Player lifecycle sequences
Matt Project
• Players can undergo a variety of business relevant lifecycle
transitions.

34. Player lifecycle sequences
Matt Project
• Number of possible combinations, complexity of transitions,
scale of data, etc. leads to a difficult modelling problem
• Goal: We want to learn more about the player base and
how their lifecycle stages change.
• Two approaches:
• 1. Building transition probability matrices to examine
trends and project player states forward into the future

35. Matt Project
• Transition matrix approach does a surprisingly good job.

36. Deep learning player lifecycle sequences Matt Project
• 2. Building a recurrent neural network (specifically, an LSTM RNN) to learn the types of transitions that
happen and project these forward into the future
• Inherent memory to the system.
• But - difficult and time-consuming to train! GPUs help.
• On the plus side, can generate sequences seeded with multiple states. Also, in theory can learn
quite complex patterns.
• Models operational but need to tune hyperparameters and compare performance to transition
matrices.
[Colah’s blog]

37. 3. Sentiment, bot and
abuse detection work
Matt Project

38. Sentiment, bot and abuse detection work
Matt Project
• At the moment we:
• Help community team understand the popularity of
updates using sentiment detection
• Very simple model – Naïve Bayes classifier run over
in-game chat to identify emotional valence
• Detect instances of harassment
• Search through text with preset list of regular
expressions
• Try to learn more complex language models
• Using word2vec to identify words with similar
semantic meanings, identify harassment using
learned vector embeddings
Matt Project
[Saed Sayad]

39. In progress…
Training deep
convolutional
neural nets to
identify bot-like
mouse
movements
[Adit Deshpande]
Common CNN architecture
[Also working to automate bot removal in other ways]

40. .
4. Trading card deck analysis
• Play a sequence of cards in a sequence to
create a quest for your legend
• Many cards have additional effects that buff
your character, provide weapons, alter
creature power, or grief / damage your
opponent
• Games span 5 chapters where each player
places up to 4 cards per chapter
• The gameplay is combo-oriented where card
play sequence is very important
Miro Project

41. .
Build card vectors with word2vec
• Learn card vectors using Word2Vec
algorithm:
• Given a card vector, predict the
remaining cards that were played by
the player in that chapter.
This worked surprisingly well!
• Cards with similar effects clustered
together
• Legend specific cards clustered together
• Common cards that combo well with
Legend specific cards are located in a
region of space close to the Legend
Miro Project

42. .
Build deck vectors with word2vec
• Tweak Word2Vec algorithm (more akin to
Paragraph2Vec) to learn vector
representation for custom decks
• Deck vectors naturally learned archetype
clusters without being explicitly taught
• Decks from different legends that play by
similar strategies tend to cluster in the same
region of space
• Deck vectors can be used down-stream in
other ML tasks
• Matchmaking
• assessing deck quality
• recommending cards
• predicting churn
Miro Project

43. • Card and deck vectors userful for game devs for exploring the space of
decks real players use
• Interactive web application for exploring decks by hovering over
decks, filtering by cards, player rank, and win rate
• Built clustering tool on top of deck vectors to allow devs to quickly
identify common cards within a region of ‘deck space’
• View summary statistics on the clusters (win rates, play rates,
cluster sizes)
• This is used to identify balance issues in game, identify how players are
playing decks, and categorize player play-styles
Miro Project
Chronicle project conclusions

44. 5. Quest recommender
system
Matt and Miro Project

45. • Ultimate goal: help guide players towards
appropriate and interest-relevant content
• Starting point: recommend appropriate quests
to players.
• Modelling using 2 approaches:
• 1. Restricted Boltzmann Machine (RBM)
trained on past quest data
• 2. Matrix factorisation on past quest data
with a REST API built to interface with game
Miro Project
Quest recommender system
Matt and Miro Project
[deeplearning4j]

46. Acknowledgements
• Data Science Team!
• Miroslaw Horbal
• Matt Dixon
• Babis Georgiadis
• Katie Scott
• Joel Graham
• Ilian Mitev
• Analytics Team 

47. Questions?