Data Scientist at Jagex - Jagex has a diverse games portfolio, including a large MMORPG (RuneScape), a tactical FPS (Block N Load) and a collectible card game (Chronicle). In this presentation, Natasha Latysheva will showcase recent and upcoming data science and machine learning projects at Jagex – including quest recommender systems, player clustering by playstyle, deep learning player lifecycle sequences, and automatic bot and abuse detection – to give a taster of the insights that a machine learning approach can provide, whilst also offering project ideas for your own games.

2. Understanding and
improving games through
machine learning
Natasha Latysheva
Data Scientist
Jagex Games Studio

3. This talk:
1. Introduction to Jagex
2. Recap of main machine learning
fields + examples
3. Fun ML in video games in general
4. ML projects at Jagex Data Science
• Identifying player personas
• Deep learning player lifecycle
sequences
• Automatic bot and abuse
detection
• Trading card deck analysis
• Quest recommender systems

4. Jagex Game Studios
• Video game development company based in
Cambridge
• Best known for Runescape, the world's
largest free-to-play massively multiplayer
online role-playing game (MMORPG). First
released in 2001!
• Extremely community oriented game
• Old School Runescape was released in 2013

5. RuneScape
Main..

6. .. and
RuneScape
Old School

7. TF2 meets
Minecraft!

12. A quick machine
learning recap

13. Machine learning recap
Essentially learning patterns from data.
Branches of ML:
• 1. Supervised learning.
• Most familiar names, e.g. linear regression,
logistic regression, decision trees, random
forests, SVMs, many neural nets.
• Algorithm is presented with input data and
explicit labels on that input. Learn signals
associating input data with labels.
• Everything from good ole iris classification to
detecting diseased patients from complex
medical images.

14. Machine learning recap
2. Unsupervised learning.
• No explicitly labelled data, instead often seeking
to find natural sub-groups of similar data points
within the data.
• Describing the structure of the dataset.
• Objective evaluation of accuracy often
difficult/not possible
• Common methods:
• Clustering methods like k-means and
hierarchical clustering
• Dimensionality reduction methods like
principal components analysis
• Outlier/anomaly detection
• Some NNs, e.g. generative adversarial
networks (GANs)
Larsen et al. 2016

15. Machine learning recap
3. Reinforcement learning.
• Agent must maximise cumulative reward by
learning to optimally interact with an
environment by choosing ‘good’ actions
• E.g. robot must come up with correct limb
positions and joint torque to move
efficiently/quickly
• Manufacturing program must maximise total
units of product produced each day by
controlling/tuning sub-processes
• Video game player agent must process the
information from pixels on the screen to
operate virtual keyboard + mouse to choose
when to turn, apply brakes, gas, etc.
DuskDrive (OpenAI Universe, 2016)
Houthooft et al. 2017

16. Fun machine learning in
video games examples

17. 1. DeepMind
• Classic Atari games
• Learning from raw pixel feeds to
maximise future cumulative reward
(i.e. score), no previous information
encoded
• Ability to generalise skill playing
across games
• In some cases, outperforming human
players
• Deep Q-network approach
• Combining deep learning of the
value function Q(s, a) with RL +
experience replay
• Deep NN is Q(s, a) approximator,
trained with Bellman updates Compulsory DeepMind demo:
https://www.youtube.com/watch?v=V1eYniJ0Rnk
Mnih et al., 2015

18. DeepMind handling quite complex games

19. DeepMind handling quite complex games

20. • 2. Hitman: NPC locomotion learning with RL

21. • 3. Hearthstone (collectible card game)

22. • 3. Hearthstone (collectible card game)
Bursztein, 2014
5. Predicting outcome:
Domain experts (like casters) generate metrics (per turn +
cumulative) that seem important to winning a game:
• Mana advantage (delta mana spent in turn)
• Board advantage (delta # minions)
• Hand size advantage
• Board mana advantage (delta total mana spent), etc.
• No info on hero class or health

23. • 4. Candy Crush (King)

24. Recent and upcoming ML
projects at Jagex

25. 1. Identifying player
personas

26. .
Identifying player personas
• Understand different archetypes of players
in Runescape
• Ultimately we would like to have a better
sense of our customer base
• Helps games devs
• Helps us understand the motivations, churn
behaviours, and value of our customers

27. .
Identifying player personas
• Draw inspiration from Bartel types
• Generated ~280 features to quantify our player
behaviour
• Loosely based on Bartle Types (Killers, Achievers,
Socializers, Explorers)
• Quickly abandoned
• Example behavioural features
• Chat Rank Score
• Skill Heterogeneity
• Forum Interactions
• Economic Activity
• Social Clustering Coefficients
• Abilities Used During Play

28. .
Identifying player personas
• Data cleaning:
• Normalising by total and that year play time,
logging certain variables
• Dimensionality reduction using principal
components analysis
• Identifying optimal number of clusters using
silhouette plots or tSNE
Then… doing the clustering in PC space!

29. Life stages
• Differences in playtime both overall and recently are such a strong differentiating factor between
players that you first need to cluster on life cycle stages

30. Life stages
Name % Median
Lifetime
Playtime
Median
Yearly
Playtime
Little Activity
Short Lifetime Playtime
30.7% 3h 3h
Little Activity
Long Lifetime Playtime
22.1% 590h 4h
Active
Mid Lifetime Playtime
22.2% 75h 35h
Extremely Active
Extreme Lifetime
Playtime
25% 2130h 320h
• The extremely dedicated players make up a huge proportion of total revenue.

31. Playstyles interacting with lifecycle stages
• Can observe which play styles emerge
or dominate at which lifecycle stages
• Can characterise transitions between
the different personas in consecutive
years
• Importantly: Quantify which types of
players tend to abandon the game

32. 2. Deep learning player
lifecycle sequences

33. Player lifecycle sequences
Matt Project
• Players can undergo a variety of business relevant lifecycle
transitions.

34. Player lifecycle sequences
Matt Project
• Number of possible combinations, complexity of transitions,
scale of data, etc. leads to a difficult modelling problem
• Goal: We want to learn more about the player base and
how their lifecycle stages change.
• Two approaches:
• 1. Building transition probability matrices to examine
trends and project player states forward into the future

35. Matt Project
• Transition matrix approach does a surprisingly good job.

36. Deep learning player lifecycle sequences Matt Project
• 2. Building a recurrent neural network (specifically, an LSTM RNN) to learn the types of transitions that
happen and project these forward into the future
• Inherent memory to the system.
• But - difficult and time-consuming to train! GPUs help.
• On the plus side, can generate sequences seeded with multiple states. Also, in theory can learn
quite complex patterns.
• Models operational but need to tune hyperparameters and compare performance to transition
matrices.
[Colah’s blog]

37. 3. Sentiment, bot and
abuse detection work
Matt Project

38. Sentiment, bot and abuse detection work
Matt Project
• At the moment we:
• Help community team understand the popularity of
updates using sentiment detection
• Very simple model – Naïve Bayes classifier run over
in-game chat to identify emotional valence
• Detect instances of harassment
• Search through text with preset list of regular
expressions
• Try to learn more complex language models
• Using word2vec to identify words with similar
semantic meanings, identify harassment using
learned vector embeddings
Matt Project
[Saed Sayad]

39. In progress…
Training deep
convolutional
neural nets to
identify bot-like
mouse
movements
[Adit Deshpande]
Common CNN architecture
[Also working to automate bot removal in other ways]

40. .
4. Trading card deck analysis
• Play a sequence of cards in a sequence to
create a quest for your legend
• Many cards have additional effects that buff
your character, provide weapons, alter
creature power, or grief / damage your
opponent
• Games span 5 chapters where each player
places up to 4 cards per chapter
• The gameplay is combo-oriented where card
play sequence is very important
Miro Project

41. .
Build card vectors with word2vec
• Learn card vectors using Word2Vec
algorithm:
• Given a card vector, predict the
remaining cards that were played by
the player in that chapter.
This worked surprisingly well!
• Cards with similar effects clustered
together
• Legend specific cards clustered together
• Common cards that combo well with
Legend specific cards are located in a
region of space close to the Legend
Miro Project

42. .
Build deck vectors with word2vec
• Tweak Word2Vec algorithm (more akin to
Paragraph2Vec) to learn vector
representation for custom decks
• Deck vectors naturally learned archetype
clusters without being explicitly taught
• Decks from different legends that play by
similar strategies tend to cluster in the same
region of space
• Deck vectors can be used down-stream in
other ML tasks
• Matchmaking
• assessing deck quality
• recommending cards
• predicting churn
Miro Project

43. • Card and deck vectors userful for game devs for exploring the space of
decks real players use
• Interactive web application for exploring decks by hovering over
decks, filtering by cards, player rank, and win rate
• Built clustering tool on top of deck vectors to allow devs to quickly
identify common cards within a region of ‘deck space’
• View summary statistics on the clusters (win rates, play rates,
cluster sizes)
• This is used to identify balance issues in game, identify how players are
playing decks, and categorize player play-styles
Miro Project
Chronicle project conclusions

44. 5. Quest recommender
system
Matt and Miro Project

45. • Ultimate goal: help guide players towards
appropriate and interest-relevant content
• Starting point: recommend appropriate quests
to players.
• Modelling using 2 approaches:
• 1. Restricted Boltzmann Machine (RBM)
trained on past quest data
• 2. Matrix factorisation on past quest data
with a REST API built to interface with game
Miro Project
Quest recommender system
Matt and Miro Project
[deeplearning4j]

46. Acknowledgements
• Data Science Team!
• Miroslaw Horbal
• Matt Dixon
• Babis Georgiadis
• Katie Scott
• Joel Graham
• Ilian Mitev
• Analytics Team 

47. Questions?