Presentation given for one of Pearson's Data Research teams. It motivates the use of recommender systems, describes common approaches to building and evaluating them and gives examples of how they are used in Mendeley. Joint work with Kris Jack, Chief Data Scientist at Mendeley.

1. Kris Jack and Maya Hristakeva
16/12/2014
Modern Perspectives on
Recommender Systems and their
Applications in Mendeley

2. Kris Jack, Chief Data Scientist
http://www.mendeley.com/profiles/kris-jack/
Maya Hristakeva, Senior Data Scientist
http://www.mendeley.com/profiles/maya-hristakeva/
Phil Gooch, Senior Data Scientist
http://www.mendeley.com/profiles/phil-gooch/

3. Overview
• The what and why of recommenders
• Evolution of the recommender problem
• Recommender algorithms
• Evaluating a recommender
• Recommender systems @ Mendeley

4. Overview
• The what and why of recommenders
• Evolution of the recommender problem
• Recommender algorithms
• Evaluating a recommender
• Recommender systems @ Mendeley

5. What is a recommender?
A recommendation system (recommender) is a push system that presents
users with the most relevant content for their context and needs
• helps users to deal with information overload
• recommenders are complementary to search
search engine
pull
recommendation engine
push
request
infers context
and needs
Information
Retrieval
Information
Filtering

6. Recommenders @ Linkedin
50% of LinkedIn connections are from recommendations

7. Recommenders @ Linkedin

8. Recommenders @ Netflix
Stop 1% of users from cancelling subscription = $500M/year
Netflix invests $150M/year (300 people) in their content rec team

9. Recommenders @ ResearchGate

10. Why recommenders?
• Search and recommendations are complementary, have arms and legs!
• Higher usability, user satisfaction and engagement
• Increase product stickiness
• Monetise them
...and in the context of research...
Help researchers keep up-to-date with latest research, connect with
researchers in their field, contextualise their work within the global body of
research (articles, researchers, conferences, research groups, etc.)

11. Overview
• The what and why of recommenders
• Evolution of the recommender problem
• Recommender algorithms
• Evaluating a recommender
• Recommender systems @ Mendeley

12. Evolution of recommender problem
Problem: We have a massive collection of items (e.g. > 1 million).
We want to recommend 5 items that the user will like.

13. Evolution of recommender problem
First, seen as a ratings prediction problem. So, given some knowledge of the
user, estimate how much they will appreciate each item on scale of 1-5.
4.9
choose top 5 items with highest predicted ratings
4.7 4.7 4.6 4.5

14. Evolution of recommender problem
But do predicted ratings give the best order? Improve the recommender by
reranking a selection of items with high predicted ratings.
rerank items that are highly predicted
4.7 4.9 4.6 4.6 4.8

15. Evolution of recommender problem
Let’s improve the recommendations by optimizing the page in which they
appear.
deliver them in style

16. Evolution of recommender problem
Take the user’s context into account.
new to this
topic?
yesno

17. Evolution of recommender problem
Actively researching how to take other properties into account in context:
trustworthiness; freshness; diversity; serendipity; novelty; recency.
at work? yesno

18. Rating prediction
Reranking
Page optimisation
Context-aware
Future: trustworthiness; freshness; diversity; serendipity; novelty; recency.
How to make recommendations?
On to the algorithms...
Evolution of recommender problem
time

19. Overview
• The what and why of recommenders
• Evolution of the recommender problem
• Recommender algorithms
• Evaluating a recommender
• Recommender systems @ Mendeley

20. Recommender algorithms
A recommender processes information and transforms it into actionable
knowledge. Here we’ll focus on the algorithms that make this possible.
information flow (components often built in parallel)

21. Recommender algorithms
• Collaborative filtering (similarity and model-based)
• Content-based filtering
• Hybrid
• Non-traditional

22. Collaborative filtering
Formal representation
• User-based CF finds users who have similar appreciations for items as
you and recommends new items based on what they like.
• Item-based CF finds items that are similar to the ones you like.
Similarity is based on item cooccurrences (e.g. the users who bought x
also bought y).
Similarity-based CF
• ti
: rating of user xi
for item yi
.
• Infer prediction function

23. Collaborative filtering
Formal representation of MF
• X: user-item ratings matrix
• U: user-latent factors matrix
• S: latent factor diagonal matrix
• V: latent factor-item matrix
• Matrix Factorisation (SVD++)
• Clustering (K-means to LDA)
• LSH (Locality sensitive hashing)
• Restricted Boltzmann Machines
Model-based CF

24. Collaborative filtering
Pros
• Minimal domain knowledge
required
• User and item features are not
required
• Produces good enough results
in most cases
• Cold start problem
• Requires high user:item ratio (1:
10)
• Needs standardised products
• Popularity bias (doesn’t play
well with the long tail)
Cons
• User-based CF
• Item-based CF
• Model-based CF

25. Content-based filtering
• Determine item similarity based on item content not usage data
• Recommend items similar to those that a user is known to like
• The user model:
• explicitly provided features/keywords of interest
• can be a classifier (e.g Naive Bayes, SVM, Decision trees)
Formal representation
• ti
: rating of user xi
for item yi
, where xi
and yi
are feature vectors
• Infer prediction function

26. Content-based filtering
Pros
• No cold start problem
• No need for usage data
• No popularity bias, can
recommend items with rare
features
• Item content needs to be
machine readable and meaningful
• Easy to pigeonhole the user
• Difficult to implement serendipity
• Difficult to combine multiple item’
s features together
Cons
• Determine item similarity based on item content not usage data
• Recommend items similar to those that a user is known to like
• The user model:
• explicitly provided features/keywords of interest
• can be a classifier (e.g Naive Bayes, SVM, Decision trees)

27. Hybrid approaches
Method Description
Weighted Outputs from several techniques (in the form of scores or votes) are combined
with different degrees of importance to offer final recommendations
Switching Depending on situation, the system changes from one technique to another
Mixed Recommendations from several techniques are presented at the same time
Feature combination Features from different recommendation sources are combined as input to a
single technique
Cascade The output from one technique is used as input of another that refines the
result
Feature augmentation The output from one technique is used as input features to another
Meta-level The model learned by one recommender is used as input to another

28. Hybrid approaches
Combining user and item features and usage to benefit from both
Pros
• Often outperforms CF and CB
alone
Cons
• Can be a lot of work to get the
right balance

29. Non-traditional approaches
• Deep learning
• Social recommendations
• Learning to rank
• ...
Pros Cons
• Good for eking out those final
performance percentage points
• You can say you’re working with
current edge approaches ;)
• Less well understood
• Less supported in
recommendation toolkits
• Not recommended approaches
for your first recommender

30. Is your recommender doing well?
• Typically employ collaborative filtering
• May need to use content-based filtering particularly to bootstrap
• Go advanced with a hybrid
• Do all of that before getting adventurous with state-of-the-art
You don’t really know unless you evaluate it...
Algorithms

31. Overview
• The what and why of recommenders
• Evolution of the recommender problem
• Recommender algorithms
• Evaluating a recommender
• Recommender systems @ Mendeley

32. • Offline testing
• Online testing (A/B testing)
Evaluating a recommender

33. Offline testing
• Test offline before deploying
• Parameter sweep is quick
• Doesn’t offend real users
• n-fold cross validation:
• Take the users, items and
relationships between them
(e.g. clicked on, bought)
• Split into n folds, for training
(n-1) and testing (1)
• Attempt to predict the testing
data based on the training
data
• Popularity as baseline
Metrics
• Precision, recall and f-measure
• Receiver operating characteristic
(ROC) curve
• Normalised discounted cumulative
gain (NDCG)
• Mean reciprocal rank (MRR)
• Fraction of Concordant Pairs (FCP)
• ...

34. Online testing
• Offline performance isn’t a very
precise indicator
• Offline test is good sanity
check
• Online test gives real
performance
• A/B testing
• Deploy your systems that
perform ‘well enough’
• Compare them with each
other in real world
• Mind the pitfalls
Metrics
• The offline metrics +
• Conversion rate
• Open, view, click through rates
• Usage data (e.g. reordered item,
completed reading book)
• Hard to evaluate: trustworthiness;
freshness; diversity; serendipity;
novelty; recency.

35. • Start with offline testing
• Perform A/B testing but be aware of the common pitfalls
• Hard to evaluate performance in terms of: trustworthiness; freshness;
diversity; serendipity; novelty; recency.
How do we use recommenders?
On to a few of our use cases...
Evaluating a recommender

36. Overview
• The what and why of recommenders
• Evolution of the recommender problem
• Recommender algorithms
• Evaluating a recommender
• Recommender systems @ Mendeley

37. Recommenders @ Mendeley

38. Recommenders @ Mendeley
Related research for an article

39. Recommenders @ Mendeley
Related research for multiple articles

40. Recommenders @ Mendeley
Mendeley Suggest - personalised batch of recommended reading

41. Recommenders @ Mendeley
Researchers to follow on Mendeley

42. Recommenders @ Mendeley
Interesting activity from your social network

43. • Recommenders are employed for a number of use cases
• Recommenders deliver different kinds of value depending upon use case
• Can reuse the same underlying recommender system and framework for all
Recommenders @ Mendeley

44. • Recommenders are complementary to search and becoming mainstream
• although arguably can cater for a wider range of use cases
• When building a recommender, it’s common to predict ratings, rerank,
optimise the page and then introduce context-awareness
• In building a recommender, start with collaborative filtering if you can,
content-based if you need to bootstrap and then explore hybrids
• Open research questions remain as recommenders are used to tackle
trustworthiness; freshness; diversity; serendipity; novelty; recency
Conclusions

45. References
• Xavier Amatriain, The Recommender Problem Revisited (http://www.
slideshare.net/xamat/recsys-2014-tutorial-the-recommender-problem-
revisited)
• Rec Sys 2014 (http://recsys.acm.org/recsys14/)

46. Thank you
www.mendeley.com