The Magic Barrier of Recommender Systems: Rating Coherence Predicts Accuracy
•
11 gostaram•2,020 visualizações
This document discusses the "magic barrier" of recommender systems, which refers to the upper bound on recommendation accuracy that is determined by the quality of user rating data. The authors propose that rating coherence, which measures consistency in a user's ratings within specific item features/genres, can serve as a good predictor of the magic barrier. Two experiments are described: 1) Correlating rating coherence with actual magic barrier measurements; 2) Using coherence to improve recommendations by separately training models on coherent vs. incoherent user groups. The results show rating coherence has a strong correlation with magic barrier and using it to separately train recommendation models leads to improved accuracy over treating all users the same.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a The Magic Barrier of Recommender Systems: Rating Coherence Predicts Accuracy
Semelhante a The Magic Barrier of Recommender Systems: Rating Coherence Predicts Accuracy (20)
Mais de Alan Said
Mais de Alan Said (14)
Último
Último (20)
The Magic Barrier of Recommender Systems: Rating Coherence Predicts Accuracy
- 1. The Magic Barrier of Recommender Systems: No Magic, Just ra;ngs Alejandro Bellogín, Alan Said, Arjen de Vries @abellogin, @alansaid, @arjenpdevries UMAP 2014 2014-‐07-‐08
- 2. What is the magic barrier? 2
- 3. The Magic Barrier The upper bound on recommenda;on accuracy Recommenda;on accuracy is only as good as the quality of the underlying data Users are noisy / incoherent in their ra;ng behavior 3
- 4. How to find the magic barrier? 1. Collect infinite re-‐ra;ngs for each item from your users. 2. Find standard devia;on of ra;ng/ re-‐ra;ng inconsistencies (noise) [Said et al. UMAP ‘12] Realis;cally, an infinite amount of re-‐ra;ngs is not probable 4
- 5. Coherence 5
- 6. What is coherence? Coherent user e slightly different. Specifically, the user in Figure 1a gives more consisten items sharing the same features, which in this case corresponds to the mov s. In the long term we argue that such a user will have a more consistent (le ehaviour, since her taste for each item feature seem to be well defined. Action Comedy Horror (a) A coherent user (c(u) = 1.28) Action Comedy Horror (b) A not coherent user (c(u) = Incoherent user htly different. Specifically, the user in Figure 1a gives more consistent rating sharing the same features, which in this case corresponds to the movie’s gen he long term we argue that such a user will have a more consistent (less noisy ur, since her taste for each item feature seem to be well defined. Action Comedy Horror A coherent user (c(u) = 1.28) Action Comedy Horror (b) A not coherent user (c(u) = 5.95) 6
- 7. DefiniAon of Coherence Ra;ng consistency within a feature space (genres, themes, etc.) c u( )= − σ f f ∈F ∑ u( ) σ f u( )= r u,i( )− rf u( )( ) 2 i∈I u, f( ) ∑ Item features User’s ra;ng devia;on within a feature 7
- 8. Ra;ng devia;on within a feature space instead of infinite re-‐ra;ngs Is this a good es;mate of the magic barrier? 8
- 9. Experiments EX1: Is ra;ng coherence a good predictor for the Magic Barrier? EX2: Can we use ra;ng coherence to improve recommenda;on results? Especially when we do not have re-‐ ra;ngs/magic barrier? 9
- 10. RaAng Coherence vs. Magic Barrier Rank users according to magic barrier/ ra;ng coherence: • High/low magic barrier • Low/high ra;ng coherence in the “genre” feature space Find correla;on (Spearman) between magic barrier/coherence ranking *Dataset from moviepilot.de, contains re-‐ra;ngs from 306 users [Said et al. UMAP ‘12] 10
- 11. Magic Barrier & Ra;ng Coherence correla;on 11
- 12. Experiments EX1: Is ra;ng coherence a good predictor for the Magic Barrier? EX2: Can we use ra;ng coherence to improve recommenda;on results? Especially when we do not have re-‐ ra;ngs/magic barrier? 12
- 13. Use raAng coherence for recommendaAon Divide users into coherent and incoherent – with higher an lower magic barrier Movielens 1M Train each group separately Evaluate each group separately 13
- 14. RecommendaAon and EvaluaAon Algorithm: K-‐NN, Pearson Training sets • “Easy” users split • “Difficult” users split • All users split Evalua;on sets • “Easy” users split • “Difficult” users split • All users split Training Easy Training Difficult Test Easy Test Difficult 14
- 15. EvaluaAon Comparisons All -‐ All All – Easy All – Difficult Easy – Easy Difficult – Difficult Training Easy Training Difficult Test Easy Test Difficult RMSE: 1.090 User coverage: 100% 15
- 16. EvaluaAon Comparisons All -‐ All All – Easy All – Difficult Easy – Easy Difficult – Difficult Training Easy Training Difficult Test Easy Test Difficult RMSE: 0.974 User coverage: 50% 16
- 17. EvaluaAon Comparisons All -‐ All All – Easy All – Difficult Easy – Easy Difficult – Difficult Training Easy Training Difficult Test Easy Test Difficult RMSE: 1.195 User coverage: 50% 17
- 18. EvaluaAon Comparisons All -‐ All All – Easy All – Difficult Easy – Easy Difficult – Difficult Training Easy Training Difficult Test Easy Test Difficult RMSE: 0.933 User coverage: 50% 18
- 19. EvaluaAon Comparisons All -‐ All All – Easy All – Difficult Easy – Easy Difficult – Difficult Training Easy Training Difficult Test Easy Test Difficult RMSE: 1.226 User coverage: 50% 19
- 20. Results Subset RMSE User Coverage All 1.090 100% All-‐Easy 0.974 50% All-‐Diff 1.195 50% Easy-‐Easy 0.933 50% Diff-‐Diff 1.226 50% 20
- 21. Results Subset RMSE User Coverage All 1.090 100% All-‐Easy 0.974 50% (easy) All-‐Diff 1.195 50% (diff) Easy-‐Easy 0.933 50% (easy) Diff-‐Diff 1.226 50% (diff) RMSE average: 1.084 21
- 22. Results Subset RMSE User Coverage All 1.090 100% All-‐Easy 0.974 50% (easy) All-‐Diff 1.195 50% (diff) Easy-‐Easy 0.933 50% (easy) Diff-‐Diff 1.226 50% (diff) RMSE average: 1.064 22
- 23. Experiments EX1: Is ra;ng coherence a good predictor for the Magic Barrier? EX2: Can we use ra;ng coherence to improve recommenda;on results? Especially when we do not have re-‐ ra;ngs/magic barrier? 23
- 24. Conclusions Ra;ng coherence within an item’s feature space is related to the magic barrier Ra;ng coherence can be used to predict users’ ra;ng inconsistencies A user’s ra;ng inconsistencies tell us how well our system performs for that user – i.e. whether we need to op;mize further 24
- 25. Thanks! Ques;ons? More recommender systems: www.recsyswiki.com 25
- 26. Image credits • Slide 1 -‐ hqp://www.burdens.net.au/product/esf-‐barriers • Slide 2 -‐ hqp://www.freepicturesweb.com/pictures/2010-‐07/5881.html • Slide 4 -‐ hqp://redgannet.blogspot.nl/2010/08/moreletakloof-‐johannesburg-‐jnb.html • Slide 5 -‐ hqp://en.wikipedia.org/wiki/Standard_devia;on#Interpreta;on_and_applica;on • Slide 9, 12, 23 -‐ hqp://www.withautodesk.com/html/science-‐fair-‐ideas-‐for-‐physical-‐science-‐inven;ons.html • Slide 13 -‐ hqp://fineartamerica.com/featured/16-‐cogs-‐les-‐cunliffe.html • Slide 24 -‐ hqps://benchprep.com/blog/the-‐ap-‐english-‐exam-‐wri;ng-‐compelling-‐conclusions/ 26