Presentation given at the UMAP 2013 Doctoral Consortium

1. Competence Center Information Retrieval & Machine Learning
UMAP‘13 Doctoral Consortium
Evaluation of Cross-Domain News Article Recommendations
Benjamin Kille
13. Juni 2013

2. Agenda
213. Juni 2013
► Problem description
► Challenges in News Article Recommendation
 Sparsity
 Dynamic item collection
 Evaluation
► Research Questions
► Data outline
► Preliminary results
► Conclusions
► Next steps

3. Problem description
313. Juni 2013
► Information overload
 amount of on-line accessible news articles increases
 limited user perception
 limited time capacity
► Solution: Recommender System  filtering news articles with
respect to relevance/utility
► Special challenges for news recommender systems
 Sparsity
 Dynamics
► General challenges for recommender systems
 Evaluation strategy

4. Problem formalization
413. Juni 2013
►

5. Sparsity
513. Juni 2013
►

6. Dynamics
613. Juni 2013
► News  dynamic content
Billsus, D. & Pazzani, M.J., 2007. Adaptive News Access. In P. Brusilovsky, A. Kobsa, &
W. Nejdl, eds. The Adaptive Web. Springer, pp. 550–570.
► Unlike music or movies rarely re-consumed
► For instance: Deutsche Presse Agentur (DPA)
 750 messages
 220k words
 1,5k images
http://www.dpa.de/Zahlen-Fakten.152.0.html

7. Evaluation
713. Juni 2013
► Strategy
 on-line: A/B testing (user-centric)
 off-line: data set (data-centric)
► Numerous facets
 utility
 relevance
 novelty
 serendipity
 …
► Dependending on the model formulation
 preference prediction (requires numerical preference data)
 item ranking

8. Evaluation (cont‘d)
813. Juni 2013
Dispatcher
● recommendation request
 click
●
●
●
●
●
●


9. Evaluation (cont‘d)
913. Juni 2013
Dispatcher
● recommendation request
 click
● ● ● ● ●●
 
? ?
Li, L. et al., 2011. Unbiased offline evaluation of contextual-bandit-based news
article recommendation algorithms. In Proceedings of the fourth ACM international
conference on Web search and data mining - WSDM ’11. p. 297.

10. Cross-domain setting
1013. Juni 2013
D1
D2
U
U
I
I
D1
D2
U
U
I
I
D1
D2
U
U
I
I
D1
D2
U
U
I
I
nooverlap
Useroverlap
Itemoverlap
fulloverlap
Cremonesi, P., Tripodi, A. & Turrin, R., 2011. Cross-Domain Recommender Systems. In
2011 IEEE 11th International Conference on Data Mining Workshops. IEEE, pp. 496–503.

11. Research Questions
1113. Juni 2013
► How can other publishers' user interactions contribute to
decrease sparsity for the target publisher?
► What characteristics must recommender algorithms exhibit to
successfully cope with dynamically changing item collections?
► How to evaluate cross-domain recommender systems with
dynamically changing item collections? How do standard
evaluation metrics compare to the observed clicks?

12. Data outline
1213. Juni 2013
► > 1-2M impressions by 12 publishers (general news, local news,
finance, information technology, sports, etc.) on a daily basis
► user features such as
 browser
 ISP
 OS
 device
► news article features such as
 title
 text
 URL
 Image
► http://www.dai-labor.de/en/irml/epen/
►Real interactions with actual users!

13. Preliminary results
1313. Juni 2013
► Sparsity
► Histogram of the relative frequency of user interactions

14. Preliminary results (cont‘d)
1413. Juni 2013
► Dynamics

15. Preliminary results (cont‘d)
1513. Juni 2013
► Popularity

16. Conclusions
1613. Juni 2013
►

17. Next steps
1713. Juni 2013
► Implementation of existing cross-domain recommender
algorithms
► Evaluating recommender algorithms with respect to
 CTR
 novelty
 diversity
► Investigate UI effects
► Analyze applicability of context-sensitive recommendations
► User/Item clustering to speed-up computation time

18. Thank you for the attention!
1813. Juni 2013
Questions???

19. Announcement: NRS 2013
1913. Juni 2013
► International News Recommender Systems Workshop and Challenge
► In conjunction with ACM RecSys 2013
IMPORTANT DATES
 July 21, 2013 paper submission deadline
 July 1, 2013 data set release
 August 15, 2013 on-line challenge kick-off
HIGHLIGHTS
 Access to a real recommender system
 Real-time requirements
 Big Data
 Cross-domain
 Implicit feedback
Website: https://sites.google.com/site/newsrec2013/home
Twitter: @NRSws2013

20. Competence Center Information Retrieval &
Machine Learning
www.dai-labor.de
Fon
Fax
+49 (0) 30 / 314 – 74
+49 (0) 30 / 314 – 74 003
DAI-Labor
Technische Universität Berlin
Fakultät IV – Elektrontechnik & Informatik
Sekretariat TEL 14
Ernst-Reuter-Platz 7
10587 Berlin, Deutschland
20
Benjamin Kille
Researcher / PhD student
benjamin.kille@dai-labor.de
74128
13. Juni 2013