Palestra ministrada no X Seminário de Engenharia da Computação (SEC) Na Escola Politécnica de Pernambuco, em Recife Pernambuco em 21.10.2011

1. Content
Recommenda,on
Based
on
Data
Mining
in
Adap,ve
Social
Networks
Marcel
Pinheiro
Caraciolo
mpc@cin.ufpe.br
Orientador:
Germano
Crispim gcv@cin.ufpe.br

2. Who is Marcel ?
Marcel Pinheiro Caraciolo - @marcelcaraciolo
Sergipano, porém Recifense.
Cientista Chefe da Orygens (http://orygens.com)
Mestrando em Ciência da Computação no CIN/UFPE na área de mineração de dados
Membro e Moderador da Celúla de Usuários Python de Pernambuco (PUG-PE)
Minhas áreas de interesse: Computação móvel e Computação inteligente
Meus blogs: http://www.mobideia.com (sobre Mobilidade desde 2006)
http://aimotion.blogspot.com (sobre I.A. desde 2009)
Jovem Aprendiz ainda nas artes pythonicas.... (desde 2007)
2

3. The
Problem
Provide features that can enhance online social learning environment
One social software in special are recommender systems
Several approaches have been applied to web-based educational systems [1] [2]
But only a few recommender systems use data mining and explanations in
the recommendations

4. Objec,ves
Design a methodology for social recommender systems that incorporate different knowledge sources
It can be structured such as messages, personal profile, timeline, questions , answers
It can be non-structured study plans, disciplines, classes watched, relationships
educational resources
ents in a social envi- 01'
g a recommender en- (%)*++%&$%,'' 01'
-&./&%' 01'
different recommenda-
ation called Crab [15]. !"#$%&"' (%)*++%&$23*&4'
al domains, including 56786-'
suggest online learn-
599786-'
eir preferences, knowl-
terests. The system is 8&;*,+23*&'
ational social network 9/<"%,/&.'
=%)>&/?#%4'
dents, helping them to
!"#$%&":4'7*.'
The innovative aspect
ing on social networks
ions at the given rec- components of Architecture proposal for the recom-
It uses several Figure 1: the social network
udent more confidence mender system
arning understanding.
s use data mining and
[5]. 3. CONTENT RECOMMENDATION IN ED-
st we describe the re- UCATIONAL SOCIAL NETWORKS

5. Objec,ves
Design an improved explanation of the recommendation to the user
Generally in the currently available systems, the recommendation only comes with an overall score
Increase the recommendation acceptance rate in order to enhance the student experience in social networks
You answered questions about X,Y, Z. You answered questions about X,Y, Z.
You have A,B,C friends in common
Your friend asked this question.
Your friends P,X and Y follows him.
You both live at the same state Pernambuco.
You have X,Y, Z followers in common
You are both interested in portuguese, maths and english.
You have X,Y, Z concursos in common
You have P,Q, R study groups in common.
Because you have difficulty in portuguese, maths and english.
Because it is a most searched course

6. What recommender systems really do?
1. Predict how much you would like a certain
product or service
2. Suggest a list of N items ranked based on your
interests
3. Suggest a list of N users ranked for a product/
service
4. Explain to you why those recommendations were
given
5. Adjust the prediction and the recommendation
based on your feedback and from others.

7. Mee,ng
Recommenda,on
Systems
Content
Based
Filtering
Similar
Text
Portuguese Maths Biology Items
Interpretation
recommend
likes
Marcel Users

8. Mee,ng
Recommenda,on
Systems
Collabora,ve
Based
Filtering
Portuguese Literature Maths Physics Items
like like
recommends
Marcel Rafael Amanda Users
Similar

9. Mee,ng
Recommenda,on
Systems
Our
Approach
-­‐
Hybrid
Recommender
System
Meta recommender system architecture
Customized control over the generation of
a recommendation list
!"#$%*'+,-)%
!"#$%&$'()#%
It adapts in accordance to the structure of the data ./0#$-+1'/%
Recommendation of Friends ?
More weight to Collaborative Filtering %
%
%
%
!"#$%#$&'()%*&+,-$%.,#/&
!"--(0".(12%&'()%*&+,-$%.,#/&
%
Recommendation of Courses ? 2$,#/3"%456575689%
%
!"#"$%&&'%()*&+,-(.'&/,-0&+,-(.'&
:'+-1'/;%%<#+,=#%
%12%&'303#2,&('",'&2,"&34&
More weight to Content Filtering *+>')-$">,?;%%@$-3A-0#3%
%
%
Cold Start: Mitigate using Popular Recommendations B#0-%<#+'CC#/3#$%
%% %&-$-C#0#$"%%
accepted from another users
<#+'CC#/3-1'/"%
Feedback and temporal slicing:
Learn from users and select limited results by time
Figure 2: Meta Recommender Components Interac-
tion
be highly beneficial given that students do not meet phys-
ically. It may result in their becoming more socially con-

10. Methodology
and
Current
Results
!"#$%*'+,-)%
./0#$-+1'/% this engine with the popular brazilian social network AtéPassar
Integrated
More than 70.000 students registered studying for the public examinations
Recommend StudyGroups, Friends,Video Classes, Questions and Concursos
More than 70.000 items available for recommend
%
%
!"--(0".(12%&'()%*&+,-$%.,#/&
%
Written in Python using a open-source framework Crab
!"#"$%&&'%()*&+,-(.'&/,-0&+,-(.'&
%12%&'303#2,&('",'&2,"&34&
%
Framework available for building recommender systems (My contribution)
It is running since January 2011
In March B#0-%<#+'CC#/3#$% was performed.
2011 , questionnaire
%% %&-$-C#0#$"%%
Liked Not Liked
-1'/"%
23%
mender Components Interac-
77%
Figure 3: AtePassar Recommender Syste
face

12. Expected
Results
Improve the Learning Process
Analyze how the recommenders can increase the learning process in online educational social networks.
Mining all types of source in social networks in forms of recommendations
Explore the Hidden Knowledge
There are several knowledge sources in a educational social network.
All those sources as basis for discovery novel content and learning resources
Better Recommendation Understanding
The explanations can be quite helpful to better understanding of the given resource
Better learning interaction and simplification of the human/computer interaction

13. Conclusions
Proposal of a personalized recommender system
that incorporate different types of source of knowledge
It can be applied in adaptive social networks
Bring machine learning to web educational systems
Help the students to find an optimal path to learning resources
Help the researchers and designers how to design
the social network to achieve those tasks efficiently

14. Contribu)ons

15. Crab
A Python Framework for Building
Recommendation Engines
Marcel Caraciolo Ricardo Caspirro Bruno Melo
@marcelcaraciolo @ricardocaspirro @brunomelo

16. What is Crab ?
A python framework for building recommendation engines
A Scikit module for collaborative, content and hybrid filtering
Mahout Alternative for Python Developers :D
Open-Source under the BSD license
https://github.com/muricoca/crab

17. The current Crab
Collaborative Filtering algorithms
User-Based, Item-Based and Slope One
Evaluation of the Recommender Algorithms
Precision, Recall, F1-Score, RMSE
Precision-Recall Charts

18. The current Crab

19. The current Crab
>>>#load the dataset

20. The current Crab
>>>#load the dataset
>>> from crab.datasets import load_sample_movies

21. The current Crab
>>>#load the dataset
>>> from crab.datasets import load_sample_movies
>>> data = load_sample_movies()

22. The current Crab
>>>#load the dataset
>>> from crab.datasets import load_sample_movies
>>> data = load_sample_movies()
>>> data

23. The current Crab
>>>#load the dataset
>>> from crab.datasets import load_sample_movies
>>> data = load_sample_movies()
>>> data
{'DESCR': 'sample_movies data set was collected by the book called
nProgramming the Collective Intelligence by Toby Segaran nnNotesn-----
nThis data set consists ofnt* n ratings with (1-5) from n users to n movies.',
'data': {1: {1: 3.0, 2: 4.0, 3: 3.5, 4: 5.0, 5: 3.0},
2: {1: 3.0, 2: 4.0, 3: 2.0, 4: 3.0, 5: 3.0, 6: 2.0},
3: {2: 3.5, 3: 2.5, 4: 4.0, 5: 4.5, 6: 3.0},
4: {1: 2.5, 2: 3.5, 3: 2.5, 4: 3.5, 5: 3.0, 6: 3.0},
5: {2: 4.5, 3: 1.0, 4: 4.0},
6: {1: 3.0, 2: 3.5, 3: 3.5, 4: 5.0, 5: 3.0, 6: 1.5},
7: {1: 2.5, 2: 3.0, 4: 3.5, 5: 4.0}},
'item_ids': {1: 'Lady in the Water',
2: 'Snakes on a Planet',
3: 'You, Me and Dupree',
4: 'Superman Returns',
5: 'The Night Listener',
6: 'Just My Luck'},
'user_ids': {1: 'Jack Matthews',
2: 'Mick LaSalle',
3: 'Claudia Puig',
4: 'Lisa Rose',
5: 'Toby',
6: 'Gene Seymour',
7: 'Michael Phillips'}}

24. The current Crab

25. The current Crab
>>> from crab.models import MatrixPreferenceDataModel

26. The current Crab
>>> from crab.models import MatrixPreferenceDataModel
>>> m = MatrixPreferenceDataModel(data.data)

27. The current Crab
>>> from crab.models import MatrixPreferenceDataModel
>>> m = MatrixPreferenceDataModel(data.data)
>>> print m
MatrixPreferenceDataModel (7 by 6)
1 2 3 4 5 ...
1 3.000000 4.000000 3.500000 5.000000 3.000000
2 3.000000 4.000000 2.000000 3.000000 3.000000
3 --- 3.500000 2.500000 4.000000 4.500000
4 2.500000 3.500000 2.500000 3.500000 3.000000
5 --- 4.500000 1.000000 4.000000 ---
6 3.000000 3.500000 3.500000 5.000000 3.000000
7 2.500000 3.000000 --- 3.500000 4.000000

28. The current Crab

29. The current Crab
>>> #import pairwise distance

30. The current Crab
>>> #import pairwise distance
>>> from crab.metrics.pairwise import
euclidean_distances

31. The current Crab
>>> #import pairwise distance
>>> from crab.metrics.pairwise import
euclidean_distances
>>> #import similarity

32. The current Crab
>>> #import pairwise distance
>>> from crab.metrics.pairwise import
euclidean_distances
>>> #import similarity
>>> from crab.similarities import UserSimilarity

33. The current Crab
>>> #import pairwise distance
>>> from crab.metrics.pairwise import
euclidean_distances
>>> #import similarity
>>> from crab.similarities import UserSimilarity
>>> similarity = UserSimilarity(m,
euclidean_distances)

34. The current Crab
>>> #import pairwise distance
>>> from crab.metrics.pairwise import
euclidean_distances
>>> #import similarity
>>> from crab.similarities import UserSimilarity
>>> similarity = UserSimilarity(m,
euclidean_distances)
>>> similarity[1]

35. The current Crab
>>> #import pairwise distance
>>> from crab.metrics.pairwise import
euclidean_distances
>>> #import similarity
>>> from crab.similarities import UserSimilarity
>>> similarity = UserSimilarity(m,
euclidean_distances)
>>> similarity[1]
[(1, 1.0),
(6, 0.66666666666666663),
(4, 0.34054242658316669),
(3, 0.32037724101704074),
(7, 0.32037724101704074),
(2, 0.2857142857142857),
(5, 0.2674788903885893)]

36. The current Crab
>>> #import pairwise distance
>>> from crab.metrics.pairwise import
euclidean_distances
>>> #import similarity
>>> from crab.similarities import UserSimilarity
>>> similarity = UserSimilarity(m,
euclidean_distances)
>>> similarity[1]
[(1, 1.0),
(6, 0.66666666666666663), MatrixPreferenceDataModel (7 by 6)
1 2 3 4 5
(4, 0.34054242658316669), 1 3.000000 4.000000 3.500000 5.000000 3.000000
(3, 0.32037724101704074), 2 3.000000 4.000000 2.000000 3.000000 3.000000
3 --- 3.500000 2.500000 4.000000 4.500000
(7, 0.32037724101704074), 4 2.500000 3.500000 2.500000 3.500000 3.000000
5 --- 4.500000 1.000000 4.000000 ---
(2, 0.2857142857142857), 6 3.000000 3.500000 3.500000 5.000000 3.000000
(5, 0.2674788903885893)] 7 2.500000 3.000000 --- 3.500000 4.000000

37. The current Crab

38. The current Crab
>>> from crab.recommenders.knn import UserBasedRecommender

39. The current Crab
>>> from crab.recommenders.knn import UserBasedRecommender
>>> recsys = UserBasedRecommender(model=m,
similarity=similarity, capper=True,with_preference=True)

40. The current Crab
>>> from crab.recommenders.knn import UserBasedRecommender
>>> recsys = UserBasedRecommender(model=m,
similarity=similarity, capper=True,with_preference=True)
>>> recsys.recommend(5)
array([[ 5. , 3.45712869],
[ 1. , 2.78857832],
[ 6. , 2.38193068]])

41. The current Crab
>>> from crab.recommenders.knn import UserBasedRecommender
>>> recsys = UserBasedRecommender(model=m,
similarity=similarity, capper=True,with_preference=True)
>>> recsys.recommend(5)
array([[ 5. , 3.45712869],
[ 1. , 2.78857832],
[ 6. , 2.38193068]])
>>> recsys.recommended_because(user_id=5,item_id=1)
array([[ 2. , 3. ],
[ 1. , 3. ],
[ 6. , 3. ],
[ 7. , 2.5],
[ 4. , 2.5]])

42. The current Crab
>>> from crab.recommenders.knn import UserBasedRecommender
>>> recsys = UserBasedRecommender(model=m,
similarity=similarity, capper=True,with_preference=True)
>>> recsys.recommend(5)
array([[ 5. , 3.45712869],
[ 1. , 2.78857832],
[ 6. , 2.38193068]])
>>> recsys.recommended_because(user_id=5,item_id=1)
array([[ 2. , 3. ],
[ 1. , 3. ], MatrixPreferenceDataModel (7 by 6)
1 2 3 4 5 ...
[ 6. , 3. ], 1 3.000000 4.000000 3.500000 5.000000 3.000000
2 3.000000 4.000000 2.000000 3.000000 3.000000
[ 7. , 2.5], 3 --- 3.500000 2.500000 4.000000 4.500000
[ 4. , 2.5]]) 4 2.500000 3.500000 2.500000 3.500000 3.000000
5 --- 4.500000 1.000000 4.000000 ---
6 3.000000 3.500000 3.500000 5.000000 3.000000
7 2.500000 3.000000 --- 3.500000 4.000000

43. Distributed Computing with mrJob
https://github.com/Yelp/mrjob

44. Distributed Computing with mrJob
https://github.com/Yelp/mrjob
It supports Amazon’s Elastic MapReduce(EMR) service, your own Hadoop cluster or
local (for testing)

45. Distributed Computing with mrJob
https://github.com/Yelp/mrjob
It supports Amazon’s Elastic MapReduce(EMR) service, your own Hadoop cluster or
local (for testing)

46. Distributed Computing with mrJob
https://github.com/Yelp/mrjob
"""The classic MapReduce job: count the frequency of words.
"""
from mrjob.job import MRJob
import re
WORD_RE = re.compile(r"[w']+")
class MRWordFreqCount(MRJob):
def mapper(self, _, line):
for word in WORD_RE.findall(line):
yield (word.lower(), 1)
def reducer(self, word, counts):
yield (word, sum(counts))
if __name__ == '__main__':
MRWordFreqCount.run()
It supports Amazon’s Elastic MapReduce(EMR) service, your own Hadoop cluster or
local (for testing)

47. Studies with Sparse Matrices
Real datasets come with lots of empty values
http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html
Solutions:
scipy.sparse package
Sharding operations
Matrix Factorization
techniques (SVD)
Apontador Reviews Dataset

48. Studies with Sparse Matrices
Real datasets come with lots of empty values
http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html
Solutions:
scipy.sparse package
Sharding operations
Matrix Factorization
techniques (SVD)
Crab implements a Matrix
Factorization with Expectation
Maximization algorithm
Apontador Reviews Dataset

49. Studies with Sparse Matrices
Real datasets come with lots of empty values
http://aimotion.blogspot.com/2011/05/evaluating-recommender-systems.html
Solutions:
scipy.sparse package
Sharding operations
Matrix Factorization
techniques (SVD)
Crab implements a Matrix
Factorization with Expectation
Maximization algorithm
scikits.crab.svd package
Apontador Reviews Dataset

50. Why migrate ?
Old Crab running only using Pure Python
Recommendations demand heavy maths calculations and lots of processing
Compatible with Numpy and Scipy libraries
High Standard and popular scientific libraries optimized for scientific calculations in Python
Scikits projects are amazing!
Active Communities, Scientific Conferences and updated projects (e.g. scikit-learn)
Turn the Crab framework visible for the community
Join the scientific researchers and machine learning developers around the Globe coding with
Python to help us in this project
Be Fast and Furious

51. How are we working ?
Sprints, Online Discussions and Issues
https://github.com/muricoca/crab/wiki/UpcomingEvents

52. How are we working ?
Our Project’s Home Page
http://muricoca.github.com/crab

53. Future Releases
Planned Release 0.1
Collaborative Filtering Algorithms working, sample datasets to load and test
Planned Release 0.11
Evaluation of Recommendation Algorithms and Database Models support
Planned Release 0.12
Recommendation as Services with REST APIs
....

54. Join us!
1. Read our Wiki Page
https://github.com/muricoca/crab/wiki/Developer-Resources
2. Check out our current sprints and open issues
https://github.com/muricoca/crab/issues
3. Forks, Pull Requests mandatory
4. Join us at irc.freenode.net #muricoca or at our
discussion list
http://groups.google.com/group/scikit-crab

55. Recommended books
Toby Segaran, Programming Collective SatnamAlag, Collective Intelligence in
Intelligence, O'Reilly, 2007 Action, Manning Publications, 2009
Sites such as TechCrunch and ReadWriteWeb

56. Recommended Conferences
- ACM RecSys.
–ICWSM: Weblogand Social Media
–WebKDD: Web Knowledge Discovery and Data Mining
–WWW: The original WWW conference
–SIGIR: Information Retrieval
–ACM KDD: Knowledge Discovery and Data Mining
–ICML: Machine Learning

57. plore more the learning resource materials, one of the goals
of the e-learning system. Conference, Bratislava, 2005. 229-234.
[7] Romero, C., Ventura, S.: Educational Data Mining: a
References
3.5.3 Better Recommendation Understanding
The explanations can be quite helpful for users to better
understand if the given resource is (or is not ) suitable for
Survey from 1995 to 2005. Expert Systems with
Applications. Elsevier 1:33, 2007. 135-146.
[8] Romero, C., Ventura, S.: Data mining in e-learning.
th the 5. REFERENCES Wit Press, 2006.
them. Those explanations can be beneficial in how to for-
er en- mulate Brusilovsky, P., Peylo, C.: Adaptive and to simply and
[1] a better learning interaction and hence Intelligent [9] Atepassar. Available at: http://atepassar.com
ed the shorten the human/computer Systems. International Journal
Web-based Educational interaction. [10] J. B. Schafer, J. A. Konstan, and J. Riedl.:
rought of Artificial Intelligence in Education. 13, 2003, 156 - E-commerce recommendation applications. Data Mining
quality 169. and Knowledge Discovery, 2001.
nalized 4. [2] Conole, G. & Culver, J.: The design of Cloudworks:
CONCLUSIONS [11] Shen, L.P., Shen, R.M.: Learning Content
In this Applying social networking practice to foster the sys-
paper, we presents a personalized recommender
Recommendation Service Based-on Simple Sequencing
tem that incorporate different types of sourcesand knowledge
exchange of learning and teaching ideas of designs In: Specification. In Proceedings of Advanced in Web-based
in a adaptative educational social network by3using a meta-
Computers & Education , Vol. 54 , Nr. Elsevier, 2010 Learning 2004 363-
recommender approach. We believe that this technique will
, S. 679-692 .
bring more personalized andE., Grigoriadou, M., Samarakou, [12] M. Montaner, B. L’opez, and J. Rosa.: A taxonomy of
orking [3] Gogoulou, A., Gouli, better recommendations for the recommender agents on the internet. In Artificial
students. and Chinou, D.: A Web-based Educational Setting
M.,
mmen- Intelligence Review, volume 19, pages 285-330. Kluwer
ing to Supporting Individualized Learning, Collaborative
Academic, 2003.
Although we have integrated the In ProceedingsAtePassar, it
Learning and Assessment. system in the of
tion of [13] J. Herlocker, J. Konstan, and J. Riedl.: Explaining
e that can, in Educational Technology & Society. 2007, 242-256.
principle, also be used in other web-based educational
systems. MoreJ.B.: The application of data-mining the area
[4] Schafer, research need to be conducted in to
collaborative filtering recommendations. In In
educa- proceedings of ACM 2000 Conference on Computer
mprove of the recommendation systemsJ. Wang (Ed.), Encyclopedia
recommender systems. In in social learning and how
they can data warehousing and mining. Hershey, PA. Idea
of influence the student satisfaction, motivation and Supported Cooperative Work, 2000.
learning process. 44 - 48.
Group 2005 [14] B. Schafer, J. Konstan, and J. Riedl.:
[5] Farzan, R., Brusilovsky, P.: Social Navigation Support Meta-recommendation systems: user-controlled
in a Course Recommendation System. In proceedings of
Bringing machine learning and artificial intelligence to edu- integration of diverse recommendations. In Proceedings
rces in of the 11th conference on Information and knowledge
of the cational social networks Conference on main contributions of
4th International is one of the Adaptive Hypermedia
this work, Adaptive resulting on a new generation of web-
and which is Web-based Systems. Dublin, 2006. management, pages 43-51. ACM Press, 2002.
of the
91-100.
educational systems. The task of recommendation in web- [15] Crab. Available at: http://muricoca.github.com/crab
ve the
mation based education A.: Recommender System for Adaptive col-
[6] Ksristofic, can be considered as special type of [16] AtePassar Video. Available at:
nd ex- laborative learning, due to the fact that they of Informatics
Hypermedia Applications. In Proceeding help student http://www.youtube.com/watch?v=gzn7I1EJmb4
e goals and Information Technology Student Research
with similar interests to find an optimal path through the
learning material and facilitate the 229-234.
Conference, Bratislava, 2005. social network design to
[7] Romero, C., Ventura, S.: Educational Data Mining: a
be effectively be used to achieve these targets. We provide
a videoSurvey from 1995 to recommender Systemsin action at
demonstrating the 2005. Expert system with
Applications. Elsevier 1:33, 2007. 135-146.
AtePassar, which can be accessible at œ[16].
better
[8] Romero, C., Ventura, S.: Data mining in e-learning.
ble for
Wit Press, 2006.
to for-
ly and [9] Atepassar. Available at: http://atepassar.com
[10] J. B. Schafer, J. A. Konstan, and J. Riedl.:
E-commerce recommendation applications. Data Mining
and Knowledge Discovery, 2001.
[11] Shen, L.P., Shen, R.M.: Learning Content
er sys-
Recommendation Service Based-on Simple Sequencing
wledge

58. AtéPassar
-­‐
hOp://atepassar.com
Data
available
for
researching
-­‐
marcel@atepassar.com

59. Where will you be ?
Source: Hunch.com
Thanks !!

60. Content
Recommenda,on
Based
on
Data
Mining
in
Adap,ve
Social
Networks
Marcel
Pinheiro
Caraciolo
marcelcaraciolo@acm.org