market based recommender system

1. Search Engines Personalization
BRACHA SHAPIRA
BSHAPIRA@BGU.AC.IL
BEN-GURION UNIVERSITY

2. Personalization
 “Personalization is the ability to provide content
and services tailored to individuals based on
knowledge about their preferences and behavior”
[Paul Hagen, Forrester Research, 1999];

3. Acceptance of Personalization
 Overall, the survey finds that interest in personalization continues to be
strong with 78% of consumers expressing an interest in receiving some
form of personalized product or content recommendations.
ChoiceStream Research

4. Motivation for Search Engine Personalization
 Trying to respond to the user needs rather than to
her query
 Improve ranking tailored to user’s specific needs
 Resolve ambiguities
 Mobile devices – smaller space for results –
relevance is crucial

5. Search Engines Recommender Systems -
Two sides of the same coin????
 Search Engines  Recommender Systems
 Goal – answer users ad  Goal – recommend
hoc queries services of items to user
 Input – user ad-hoc  Input - user preferences
need defined as a query defined as a profile
 Output- ranked items  Output - ranked items
relevant to user need based on her preferences
(based on her
preferences???)

6. Search Engines Personalization Methods
adopted from recommender systems
 Collaborative filtering
 User-based - Cross domain collaborative filtering is required???
 Content-based
 Search history – quality of results????
 Collaborative content-based
 Collaborate on similar queries
 Context-based
 Little research – difficult to evaluate
 Locality, language, calendar
 Social-based
 Friends I trust relating to the query domain
 Notion of trust, expertise

7. Marcol- a collaborative search engine
Bracha Shapira, Dan Melamed, Yuval Elovici
 Based on collaborations on queries
 Documents found relevant by users on similar
queries are suggested to the current query
 An economic model is integrated to motivate users to
provide judgments.

8. MarCol Research Methods
System Architecture

9. MarCol Example

10. MarCol Example

11. MarCol Example

12. MarCol Example

13. MarCol Example
Ranking reward: up to 3

14. MarCol Ranking Algorithm
• Step 1: Locate the set of queries most similar to the current user
query.
Q'  Q  Sim(Squ , Lqi )  t1
Where:
Squ – a (“short”) query submitted by a user u
Q  {Lq1 , Lq2 ,..., Lqn } – the set of all (“long”) queries
Sim(Squ , Lqi ) – the cosine similarity between Squ and Lqi  Q
t1 – a configurable similarity threshold

15. MarCol Ranking Algorithm
• Step 2: Identifying the set of most relevant documents to the
current user's query.
D' (Q' )  D(Q' )  Sim(Squ , di )  t2
Where:
D(Q' ) – the set of all documents that have been ranked relevant to
queries in Q '
d i  D(Q' )
t 2 – a configurable similarity threshold

16. MarCol Ranking Algorithm
• Step 3: Ranking the retrieved documents according to their
relevance to the user query.
The relevance of document d i  D' (Q' ) to query Squ :
rel (Squ , di )  Sim(Squ , di )  Sim(Squ , qo )  J (di , qo )
Where:
Sim(Squ , di ) – similarity between user query and the document.
Sim(Squ , qo ) – similarity between user query and documents’ query (qo  Q' ).
J (d i , qo ) – the average relevance judgment assigned to the set of the documents
d i for the query qo (measured in a 1..5 scale).

17. Experiment Results – first experiment
Satisfaction
4.60
4.43
4.40
4.32
4.19 4.24
4.20
Satisfaction
4.00 4.00 3.95
3.88
3.80 3.78
3.74
3.66 3.66
3.60
3.47 MarCol Free MarCol
3.40
1 2 3 4 5 6
Sub-Stage
• There is not a significant difference between the modes
(p=0.822535) for a 99% confidence interval.

18. The properties of a pricing model
• Cost is allocated for the use of evaluation, and users are
compensated for providing evaluations.
• The number of uses of a recommendation does not affect its cost
(based on Avery et al. 1999). That value is expressed by the relevance of
a document to users query and the number of evaluations
provided for that document representing the credibility of
calculated relevance.
• Voluntary participation (based on Avery et al. 1999). The user decides
whether he wants to provide evaluations.
• The economic model favors early or initial evaluations.
Therefore, a lower price is allocated for early and initial
evaluations than for later ones and a higher reward is given for
provision of initial and early evaluations than for later ones.

19. Cost of document Calculation
• An item that has more evaluations has a higher price (until
reaching upper limit).
• An item that has few recommendations offers a higher reward for
evaluation.
• The price of an information item is relative to its relevance to the
current users query.
• The price is not affected by the number of information uses.

20. Document Cost Calculation
Pay (qu , d i ) – the price of document d i for a query qu
rel (qu , di ) min(  ,  )
Pay(qu , di )  
5 
Where:
 – the number of judgments
 – upper bound

21. Reward Calculation
reward (qu , d i ) – is the amount of MarCol points that a user is awarded for providing
an evaluation for document d i that was retrieved for query qu
rel (qu , di )   min(  ,   1)
Reward (qu , d i )  
5 
Where:
 – the number of judgments
 – upper bound

22. Experiment Methods
Independent variable:
• The only variable manipulated in the experiment is an
existence of the economic model.
Mode Short description
Users should pay “MarCol points” to
With economic access a document suggested by the
model system. While submitting a judgment, they
will be awarded with “MarCol points”
Users can freely access any suggested
Without economic
document and are not awarded by
model
submitting their judgments

23. Experiment Methods
The following questions (tasks) were used (Turpin and Hersh 2001):
1. What tropical storms hurricanes and typhoons have caused property
damages or loss of life?
2. What countries import Cuban sugar?
3. What countries other than the US and China have or have had a
declining birth rate?
4. What are the latest developments in robotic technology and it use?
5. What countries have experienced an increase in tourism?
6. In what countries have tourists been subject to acts of violence
causing bodily harm or death?

24. Experiment Procedure
• There were six equal subgroups, while every subgroup was given
its unique sequence of questions (a Latin square).
• There were six sub stages; on each sub stage the participants were
provided with a different question.
Substage
1 2 3 4 5 6
1 5 1 3 4 2 6
Participants
Subgroup
2 6 5 4 2 3 1
3 2 4 1 6 5 3
4 1 3 2 5 6 4
5 4 6 5 3 1 2
6 3 2 6 1 4 5

25. Experiment Results – first experiment
Performance
100%
98%
96%
94%
92%
Performance
90%
88%
86%
84%
82%
MarCol Free MarCol
80%
1 2 3 4 5 6
Sub-Stage
• There is a significant difference between the modes (p≈0) for a
99% confidence interval.

26. Experiment Results – second experiment
100%
Performance
90%
80%
Performance
70%
60%
50%
MarCol Free MarCol
40%
1 2 3 4 5 6
Sub-Stage
• There is a significant difference between the modes (p≈0) for a
99% confidence interval.

27. Experiment Results – first experiment
Participation
3.50
MarCol Free MarCol
3.00 3.00
2.92
2.71
2.63
2.50 2.50
Participation
2.33
2.29 2.25
2.00
1.67
1.63
1.50
1.21 1.17
1.00
1 2 3 4 5 6
Question
• There is a significant difference between the modes (p=0.008204)
for a 99% confidence interval.

28. Experiment Results – first experiment
Accumulated Participation
18.00
MarCol Free MarCol
16.00
15.63
14.00
13.33
12.00
10.67
Participation
10.00 10.42
8.88
8.00 7.54 7.54
6.00
4.38 5.88
4.00 4.21
2.42
2.00
1.46
0.00
1 2 3 4 5 6
Sub-Stage

29. Experiment Results – first experiment
Accumulated Participation
18.00
MarCol Free MarCol
16.00
15.63
14.00
13.33
12.00
10.67
Participation
10.00 10.42
8.88
8.00 7.54 7.54
6.00
4.38 5.88
4.00 4.21
2.42
2.00
1.46
0.00
1 2 3 4 5 6
Sub-Stage

30. Experiment Results – second experiment
Participation
1.80
MarCol Free MarCol
1.60 1.60
1.40
1.20
1.10
Participation
1.00
0.90 0.91
0.80
0.70
0.60 0.55
0.50 0.50
0.40 0.45
0.36
0.27
0.20 0.18
0.00
1 2 3 4 5 6
Question
• There is a significant difference between the modes (p=0.000164)
for a 99% confidence interval.

31. Experiment Results – second experiment
Accumulated Participation
6.00
MarCol Free MarCol
5.30
5.00
4.30
4.00
Participation
3.80
3.10
3.00
2.73
1.90 2.00 2.36
2.00
1.10 1.36
1.00
0.55 1.00
0.00
1 2 3 4 5 6
Sub-Stage

32. Experiment Results – first experiment
Satisfaction
4.60
4.43
4.40
4.32
4.19 4.24
4.20
Satisfaction
4.00 4.00 3.95
3.88
3.80 3.78
3.74
3.66 3.66
3.60
3.47 MarCol Free MarCol
3.40
1 2 3 4 5 6
Sub-Stage
• There is not a significant difference between the modes
(p=0.822535) for a 99% confidence interval.

33. Experiment Results – second experiment
Satisfaction
5.00
4.00
4.00
3.67 3.83
3.50 3.25
3.54
Satisfaction
3.00 3.33
2.90
2.38
2.39
2.22
2.00
1.25
1.00
MarCol Free MarCol
0.00
1 2 3 4 5 6
Sub-Stage
• There is not a significant difference between the modes
(p=0.746576) for a 99% confidence interval.

34. Summary of Results
• User performance is significantly better when using MarCol mode.
– The average superiority of is 6% in the first experiment, and 16% in the
second.
– The user performance superiority of MarCol increases as the task is more
difficult.
• User participation is significantly higher when using MarCol
mode.
– The average superiority of MarCol is 46% in the first experiment, and 96%
in the second.
– The user participation superiority of MarCol increases as the task is more
difficult.
– The participation grows constantly over time and so does the gap between
the MarCol and MarCol Free modes in both experiments.
• There is not any significant difference in user satisfaction between
the modes.

35. Conclusions and Trends
search engines personalization
 Search engines already integrate personal ranking
 Technology is yet to be developed to enahance
personalization
 Still needs evaluations to calibrate the degree of
personalization
 Privacy issues are to be considered

36.  Paper: Dan Melamed, Bracha Shapira, Yuval Elovici:
MarCol: A Market-Based Recommender
System. IEEE Intelligent Systems 22(3): 74-78
(2007)