How to Remove Document Management Hurdles with X-Docs?
Multiple objectives in Collaborative Filtering (RecSys 2010)
1. Multiple Objectives in Collaborative Filtering
Tamas Jambor and Jun Wang
University College London
2. Structure of the talk
• Motivation
• Multiple objectives
• User perspective
– Promoting less popular items
• System perspective
– Stock management
3. Motivation
• In the RecSys community, many research efforts
are focused on recommendation accuracy
• And yet accuracy is not a only concern
• Practical recommender systems might have
multiple goals
4. Improved Accuracy != Improved User experience
Algorithm
Additional
factors
Available resources
Cost of delivery
User interface
Diverse choices
Profitability per item
Advertisement
5. Improved user experience
Available resources
Cost of delivery
User interface
Diverse choices
Profitability per item
Advertisement
Additional
factors
Accuracy
Improved
user
experience
6. Handling Multiple objectives
• Accuracy is the main objective
– Defined in the baseline algorithm
• User perspective
– Define and consider user satisfaction as priority
• System perspective
– Consider additional system related objectives
• Objectives of the system might contradict
7. Where to optimize?
• In the objective function or as a post-filter?
• Post-filters have the advantage to
– Add to any baseline algorithm
– Extend easily
– Add multiple goals
8. The proposed optimization framework
(for each user)
• Add additional constraints of w
0
11:tosubject
ˆmax
w
w
rw
T
T
w
9. Properties of the framework
• Linear optimization problem
• Recommendation as a ranking problem
• Constraints provide the means of biasing the
ranking
10. User case – Promoting the Long Tail
Current systems are biased towards popular items
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 8 9 10
Probabilityofanyof100mostpopularitembeingat
rankingposition
Ranking Position
SVD
User-based
Item-based
Random Sample
11. Promoting the Long Tail
• Does that reflect real user needs?
• Popular items might not be interesting for the user
• Discovering unknown item could be more valuable
• The aim is to reduce recommending popular items
– if the user is likely to be an interested in alternative
choices
– keep recommending popular items otherwise
12. Promoting the Long Tail
• Extending the optimization framework
0
11
:tosubject
ˆmax
w
w
mwm
rw
T
u
T
T
w
13. Promoting the Long Tail and Diversification
• Diversifying the results
0
11
:tosubject
ˆmax
w
w
mwm
wwrw
T
u
T
TT
w
14. Diversification
• Increase the covariance between recommended
items
– Reduce the risk of expanding the system
– Provide a wider range of choice
16. Evaluation metrics
• Recommendation as a ranking problem
• IR measures
– Normalized discounted cumulative gain (NDCG)
– Precision
– Mean reciprocal rank (MRR)
• Constraint specific measures
17. Results: Promoting the Long Tail
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 8 9 10
Probabilityofanyof100mostpopularitembeingat
rankingposition
Ranking Position
Baseline (SVD)
Long Tail Constraint
Long Tail Constraint and
Diversification (λ=6)
Random Sample
18. Results: Promoting the Long Tail
Baseline (SVD) LTC LTC and Div (λ=6)
NDCG@10 0.8808 0.8780 (-0.3%) 0.8715 (-1.0%)
P@10 0.8204 0.8207 (+0.2%) 0.8177 (-0.3%)
MRR 0.9518 0.9453 (-0.6%) 0.9349 (-1.7%)
19. System case – Resource Constraint
• Introducing external factors to the system
• Stock availability of recommended items
• The aim is to rank items lower, if less of them are
available
• Minimizing performance loss
20. Simulation
• Online DVD-Rental company
– Operates a warehouse
– Only a limited number of items are available
• Recommend items that are in stock higher in the
ranking list
21. Simulation
• User choice is based purely on recommendation
• Simulating the stock level for 50 days
– Present a list of items to a random number of users
– The probability that the item is taken depends on the
rank
– Cumulative probability depends on how many times the
item was shown and at which rank position
22. Cut-off point
• Threshold c controls the cut-off point from which
the system starts re-ranking items
cp
cps
s ti
titi
ti
,
,,
,
if0
if
24. Evaluation: Monitoring the waiting list size
• Waiting list
– If item is not in stock, user puts it on their waiting list
– When item returns, it goes out to the next user
• Waiting list size represents how long a user has to
wait to get their favourite items
28. Conclusion
• Recommender systems have multiple objectives
• Multiple optimization framework
– Expand the system with minor performance loss
– It is designed to add objectives flexibly
– It can be added to any recommender system
• Two scenarios that offer practical solutions
– Long-tail items
– Stock simulation
29. Future plan
• Personalized digital content delivery
– Reduce delivery cost
• Diversification and the long tail
– Does recommendation kill diversity?
• Evaluate improved user experience
– User studies
31. References
• Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms.
ACM Trans. Inf. Syst. 22(1) (2004)
• Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic
framework for performing collaborative filtering. In: SIGIR '99. (1999)
• Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for
recommender systems. Computer 42(8) (2009)
• Wang, J., de Vries, A.P., Reinders, M.J.T.: Unifying user-based and item-
based collaborative filtering approaches by similarity fusion. In: SIGIR '06:
Proceedings of the 29th annual international ACM SIGIR conference on
Research and development in information retrieval, New York, NY, ACM Press
Notas do Editor
Background statement
Explain what we mean by multiple objectives in this context
Give two examples
Motivated from the user point of view
Motivated from the system point of view
User point of view would improve the user experience
System point of view would take into account other external factors
Background statement
Explain what we mean by multiple objectives in this context
Give two examples
Motivated from the user point of view
Motivated from the system point of view
User point of view would improve the user experience
System point of view would take into account other external factors
Improving the accuracy of the system does not necessarily equal to improving user experience
Defined the recommender algorithm how good it performs on a metrics and how fast it can do its job
But we need additional factors that would define the whole system
For example if we take a VOD service
User interface affects how user reads, understands find information including the recommendation
How interesting the movies are that the user gets
System related factors include
The underlying hardware of the system
How much, how fast an item can be delivered
External factors include
How much the company earns on an item, for example some items have a higher profit margin
Personalised advertisement, recommending preferred items
User related and system related factors can directly improve user experience
And we suggest that the combination of accuracy and other factors might improve user experience
And the rest can help to maximize profit
Consider accuracy as the main objective
Combine these into a single optimization framework.
Take the prediction values of any baseline algorithm - r-hat.
And add additional constraints that are equally important.
Baseline algorithm to minimize errors.
Higher weights represent higher importance
Without constraints it returns the same as the original order
Recommendation is considered to be a ranking problem
Top-N list
It is a linear optimization problem, so global solution can be found.
Biasing the original order by using constraints
Introducing two case studies to illustrate the use of the framework
The first one is aim to promote items from the long tail.
Assumption
Current systems are biased towards popular items
Picked the first 100 items that have received the highest ratings
Where SVD is likely to place them
Plot the distribution
Figure show the probability that some popular are placed higher
on the recommendation list by all the widely used recommender algorithms
Does that reflect user needs?
We assume that discovering unknown items are more valuable
We aim to identify users who would like alternative choices
And recommend from the long tail for them
Keep recommending popular items if the user has a more mainstream taste
We add this as an inequality constraint to the framework
m is a vector that contains the mainstreamness value of each items in the recommended list
m_u measures the mainstreamness of the user
How these values are calculated are in the paper if you are interested
We added another extra bit to the framework to diversify items
Experimenting with diversification
Promote item from the long tail that differ from each other
Higher covariance
That would reduce the risk of such an extension
Since we approach recommendation as a ranking problem
We used the following IR measures
The probability that popular items ranked higher is significantly reduced
Only 32% of the users have popular items in the top position
The baseline is 45%
We get reduction until position four
Then it is slightly worse than the baseline
This is the case when user studies would provide a better way to evaluate performance
Long tail constraint alone
Long tail constraint with diversification
Slight performance loss for all measures, except one.
The other case – system case
Adding other non-user related factors to the system
the availably of certain items
The aim is to rank items lower if we are about to run out of stock
But also minimize performance loss
The second scenario was evaluated by simulating the stock level
Of an imaginary company for 50 days
We presented a recommendation list to a random number of users each day
The probability that a user took an item depended on the rank
The cumulative probability up to the present point was based on
a, How many times an item was show in the past
B, And at which ranking position
To evaluate the system we monitored the waiting list size
We introduce a cut-off point from which the system will start to reorganize the ranking list.
Adding it as a constraint to the system, in a same fashion
s is a vector that represents the probability that the item is available at the given time
For each item in the list
s_u controls how cautious the system should be with stock level.
e.g if s_u is higher the system starts penalizing items later
As they are getting less available
The experiment is designed to run out of stock
This graph shows the waiting list size for the first 20 days
With respect to parameter c
It controls when the system should start penalizing items as they are running out of stock
e.g if c is set to 0.8 then
after it is 80% likely that an item is taken the system starts penalizing
This table show the average and the maximum performance loss per day
As you can see from c=0.8 and above there is only a slight performance loss of the system.
Two papers
Optimizing algorithm from the user point of view
One way is to identify different errors
A general framework to handle multiple goals
Two scenarios to illustrate that
Improving user experience might be validated using user studies