In this paper we demonstrate how a gamification approach increases the attractiveness of an assessment exercise in the context of expertise profiling. We present an online game, in two difficulty modes, where users have to guess the authors of publications. We analyze the collected data along different dimensions and identify four types of gaming personalities based on behavioral patterns. Further, we examine the relation between popularity and recognizability for both papers and authors. Finally, we provide insights into game mechanics that extend beyond our specific use case.

4. Expertise Profiling
01 / 20
http://bit.ly/expertime

5. Expertise Profiling
02 / 20
http://bit.ly/expertime

6. Expertise Profiling
03 / 20

7. Gamification
04 / 20
๏ question-answering quiz
๏ IR publications
(1111 papers from SIGIR, WWW, CIKM, KDD & WSDM)
๏ two difficulty modes
๏ max 3 mistakes; time limit
๏ goal: answer as many questions as possible
๏ motivation for players: position on the leader board

8. Research questions
05 / 20
๏ Which level of difficulty is preferred?
๏ Does a competitive element, such as a leader board,
increase the level of engagement?
๏ When do users stop playing?
๏ Do users return to play again? After how long?
๏ What types of players can we identify?
๏ Are more cited papers also more easily recognized?
๏ Are more popular authors also more easily recognized?
๏ Do people prefer to play anonymously?

9. IR Game
06 / 20
a) Beginner mode:
http://bit.ly/ir-game

10. IR Game
06 / 20
b) Advanced mode:
http://bit.ly/ir-game

11. IR Game
06 / 20
b) Advanced mode:
http://bit.ly/ir-game

12. Usage stats
09 / 20
Duration of measurements:
# webpage visitors:
# game players:
# games played:
# games (beginner / advanced):
# avg. #games per player:
5 days (Jan 31 - Feb 4, 2015)
302 (from 33 countries)
116
387
347 / 39
3.34

13. Analysis of results
by answers
10 / 20
Time limit (both game modes):
Avg. time to answer (beginner / advanced):
15 s
6.7 s / 8.5 s

14. Analysis of results
by answers
11 / 20
Score distribution
a) beginner mode b) advanced mode

15. Analysis of results
by players
12 / 20
Returning players
# players who played more than 1x:
# games played within 1 hour:
56
42
Time elapsed between games

16. Analysis of results
by players
13 / 20
Player type: Jumper

17. Analysis of results
by players
14 / 20
Player type: Give-uper

18. Analysis of results
by players
15 / 20
Player type: Fighter

19. Analysis of results
by players
16 / 20
Player type: Achiever

20. Analysis of results
by papers
17 / 20
Paper’s recognition ratio
Number of times the publication was successfully recognized by players
divided by number of times it was shown to players.
Citation counts vs. recognition ratio

21. Analysis of results
by authors
18 / 20
Author’s recognition ratio
Number of times the author’s publications was successfully recognized by players
divided by total number of times her publications were shown to players.
Number of author’s publications vs. recognition ratio.

22. Observations
19 / 20
๏ Learning
The game was useful for learning about new publications.
๏ Unfair behavior
The best scoring user had the longest response time.
๏ Head-start
A user was restarting the game until he was able to answer the first question
correctly.
๏ Engaging users
It is important to keep the user stay in the game as long as possible when she
comes for the first time.
๏ Identity
Some people (∼ 10%) opted to use their full civil name as opposed to a nickname.

23. Conclusions
20 / 20
๏ Could we make an assessment exercise, in the context of expertise
profiling, more appealing for users?
Yes.
๏ We gathered valuable data about authors and publications.
Future work
๏ Introduce a controlled bias into selection of alternative answers.
๏ Adjust the difficulty of the questions as the game progresses.
๏ New game modes.
๏ Expand to other research fields.

24. Questions?

25. Thank you

26. Credits
© Luboš Volkov
© iconsmind.com
© Cédric Villain
© convoy
© baabullah hasan
© Luis Prado
© Mark Shorter
© Gerhard Meier
Presented by
Jenda Rybák at ECIR 2015, Wien