I presented our work on games with purpose within the context of CrowdTruth.org, with specific focus on medical and biology domain

1. The Wisdom of the Crowd
through Gamification
Lora Aroyo, Merel van Empel & Carlos Martinez Ortiz
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

2. massive
digital
content
online
…
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

3. needs
massive
resources
to
process
…
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

4. crowdsourcing
–
interpreta5on
at
scale
…
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

5. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
paid
crowdsourcing
solves
only
part
…

6. there
are
massive
niches
of
experts
online
…
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

7. but
they
need
to
be
engaged
…
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

8. gamifica5on
is
the
answer
…
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo

9. • entertaining
• compe--ve
• quick
• s-cky
• targeted
campaigns
• effec-ve
engagement
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
games
with
a
purpose
(GWAP)

10. Winner EuroITV Competition
Best Archives on the Web Award
h7p://waisda.nl
Video Tagging – time-stamped tags

11. Image Tagging – understanding & training crowds

12. h7p://annotate.accurator.nl
Nichesourcing domain expertise

13. diversity
of
opinions
from
the
crowd
mul-tude
of
contexts
collected
in
a
decentralized
way
independent
interpreta5ons
aggregated
view
–
the
big
picture
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
understanding
perspec5ves

14. a
new
approach
to
understanding
seman5cs
harnessing
the
power
&
diversity
of
the
crowd
harnessing
disagreement
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
CrowdTruth.org

15. informs
a
vector
space
model
of
truth
instead
of
a
boolean,
fuzzy
or
sta-s-cal
model
disagreement
is
signal
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
CrowdTruth.org

16. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
CrowdTruth.org
http://game.CrowdTruth.org

17. h7p://game.crowdtruth.org

18. h7p://game.crowdtruth.org

19. h7p://game.crowdtruth.org

20. h7p://game.crowdtruth.org

21. h7p://game.crowdtruth.org

22. h7p://game.crowdtruth.org

23. h7p://game.crowdtruth.org

24. h7p://game.crowdtruth.org

25. h7p://game.crowdtruth.org

26. h7p://game.crowdtruth.org
• Games engage niche crowds - effective in filling the gap of
expert annotations in traditional crowdsourcing
• Annotator disagreement is signal, not noise:
• indicative of the variation in human semantic
interpretation of signs
• can indicate ambiguity, vagueness, similarity, over-
generality, etc, as well as quality
• Data analysis challenges
Take Home Message

27. h7p://game.crowdtruth.org

28. CrowdTruth
Team:
Lora
Aroyo,
Chris
Welty
Robert-­‐Jan
Sips,
Carlos
Mar5nez
Or5z,
Anca
Dumitrache,
Oana
Inel,
Benjamin
Timmermans,
Susanna
van
de
Ven,
Merel
van
Empel,
Jelle
v.d.
Ploeg,
Ta-ana
Cristea,
Khalid
Khamkham,
Harrië7e
Smook,
Rens
van
Honschooten,
Arne
Rutjes
CrowdTruth.org
github.com/CrowdTruth

29. http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo
CrowdTruth.org
http://game.CrowdTruth.org

30. massive
crowdsourcing
in
cultural
heritage
…
http://lora-aroyo.org ! http://slideshare.net/laroyo ! @laroyo