This document describes a study that analyzed the effects of different components of games with a purpose (GWAP) on output quality and motivation. The study tested 5 interfaces that varied the inclusion of feedback, social interaction, and gaming environment components. It found that interfaces with gaming environment components collected significantly more high-quality labels. It also found that interfaces with social interaction motivated more volunteers to participate voluntarily in the future. The study provides empirical evidence that gaming environment improves output quality in GWAPs, while social interaction increases participant motivation.

1. Systematic Analysis of
Output Agreement Games:
Effects of Gaming Environment,
Social Interaction, and Feedback
Shih-Wen Huang and Wai-Tat Fu

2. GWAP Today – ESP Game
[Von Ahn and Dabbish, CHI 2004]

3. GWAP Today - Foldit
[Cooper et al., Nature 2010]

4. Two Unanswered Questions
1. Can GWAP collect outputs with high quality?
Why?
2. Which component of GWAP motivate
volunteers to work?
• To answer these questions, a systematic analysis
that tests the effects of different components of
GWAP needs to be done

5. Output Agreement Games

6. Output Agreement Games
Database
1. Randomly match
multiple strangers
and provide them with
the same input
Game Interface

7. Output Agreement Games
Database
1. Randomly match
multiple strangers
and provide them with
the same input
Game Interface
2. Players are asked to
generate outputs that are
related to the input

8. Output Agreement Games
Database
1. Randomly match 3. Players will be rewarded
multiple strangers if the outputs are matched.
and provide them with The matched outputs would
the same input be used to describe the input
Game Interface
2. Players are asked to
generate outputs that are
related to the input

9. Three Major Components
• Gaming Environment (G): Encouraging players
to generate matched answers using scoring
system, leaderboard, etc.
• Social Interaction (S): Players need to
cooperate with other players
• Feedback (F): Players can implicitly learn the
answers of other players

10. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction

11. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction

12. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G) Teammate searching page Teammate waiting page
B: baseline F: feedback G: gaming environment
S: social interaction

13. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction

14. Five Different Interfaces
• (B)
• (F)
• (F+S)
• (F+G)
• (F+S+G)
B: baseline F: feedback G: gaming environment
S: social interaction

15. Experimental Design
• 150 workers (30 per interface) were recruited
from Amazon Mechanical Turk within two
weeks.
• Each worker earned $0.05 for generating 20
labels
• A label was considered as a high-quality label
if its distance to a gold standard label < 1

16. Gaming Environment Helps Collect
More High-Quality Labels
Interfaces with
Gaming Environment
collect significantly
more high quality
labels than other
three interfaces
Amount of high-quality labels collected by each interface

17. Social Interaction Motivates
More Volunteers
Interfaces with
Social Interaction
motivate more
volunteers than
other three
interfaces
Percentage of workers reported that they
would love to do the task again voluntarily

18. Implications for Human Computation
1. Gaming Environment allows output
agreement games to collect more high-
quality outputs
2. Social Interaction helps motivate more
volunteers in output agreement games

19. Questions?