In this paper, we propose a competitive human computation game, KissKissBan (KKB), for image annotation. KKB is different from other human computation games since it integrates both collaborative and competitive elements in the game design. In a KKB game, one player, the blocker, competes with the other two collaborative players, the couples; while the couples try to find consensual descriptions about an image, the blocker’s mission is to prevent the couples from reaching consensus. Because of its design, KKB possesses two nice properties over the traditional human computation game. First, since the blocker is encouraged to stop the couples from reaching consensual descriptions, he will try to detect and prevent coalition between the couples; therefore, these efforts naturally form a player-level cheating-proof mechanism. Second, to evade the restrictions set by the blocker, the couples would endeavor to bring up a more diverse set of image annotations. Experiments hosted on Amazon Mechanical Turk and a gameplay survey involving 17 participants have shown that KKB is a fun and efficient game for collecting diverse image annotations.

1. KissKissBan
A Competitive Human Computation
Game for Image Annotation
Chien-Ju Ho*, Tao-Hsuan Chang+, Jong-Chuan Lee+,
Jane Yun-jen Hsu+, Kuan-Ta Chen*
*Academia Sinica
+National Taiwan University

2. Human Computation Game
Collaborative Mechanisms

3. Coalition Problem
Limited Data Diversity

4. The ESP Game
• Coalition problem
• partner with themselves
• unified strategy
• Limited diversity
• easy words first

5. Competitive Mechanism?

6. Player 1 Player 2
DOG CUTE
FLOWER WOOD
FLOOR DOG

7. Player 1 Player 2
DOG CUTE
FLOWER WOOD
FLOOR DOG
DELIVER EMOTION BY TYPING THE SAME WORD

8. Couple Couple
DOG CUTE
FLOWER WOOD
FLOOR DOG
DELIVER EMOTION BY TYPING THE SAME WORD

9. Blocker
Couple Couple
DOG
DOG CUTE
WHITE
FLOWER WOOD
FLOOR DOG

10. Blocker
Couple Couple
DOG
DOG CUTE
WHITE
FLOWER WOOD
FLOOR DOG
KissKissBan
A Competitive Human Computation
Game for Image Annotation

11. Blocker
Couple Couple
DOG CUTE
DOG Blocked
WHITE WOOD
FLOWER
FLOWER
FLOOR The blocked words
-5s game time -5s game time
1. First, the blocker has 7 seconds to type words
2. Then, the couples has 30 seconds to type words

12. Blocker
Couple Couple
DOG CUTE
DOG
WHITE WOOD
FLOWER
FLOWER
FLOOR
Couples win for matching within time limit
1. First, the blocker has 7 seconds to type words
2. Then, the couples has 30 seconds to type words

13. Screenshots of KissKissBan

14. Intuitions Behind the Game

15. Incentive Structure
• Zero sum game
• strict competitive
• players can not benefit from coalition
• Different decision time periods
• harder words, longer consideration [Barry et al. 2001]
• diverse data contribution

16. Small-scale Experiment
• Image Source
• 125 images randomly from ESP dataset
• Experiment Setup
• 17-player gameplay survey
• Data collection on Mechanical Turk

17. Gameplay Survey
• Fun
• 3.76/5 overall rating
• 15/17 players claim that would play again
• over 60% like to play as a blocker

18. MTurk Experiment
• Collecting data from anonymous users
• Publish two games for comparison
• KissKissBan
• ESP Game without taboo words
• Each game is played for ~5000 rounds

19. Example of Collected Data
ESP ML-KKB BL-KKB
beach 3
sea 9
water 3
man 21 man 8
sand 3
beach 10 ocean 3
sea 2
karate 5 black 1
ninja 1
water 1 china 1
kungfu 1
sand 1
ocean 1
ML-KKB: matching between couples
BL-KKB: matching between couple and blocker

20. MTurk Experiment
• Data property
• For each image
• ESP: 39.96 labels, 6.56 distinct labels
• KKB: 44.17 labels, 11.53 distinct labels
• A more diverse player contribution

21. Conclusion
• We proposed a competitive human
computation game, KissKissBan.
• KissKissBan has two nice properties
• a player-level anti-coalition mechanism
• diverse player input

22. More in the Future
• Publish the game on Facebook within the
next one to two months.
• Study the different tagging behaviors of the
blocker and the couples.
• Apply the competitive mechanism to more
general human computation game design.

23. Q &A