1. A Draw-and-Guess Game to
Segment Images
Luca Galli, Piero Fraternali
Davide Martinenghi, Marco Tagliasacchi
Politecnico di Milano, Italy
Email: first.last@polimi.it
Jasminko Novak
European Institute for Participatory Media,
Germany
Email: j.novak@eipcm.org

2. The CUbRIK Project
● CUbRIK is a research project
financed by the European Union
● Goals:
● Advance the architecture of
multimedia search
● Exploit the human
contribution in multimedia
search
● Use open-source
components provided by
the community
● Start up a search business
ecosystem
● http://www.cubrikproject.eu/
Luca Galli

3. Humans in Multimedia Information
Retrieval
● Problem: the uncertainty of analysis algorithms leads to low confidence
results and conflicting opinions on automatically extracted features
● Solution: humans have superior capacity for understanding the content
of audiovisual material
● State of the art: humans replace automatic feature extraction processes (human
annotations)
● Our contribution: integration of human judgment and algorithms
– Goal: improve the performance of multimedia content processing
Luca Galli

4. 4
The Fashion Trend Mining Scenario
● Problem statement: segment fashion images for mining trends based on visual
features of garments (e.g. color and texture)
Color descriptors
coarse
(sub-)image
similarity
Texture descriptors
● Use case: identifying trends in collections of images of people and garments
● Applications: retrieving similar garments, inspect clothing trends in image
collections, analyzing trends change in the years
Luca Galli

5. 5
Issues of traditional approaches
 Problems in automatic feature detection:
 Body part and feature recognition is affected by the posture of the
portrayed subject
?
 Problems arise when part of the body is occluded or in pictures with
complex human poses
Luca Galli

6. 6
Solution: Human Contribution
 Issues are solved by augmenting the capabilities of algorithms
with human intervention.
 Humans have superior capacity to understand complex content and
to perform high level abstractions on visual representations
 For these reasons, they are suitable for tasks such as:
● Body part recognition
● Garments segmentation
● Gender identification
Luca Galli

7. The CUbRIK Architecture
Content and users
registration in the
system
Managing tasks and
processing contents
Provides front end
for issuing queries
and viewing results
Search Engines used to
access the content and
annotations
Luca Galli

8. 8
Users Motivation: Games with a Purpose
● Users need motivation to complete a task in a human computation
platform, such as monetary rewards, prestige, entertainment.
● Games with a Purpose (GWAPs) are digital games that generate useful data as
a by-product of play.
● The design of a GWAP requires to create a game so that its structure
encourages computation, correctness of the output and players retention.
● Introduced by Luis Von Ahn with the ESP Game, several GWAPs have been
developed with a number of different purposes
Luca Galli

9. 9
The Fashion Trend Mining Pipeline
Male, 24 Female, 22 Female?, ??
Luca Galli

10. 10
Gwap Design: Sketchness!
● Multiplayer Puzzle Game
● Established genre: Guess and Draw
(Pictionary, iSketch…)
● Inversion Problem Mechanic
● Players take turns into drawing the
shapes of objects inside an image
in order to make the other players
guess the underlying object.
● Two different roles:
Sketcher
Guesser
● Objective: Garments tagging, body part
segmentation, garment segmentation
Luca Galli

11. 11
Player Role: Sketcher
Tag of the target object
● The only player to see the
low confidence image
● “May” be asked to provide a
tag for the image
● Is asked to draw the contour
of the object for which the
tag is provided within the
allotted time
● Goal of the Sketcher is to
let the other players guess
the tag without providing
any other hints than the
contour Fashion image provided Contour of the garment
by the pipeline provided by the player
Luca Galli

12. 12
Player Role: Guesser
● Any other player in the game
● His/Her goal is to guess the
object for which the Sketcher
has provided the contour
● Not allowed to draw on the
whiteboard, just to type
in the chat box the probable
answer as fast as possible
● Scoring:
● Sketcher: 10 pts + 1 for each guesser
● Guesser: 10 pts to the first, then Contour of the garment
decreasing down to five provided by the Sketcher
Luca Galli

13. 13
Task Injection:
new approach to GWAPs
● Traditional gwaps developed on ad-hoc basis and tailored for a specific task
● Resulting gwaps may be not fun, unappealing to players or provide an
experience that may still perceived by the users as work
● The proposed GWAP exploits suitable modified game mechanics of a well
known and appreciated category of games, called “drawing and guessing
games” to implicitly solve segmentation problems while playing
Luca Galli

14. 14
Future Directions
● The proposed game concept is still under development and there are still
several open questions and future improvements:
● Task Injection:
● Does the task injection approach guarantee the same level of
entertainment and participation for the players as the original instance of
the game?
● How to measure entertainment anyhow?
● Output Aggregation:
● How the results provided by different players on the same input be
processed and grouped together in order to obtain a unique result?
● Majority voting not suitable for lines!
● Integration within CUbRIK’s Gaming Framework:
● Exploiting the features provided by CUbRIK in order to manage players
and their abilities and provide retention and reward features through
gamification techniques.
Luca Galli