Innovation Platforms for increasing impact of research in Mozambique & India
Methods and techniques (2/6)
1. Methods and techniques to
analyze and design incentivized
semantic applications
www.insemtives.eu
Roberta Cuel, University of Trento, IT and
Markus Rohde, University of Siegen, DE
ISWC 2010
2. 1/29/2015 www.insemtives.eu 2
Why is my app not as successful as
Facebook…
1/29/2015 www.insemtives.eu 2
• Motivation and incentives
– Reciprocity
– Reputation
– Competition
– Altruism
– Self-esteem
– Fun
– Money
• This session is about the methods
and techniques you can use
– To study your scenario prior to
application design
– To evaluate your incentives strategy
and adjust your application
3. Overview
• Basic notions and definitions
• Methods and techniques for analysis, design
and evaluation
• Case studies
1/29/2015 www.insemtives.eu 3
4. BASIC NOTIONS AND DEFINITIONS
Basic notions and definitions
Methods and techniques for analysis, design and evaluation
Case studies
1/29/2015 www.insemtives.eu 4
5. Starting with motivation…
• Basic tenets of organizational behavior
– Performance : f(ability*motivation)
1/29/2015 www.insemtives.eu 5
6. Motivation and incentives
• Intrinsic vs extrinsic motivations
• Incentives are ‘rewards’ assigned to performer
to make him/her commit
– Can be totally uncorrelated to the nature of the
task
– Need to be compliant with the values and beliefs
of the acting person
1/29/2015 www.insemtives.eu 61/29/2015 www.insemtives.eu 6
7. Example: DBPedia
The DBpedia community collaborates using
the following tools
– Mailing lists, Facebook group
– Blog, quality assurance and bug tracking
– Development
• “The framework is easily extended and we offer the possibility to do this
yourself, increasing the likelihood of your desired data being included
in the next DBpedia release.”
– DBpedia consulting (money)
1/29/2015 www.insemtives.eu 7
8. 1/29/2015 www.insemtives.eu 8
Typology of motivations
Motivations Internal
(embedded in structure,
e.g., task, tools)
External
(additional to structure,
external re-inforcements)
Intrinsic
(predispositioned in
person, e.g., drives,
needs, desires )
Fun, joy, gaming,
interest, satisfaction,
self-actualization, self-re-
inforcement
Social appreciation,
reputation, love, trust,
social capital, community
support
Extrinsic
(additional to personal
predispositions, extern re-
inforcements )
Usability, sociability,
Design-for-fun, curiosity,
community-building
support
Material/financial capital,
money, rewards, prices,
medals, credit points
Structure
Person
Example: FLOSS software (Ghosh & Prakash, in Lerner & Tirole, 2005)
9. METHODS AND TECHNIQUES FOR
ANALYSIS, DESIGN AND EVALUATION
Basic notions and definitions
Methods and techniques for analysis, design and evaluation
Case studies
1/29/2015 www.insemtives.eu 9
10. Overview of methods and
techniques
• To analyze your scenario and design your
application
– Game theory, mechanism design
– Participatory design and end-user development
• To evaluate an incentives strategy in its context
– Interviews and questionnaires and field experiments
– Heuristic evaluation, guideline reviews
– Cognitive and pluralistic walk-throughs
– Participatory/User evaluation
1/29/2015 www.insemtives.eu 101/29/2015 www.insemtives.eu 10
11. Game theory
• Game theory is a formal way to analyze interaction among a
number of rational agents who behave strategically
– The rational agents: players involved in the situation (best choice)
– A number of players: more than one
– Rationality/payoffs: what are the players’ preferences over the
outcomes of the game
– The interactions: one player’s behavior affects another
– The rules: who moves when, what do they know, what can they do
– The outcomes: what is the outcome of the game (for each move)
1/29/2015 www.insemtives.eu 11
12. Mechanism design
• Mechanism design is about how to translate game
theory in effective behavior
– To design rules such that a desired set of outcomes
happens
– Alignment of interests between parties and production of
maximum social welfare
• Relevant variables
– Structural and personal motivations
– Goal and nature of good produced
– Tasks and skills required
– Social structure
1/29/2015 www.insemtives.eu 12
13. Analysis matrix
• A multidimensional tool that allows for an
analysis of a particular task with respect to the
variables relevant for mechanism design
1/29/2015 www.insemtives.eu 13
Goal Tasks Social structure Nature of good
produced
Skill
variety/level
Communication
level
Variety of Hierarchy-
neutral
Private good Skilled ability
Participation
level
Specificity of Hierarchical Public good Competence
Clearness level Identification
with
Common
resource
Club good
14. 1/29/2015 www.insemtives.eu 14
The matrix in practice: semantic
annotation
• Goal of the annotation or ontology population exercise
• Task, or more typically, an ordered collection of tasks into
which the annotation exercise can be broken down
• Social structure, a stylized and simplified set of social
relationships among the subjects participating in the
exercise
• Nature of good, a stylized description, in game-theoretical
terms, of the relationship between what good is produced
and who consumes it
• Required skills of the agents to complete the annotation
task
1/29/2015 www.insemtives.eu 14
15. The matrix in practice: crowd-sourcing
of ontology evolution
• Goal: communication and participation level
– Coordination (free-open)
• Interests are already aligned (free riding)
• Private and public benefit coincide
• Task: variety vs specificity
• Social structure: hierarchy neutral (large vs small
groups)
• Nature of good: public good non-rival (use does not
limit use of others), non-exclusive (open access)
• Skill variety/level: Skilled ability
1/29/2015 www.insemtives.eu 151/29/2015 www.insemtives.eu 15
16. Participatory Design (PD)
• To develop software for the participation of end-users, we
propose a participatory way of designing these software
tools, integrating potential users by participatory design
methods
• Participatory Design
– Improvement of the participation of workers in software development
processes
– The cooperation between software developers and end-users
– Participate in IT development projects as experts of their own work
1/29/2015 www.insemtives.eu 16
17. 1/29/2015 www.insemtives.eu 17
Methods for PD
• Participatory-design projects combine
– Design-by-doing methods
– PD workshops
– Scenarios
– Different forms of prototyping (mockups, rapid
prototypes)
– Work organization games
– Ethnographic methods
18. 1/29/2015 www.insemtives.eu 18
End-User Development (EUD)
• EUD is about designing highly flexible systems that enable
users to participate during the use of the system by
adapting and modifying the tools according to their
needs/preferences
• Goal: Empower end-users to develop and adapt systems
themselves by designing these systems to be easy to
– Understand
– Learn
– Use
– Teach
19. 1/29/2015 www.insemtives.eu 19
How to design an incentivized
application
• Ideally: field desk lab field
• Analyze the domain and find yourselves in the matrixes
– Find the relevant point of that situation (goal and tasks)
– Focus on a small group of individuals (social structure)
– Analyze their motivation (internal/external intrinsic/extrinsic)
– Analyze the other relevant variables (nature of good being
produced, kill variety/level)
• Design a simplest possible model that can effectively
support contributors
• Test and get feedback
• Fine-tune the experiment and add other elements
21. CASE STUDIES
Basic notions and definitions
Methods and techniques for analysis, design and evaluation
Case studies
1/29/2015 www.insemtives.eu 21
22. Examples from our project
• Telefónica Investigación y Desarrollo (TID - Spain): Knowledge
Intranet Platform
• Pepper’s Ghost (PGP-UK): Virtual Games/Worlds
• Seekda! (Austria): Web Services Search Portal
1/29/2015 www.insemtives.eu 22
23. 1/29/2015 www.insemtives.eu 23
Field work: TID
• Domain analysis
– Site visit, semi-structured, qualitative interviews
• Communication processes
• Existing usage practices and problems
• Existing tools/solutions
• Semantic annotation solutions
– Tape recording, transcription
– Data analysis per ex-post categorization
• Focus group discussion
– Usability lab tests
– Expert walkthroughs
24. 1/29/2015 www.insemtives.eu 24
Field work: TID (2)
1/29/2015 www.insemtives.eu 24
• Find TID in the matrix
– Goal: communication and participation (contribute to the
knowledge portal)
– Social structure (various structures co-exist)
• Strongly hierarchical organization (control is an issue)
• working groups and community of experts
– Nature of good: public good vs. private, club goods
– Skill variety/level: Skilled ability (knowledge workers)
– Motivations: fun, visibility, reputation, promotion, money
25. 1/29/2015 www.insemtives.eu 25
Field work: PGP
• Domain analysis
– Data analysis (Benchmark: Galaxyzoo, Moonzoo)
– Usability lab tests and expert walkthroughs
– Collect data on users such as logs files
• Find PGP in the matrix
– Goal: participation level
– Task: specific
– Social structure: hierarchy neutral
– Nature of good: public good non-rival (use does not limit
use of others), non-exclusive (open access)
– Skill variety/level: non skilled ability
26. 1/29/2015 www.insemtives.eu 26
Field work: SEEKDA
• Domain analysis
– Interviews and workshops
– Replicating Telefonica model
• Find SEEKDA in the matrix:
– Goal: communication and participation level
– Task: variety vs. specificity
– Social structure: hierarchy neutral (mainly)
– Nature of good: public good non-rival (use does not limit
use of others), exclusive, or club good
– Skill variety/level: Skilled ability