https://www.insight-centre.org/content/capability-requirements-approach-predicting-worker-performance-crowdsourcing
Presented at CollaborateCom 2013
Abstract:
Assigning heterogeneous tasks to workers is an important challenge of crowdsourcing platforms. Current approaches to task assignment have primarily focused on contentbased approaches, qualifications, or work history. We propose an
alternative and complementary approach that focuses on what capabilities workers employ to perform tasks. First, we model various tasks according to the human capabilities required to perform them. Second, we capture the capability traces of the
crowd workers performance on existing tasks. Third, we predict performance of workers on new tasks to make task routing decisions, with the help of capability traces. We evaluate the effectiveness of our approach on three different tasks including fact
verification, image comparison, and information extraction. The results demonstrate that we can predict worker’s performance based on worker capabilities. We also highlight limitations and extensions of the proposed approach.
Best VIP Call Girls Noida Sector 22 Call Me: 8448380779
A Capability Requirements Approach for Predicting Worker Performance in Crowdsourcing
1. Digital Enterprise Research Institute
www.deri.ie
A CAPABILITY REQUIREMENTS APPROACH FOR
PREDICTING WORKER PERFORMANCE IN
CROWDSOURCING
Umair ul Hassan, Edward Curry
Digital Enterprise Research Institute
National University of Ireland, Galway
9th IEEE International Conference on Collaborative Computing:
Networking, Applications and Worksharing
Austin, Texas, United States
October 20–23, 2013
Copyright 2010 Digital Enterprise Research Institute. All rights reserved.
2. Agenda
Digital Enterprise Research Institute
Motivation
Background
Task Modelling
Capability Requirements
www.deri.ie
Capabilities Taxonomy
Capability Tracing
Experiment
Summary
2
4. Motivation: Task Routing
Digital Enterprise Research Institute
www.deri.ie
Assigning heterogeneous tasks to heterogeneous workers
TASK MODELLING
Models
Models
Models
TASK ROUTING
WORKER PROFILING
Matching
Profiles
Profiles
Profiles
Task↔Worker
4
5. Proposal: Performance Prediction
Digital Enterprise Research Institute
www.deri.ie
Predict performance of workers on new tasks based on the
capabilities required for tasks and assign tasks accordingly
TASK MODELLING
Models
Models
Models
Capability
Requirements
Approach
TASK ROUTING
WORKER PROFILING
Matching
Profiles
Profiles
Profiles
Task↔Worker
Performance
Prediction
Capability Tracing
Model
5
6. Background: Micro tasks
Digital Enterprise Research Institute
www.deri.ie
When micro tasks are crowd sourced
Single person cannot do the task
Computers cannot do the task
Work can be split into smaller tasks
Some online microtask platforms
6
7. Background: Micro tasks
Digital Enterprise Research Institute
www.deri.ie
Most common tasks in Amazon Mechanical Turk (AMT)
and CrowdFlower (CFL)
7
10. Task Modelling
Digital Enterprise Research Institute
www.deri.ie
Appropriate models are needed to compare and contrast
micro tasks.
Capability Requirements approach
Capability is defined as the ability of humans to do things in
terms of both the capacity and the opportunity.
Four types of capabilities
–
–
–
–
Knowledge,
Skill,
Ability,
Other characteristics (e.g. motivation, price, etc)
10
11. Capability Requirements
Digital Enterprise Research Institute
www.deri.ie
Taxonomies have be used to study human task
performance, e.g.
Bloom’s taxonomy of classification of learning objectives
Fleishman’s taxonomy of human abilities
O*NET-SOC taxonomy of occupational classification
We are interested in taxonomy that
Describes tasks in terms of human capabilities
Helps in comparing tasks in terms of differences and similarities
of capabilities
11
12. Capabilities Taxonomy
Digital Enterprise Research Institute
www.deri.ie
Based on Fleishman’s abilities taxonomy
Selected abilities relevant to micro tasks
Comprehension (C): The ability to understand the meaning or importance of
something
Bilingualism (B): The ability to speak and understand two languages
Writing (W): The ability or capacity to write text in a given language
Comparison (M): The ability or capacity to compare things based on some
criteria
Judgment (J): The act or process of judging; the formation of an opinion after
consideration
Perception (P): The ability or capacity to perceive items visually or phonetically
Identification (I): The process of recognizing something
Reasoning (R): The ability to draw conclusions from
facts, evidence, relationships, etc.
12
14. Capability Tracing
Digital Enterprise Research Institute
www.deri.ie
How to model worker’s capabilities?
Capability tracing
Inspired by Knowledge Tracing*
Estimates probability of a worker knowing a capability given
worker’s responses to test tasks
Worker Profile constrains
Set of binary variables representing capabilities
Probability estimates of each variable being in a state
* A. T. Corbett and J. R. Anderson, “Knowledge tracing: Modeling the acquisition of procedural knowledge,” User Modeling and User-Adapted
Interaction, vol. 4, no. 4, pp. 253–278, 1994.
14
15. Capability Tracing
Digital Enterprise Research Institute
www.deri.ie
Probabilistic network of a capability and four parameters of
capability tracing model
States of
Capability
Variable
Not
Learned
p(T): Probability of
transition between states
p(T)
Learned
p(L)
p(G)
Values of
Response
Variable
p(L): Probability of a
worker learning to
employ the capability
p(S)
p(G): Probability of
guess
Correct
Incorrect
15
p(S): Probability of slip
16. Experiment
Digital Enterprise Research Institute
www.deri.ie
Objective
Solicit capability requirements of tasks from crowds
Evaluation of capability tracing for performance prediction
Three types of micro tasks with manually created ground
truth data
Image comparison
Fact verification
Information Extraction
37 crowd workers including
University students
Workers from Shorttask.com
16
18. Capability Requirements of Tasks
Digital Enterprise Research Institute
www.deri.ie
Objective 1: Solicit capability requirements of tasks
from crowds
(a) fact verification
(b) image comparison
18
(c) information extraction
19. Crowd Performance
Digital Enterprise Research Institute
www.deri.ie
How the crowd performed on each type of task?
Fact Verification task
• 37 workers
• Best workers perform with
both precision and recall
above 0.8
• More variation in recall means
some workers were could not
spot the incorrect facts
• Ideally tasks should be
assigned to workers that lie in
the top-right quadrant of the
plot
19
21. Performance Prediction
Digital Enterprise Research Institute
www.deri.ie
Objective 2: Evaluation of capability tracing for
performance prediction
Two phases
Build model with observation tasks
Predict performance on new tasks
AC: Consider previous
Accuracy as prediction
of future performance
CT: Capability Tracing
21
22. Summary
Digital Enterprise Research Institute
Capabilities taxonomy is first steps towards modelling of
micro tasks based on human factors
Capability tracing is effective in predicting future
performance
www.deri.ie
Even across tasks if there are similar capabilities
Predicted performance can be used to make right task
routing decisions
Future Work
Evaluate on more types of tasks
Evaluate capabilities such as domain knowledge and skills
Define standard tests for measuring worker capabilities
22
23. Further Reading
Digital Enterprise Research Institute
www.deri.ie
9th IEEE International Conference on Collaborative Computing:
Networking, Applications and Worksharing
Austin, Texas, United States
October 20–23, 2013
U. Ul Hassan and E. Curry, “A Capability Requirements Approach for Predicting
Worker Performance in Crowdsourcing,” in 9th IEEE International Conference on
Collaborative Computing: Networking, Applications and Worksharing, 2013.
http://deri.ie/users/umair-ul-hassan
23
24. Capability Tracing
Digital Enterprise Research Institute
www.deri.ie
Conditional probability of worker learning to employ
capability p(Ln|On) is calculated
When evidence On is positive
When evidence On is negative
24
25. Capability Tracing
Digital Enterprise Research Institute
www.deri.ie
Probability of worker learning to employ capability
Performance of worker on next task
25
Notas do Editor
AMT is amazonmechincalturk
AMT is amazonmechincalturk
AMT is amazonmechincalturk
AMT is amazon mechanical turk
Probability estimates are generated while learning the model through capability tracing
As can be seen, more than majority of workers believed that identification capability is essential for all three types of tasks. Majority of workers also agreed that the judgment and comprehension capabilities are important for the Fact Verification task. In the case of the Image Comparison task most workers specified that comparison and perception are important as well. There is general consensus between workers that comprehension, judgment and reasoning capabilities are also useful for Information Extraction tasks. We selected top-3 capabilities for each type of task for building capability tracing models.
Interestingly, no worker achieved the highest recall for the information extraction task, which highlights the difference between workers and ground truth in terms of the entities extracted from the Wikipedia articles. Nevertheless, these distributions emphasize that in order to achieve high accuracy tasks should be assigned to workers that lie in the top-right quadrant of the plots.
Results show that the capability tracing approach is comparable to the baseline approach in general and achieves better accuracy of prediction between similar tasks. The Fact Verification and Information Extraction tasks have similar capabilities requirements, therefore capability tracingcan better predict the performance of workers between them. The drop in prediction quality of capability tracing for Image Comparison task can be attributed to the little variation is the performance of workers on this task.