Crowdsourcing represents a significant source of data which needs to be analyzed and interpreted. These tasks influence the quality of the output as well as the efficiency of the process. Visualization proved to be an effective way of dealing with large amount of data. In this paper we propose a visualization analytic model in the context of the CrowdTruth framework and CrowdTruth metrics for optimizing the crowdsourcing process and improving its data quality. The requirements for the dynamic, scalable and interactive visualizations were extracted through literature and interviews with users of the framework.
3. Current practices: based on the consensus of workers
CrowdTruth metrics : considers disagreement informative
4. Select from the list the objects depicted in the image:
Unclear image (content unit)
Worker 1 Worker 2 Worker 3
Balloon
Flower
Human
Car
Ghost
Person
Balloon
Flower
Human
Car
Ghost
Person
Balloon
Flower
Human
Car
Ghost
Person
Can you identify the low quality worker(s)?
5. Select from the list the objects depicted in the image:
separable
Worker 1
Balloon
Flower
Human
Not clearly Car
answers
Ghost
Person
Worker 2
Balloon
Flower
Human
Car
Ghost
Person
Worker 3
Balloon
Flower
Human
Car
Ghost
Person
Can you identify the low quality worker(s)?
6. Select from the list the objects depicted in the image:
Worker
2 Balloon
Worker
1 Balloon
Balloon
Flower
workers
Human
Low quality Car
Ghost
Person
Flower
Human
Car
Ghost
Person
Worker
3
Flower
Human
Car
Ghost
Person
Can you identify the low quality workers?
7. How good is the
unit for the
specific task?
How well the
worker
understood the
task?
Are the
annotation
options clear and
separable?
Unit
Worker Annotation
8. JOB 1 JOB 2
Unit
Worker Annotation
Annotation
JOB N
Unit
Unit
Worker
Worker
Annotation
9. Visualization approach for quality assessment of
crowdsourced data :
a) at aggregate level
b) at a specific level
c) and in the context of their interdependencies
10.
11. Extracted through interviews
Visualization of properties, statistics and metrics of:
single job/unit/worker
collection of jobs/unit/workers
Functional requirements:
Filtering, sorting
Support for detection of outliers
Visualization of connected workers, content units and jobs
Support of comparative analysis
Support for navigation between connected elements, etc.
18. Create user profiles
Decouple the visualization component and
provide it as a separate plugin
Add the time dimension Time to the visualizations