This document discusses dimensions of microwork and how microworkers can help data scientists. It notes that microworkers can assist with data collection, providing human judgments as training data, and evaluating data. However, while independent judgments may increase accuracy, it also prevents collaboration between workers. The document recommends keeping microwork tasks simple by providing clear instructions, being transparent about goals, and managing tradeoffs between task difficulty and value. It also advises watching out for potential systematic biases.

1. Humans, Machines, and the
Dimensions of Microwork
Daniel Tunkelang, LinkedIn
Claire Hunsaker, Samasource
Recruiting Solutions 1

2. What is LinkedIn?
Identity
Rolodex, Resume, Connect, find and be found
Business Card LinkedIn Profile, Address Book, Search
Insights
Newspapers, Be great at what you do
Trade Magazines, Events Homepage, LinkedIn Today, Groups
Everywhere
Work wherever our members work
Desktop Mobile, APIs, Plug-Ins
2

3. 3

4. 4

5. 5

6. 6

7. 7

8. How do Microworkers help Data Scientists?
 Data Collection
 Human Judgments as Training Data
 Evaluation
8

9. Quality Assurance
9

10. Foreign University Research Project
10

11. Stay Objective
vs.
11

12. At What Price Independence?
 Independent judgments enable statistical reasoning.
– Can increase accuracy by requiring agreement of independent
workers on the same task.
12

13. At What Price Independence?
 Independent judgments enable statistical reasoning.
– Can increase accuracy by requiring agreement of independent
workers on the same task.
 But independent workers can’t help each other out.
– No benefit from collaboration = less accurate workers.
– Number of workers becomes bottleneck, and workers may be
incented to create fake alter egos.
13

14. Add slide of workers helping each other out at delivery center
14

15. At What Price Independence?
 Independent judgments enable statistical reasoning.
– Can increase accuracy by requiring agreement of independent
workers on the same task.
 But independent workers can’t help each other out.
– No benefit from collaboration = less accurate workers.
– Number of workers becomes bottleneck, and workers may be
incented to create fake alter egos.
 Market for lemons (Akerlof, 1970)
15

16. 16

17. Keep It Simple
 Avoid unnecessary difficulty.
– Provide clear instructions with examples.
– Be transparent about what you’re trying to achieve.
– Check an early sample of the work closely.
– Set expectations on quality and accuracy and manage to those.
17

18. Keep It Simple
 Avoid unnecessary difficulty.
– Provide clear instructions with examples
– Be transparent about what you’re trying to achieve
– Check an early sample of the work closely
– Set expectations on quality and accuracy and manage to those
 Trade-offs between task value and difficulty.
– Easier to select from options than answer open-ended questions.
– Even easier if there are only two options.
– But open-ended questions leverage more intelligence.
18

19. Keep It Simple
 Avoid unnecessary difficulty.
– Provide clear instructions with examples
– Be transparent about what you’re trying to achieve
– Check an early sample of the work closely
– Set expectations on quality and accuracy and manage to those
 Trade-offs between task value and difficulty.
– Easier to select from options than answer open-ended questions.
– Even easier if there are only two options.
– But open-ended questions leverage more intelligence.
 Watch out for systematic bias.
– Even independent judges may make the same mistakes.
– Especially if they use the same tools.
19

20. Take Aways
 Independent judgments are nice for some tasks, but not
always worth the cost.
 Keep crowdsourcing tasks as simple as possible.
 Manage the trade-off between task value and difficulty.
 Watch out for systematic bias.
20

21. Thank You!
Questions?
Contact:
Daniel: dtunkelang@linkedin.com
Claire: sales@samasource.org
We’re (both) hiring (a lot)!
21