4. What is Mechanical Turk?
Co-ordinate a crowd (a large group of people), to do micro-work (mostly small
contributions) that solves problems (that software and one user cannot do)
Image: mturk.com
5. Amazon Mechanical Turk (AMT):
• mturk launched November 2, 2005
• Artificial artificial intelligence
• Work : Requesters
Providers / Workforce: Turkers
• Human Intelligence Tasks (HITs): payment often in cents/seconds
• Other crowdsourcing platforms, other types of work e.g. freelancer,
odesk, elance
6. Types of task on AMT
• Micro-tasking: image tagging, duplicate recognition, SEO, data
entry, translation, transcription, content generation, obj. classification
Image: mturk.com
7. Why research on AMT ?
• Little in-depth qualitative studies of Turkers
• Survey of demographics – Ross et. al.
• Turkers Bill of Rights – Irani, Silberman
• Law in crowdsourcing industry – Felstiner
• AMT design – Khanna et. al.
• Concerns:
Are workers perfectly happy with conditions and wages?
• Information deficit/opacity
• Lack of choice and opportunities
• Imbalance / Lack of power of redress
8. How did we conduct research ?
• Virtual ethnography of online forums like mturkforum and Turker Nation
by Dr. Dave Martin, XRCE
• Study involved deep reading of threads, collection of key threads/
posts, recurrent themes, exemplars
• Ethnographic Studies of Turkers in their respective homes and places
of work in India
• Study involved Interviews (skype, phone and in-person), walkthroughs,
observations, data capture through audio / video / screen recordings
9. Why Do Turkers Turk?
• Motivated by earning money
• Learn new things, “earn money while having fun”, but the aim still is
to earn money
• Compare themselves to others
• Set themselves goals and targets
• For some AMT is a compromise: Problems accessing the regular job
market, Need to supplement their income, Some housebound.
10. Process of Turking for a Turker
Look for HIT Accept HIT Work on HIT
Submit HIT
and wait for
approval
If approved,
get paid, if not,
get rejected.
Wait for
feedback
Wait for money
to arrive
Go back for
more work
Worry about blocks, ratings,
qualifications, suspensions,
future work.
11. Turking = flexible working?
Example 1: Is turking flexible?
Mansoor
You cannot find much work on AMT during the day… in the morning I
go to institute for BBA for taught and practical classes.. Class starts
at 11.30 am, I leave home at 10.30am, we have 1 hour theory, 1
hour practical and then we practice for 1 – 2 hrs.. and then I come
back at 4 pm and rest. Then do work on AMT. I do the most work at
night time after 7.30 sometimes till 1, 2 or 3 am at night… I also
have to do house work in the morning […] but if I got more work in
the mornings I am willing to sit and work all day.
12. Turkers Turking for Fun?
Example 2, 3: Turkers Turking for Fun?
Danturker
This attitude would be requesters dream come true. The workers come
here to have fun and play and the lousy pay for work is not an issue.
This attitude helps create low pay for the AMT work force that does
care about fair pay.
Larak56
I agree with most everyone here. While I do find some of the HITS fun
and actually learn an incredible amount by doing HITS, I do it for the
cash.
13. Varying Expectations
Example 4: Varying Expectations
Defectturk
I would rather pull weeds, do laundry, wash the car, clean the toilet
than work for this shit paying requester.
Clunky interface, too many tags/ratings/whatever per hit, works out to
about a buck an hour. They should be banned from mturk, or even
better forced to work on their own shit paying hits.
14. Varying Expectations
Example 5: Varying Expectations
Nagen
I only do tasks that I have done before so that I am familiar with the
instructions. […] But I think one of past few times, something I did
went wrong as I got some warning emails from Amazon and finally
got blocked. I was pretty sure I had done the task correctly and had
seen the instructions before but perhaps I misunderstood something
and now I might get suspended. We keep trying to contact them
(AMT) via emails. I still have around $50 in my account. They are an
American company, they are pukka (genuine/honest/honourable) in
their procedures, I trust they‟ll be fair to me and give me my money
back..
16. Reputation of Turkers
Example 6: Reputation based on qualifications, ratings
Mansoor
Now these HITs are available 24X7 for these guys (who have score of
a 100) and I see them everyday but can‟t do them, which hurts a
little. I was careless when the Requester was testing Turkers with
qualifications, so my score is less. I should have worked hard [...]
but I was in a rush.”
17. How Much Do Turkers Earn?
• Considerable variation but Turking is low wage work, wages off-set
against time taken to find work, amount of concentration required etc.
• Highest earners on Turker Nation ~$15-16k per year
High earners in India ~$8k-$10k per year
Low earnings can be as low as $260 - $520 per year ($5-10 per
week)
• U.S Turkers generally aspire to earning $7-10 per hour.
Indian Turkers aspire to earn according to benchmarks set e.g. Call
centre work ~ Rs 10,000 ~ $160 per month
• Lower paid easy work to increase reputation and ranking e.g. Oscar
Smith
18. How Much Do Turkers Earn?
• Lower paid easy work to increase reputation and ranking e.g. Oscar
Smith
19. Invisible Turkers & their hidden work
• AMT as a labour market:
• Easy for requesters to devalue/ignore workers
• Invisibility accentuated by anonymity, distance, digital interaction
The work to make the Turking work:
• Work of searching, learning, optimising, organising. hidden, no
monetary compensation
• Discussing best practices with fellow-turkers, supporters &
requesters
21. Good and Bad Requesters
Example 7: A Good Requester
Explodeman
All hits I have done for Project Tatooine in the past week have approved
and are fair pay. Good Requester.
22. Good and Bad Requesters
Example 8: A Bad Requester
Iambob
I have done a number of his hits. Then one day I get this email:
“You received a bonus from Top Page Solutions. The value of your bonus
is $0.01 USD yes ONE CENT. The requester included this note:
If you spent half the amount of time looking for a job as you spent
scamming your way through Mechanical Turk HITs maybe you’d be
able to find a real job”
The next day I get a block message from him. WTF?! … My hit total is
under 2000 so I do not see how I am a scammer in his mind.
23. Turker Ethics?
Both U.S. and Indian Turkers take a strong ethical position
• Duty to be honest about requesters and admit mistakes.
• Good behaviour is practically and ethically the best way to help
fellow Turkers and influence the market
• Ripping requesters off is not endorsed
• Turkers would like their voice heard and help from Amazon
24. Turker-Requester Relationships
• U.S. Turkers like anonymity and flexibility, Indian Turkers don‟t mind
anonymity but prize relationship with American requesters
• Turkers want productive relationships with requesters
• They want fair pay for fair work: decent wages, fairness in judging
work, timely payment
• Value courteous and fair communication
• Want to communicate with requesters about work, design of tasks etc.
• Respect works both ways: regular work from good requesters is highly
prized
25. Key Takeaways for Requesters
• Think about Turkers as people, colleagues
• Communicate your thoughts
• Pay Turkers fairly
• Provide feedback (positive / negative) when possible
• Try to be involved / engaged
26. Thank You !
If you use / would like to use AMT to get some work done, I would love
to hear your experiences, questions or anything you would like to
discuss. Please feel free to talk to me.
neha.gupta@nottingham.ac.uk
@Neha_Gi_Ji
