Jim is the the creator of the Turkee gem, a Rails form library that allows a developer to easily integrate with Mechanical Turk using the standard Rails form helpers. The gem also makes it easy to retrieve the data submitted to Mechanical Turk and map those values back to your existing models. Mark is the creator of the RTurk gem (upon which Turkee is built). Turkee - Blurring the lines between Rails and Mechanical Turk. https://github.com/aantix/turkee RTurk - A simple wrapper and library for Amazon's Mechanical Turk. https://github.com/mdp/rturk

1. Who We Are
Jim Jones ( jim.jones1@gmail.com )
Creator of the Turkee gem, a Rails form library that allows a
developer to easily integrate with Mechanical Turk using the
standard Rails form helpers. The gem also makes it easy to
retrieve the data submitted to Mechanical Turk and map those
values back to your existing models.
Mark Percival ( m@mdp.im )
Creator of the RTurk gem, a simple wrapper and library for
Amazon’s Mechanical Turk API.

2. What is Mechanical Turk
The Amazon Mechanical Turk (MTurk) is one of the suites of
Amazon Web Services, a crowd-sourcing Internet marketplace that
enables computer programmers (known as Requesters) to co-
ordinate the use of human intelligence to perform tasks which
computers are unable to do. The people that perform these tasks
are informally referred to as “Turkers”.

3. Mechanical Turk Demographics
http://waxy.org/2008/11/the_faces_of_mechanical_turk/

6. More Demographics
http://behind-the-enemy-lines.blogspot.com/2010/03/new-
demographics-of-mechanical-turk.html
United States: 46.80%
India: 34.00%
Miscellaneous: 19.20%

8. The Mechanical Turk Interface
Worker Interface https://www.mturk.com/mturk/findhits?
match=false
Requester Interface https://requester.mturk.com/bulk/batches

9. Common Uses
Tagging of images
Collection of survey data
Sentiment analysis

10. Creative Uses of Mechanical Turk
10,000 Sheep All drawn by Turkers
http://www.thesheepmarket.com/
Dr. Suessify the News
http://groups.csail.mit.edu/uid/deneme/?p=671
Russia to launch 520-day mock mission to Mars ! Oh, The
Places Russia Will Go!
Fisherman’s wife breaks the silence ! One Fish Two Fish Sick
Fish Who Fish
BP tries again to cap well; protests set to start ! BP tries to
cap with a hat while protesters start to raise hell about the well
“Take a Bow” http://nowtakeabow.com/
This isn’t created with Mechanical Turk, but you can envision the
possibilities of utilizing Mechanical Turk workers for the creation of
a piecemeal video such as this.

11. Iterative Uses for Mechanical Turk
Exploring Iterative and Parallel Human Computation
Processes Greg Little1, Lydia B. Chilton2, Max Goldman1, Robert
C. Miller1
We are interested in human computation processes which
coordinate small contributions from many humans to achieve
larger goals. For example, an algorithm might coordinate many
workers to write a description for an image.

12. Ruby Integration
RTurk gem http://github.com/mdp/rturk
Creating HITs
h = RTurk::Hit.create(:title => hit_title) do |hit|
hit.assignments = num_assignments
hit.description = hit_description
hit.reward = reward
hit.lifetime = duration.days.seconds.to_i
hit.question(f_url, :frame_height => HIT_FRAMEHEIGHT)
hit.qualifications.add :approval_rate, { :gt => 80 } end
Reviewing HITs
hits = RTurk::Hit.all_reviewable
puts “#{hits.size} reviewable hits. n”
unless hits.empty?
puts “Reviewing all assignments”
hits.each do |hit|
hit.assignments.each do |assignment|
puts assignment.answers[‘tags’]
assignment.approve! if assignment.status == ‘Submitted’
end
end
end

13. Rails Integration Turkee Gem
Turkee gem http://github.com/aantix/turkee
The Turkee gem builds on top of RTurk providing a simple form
helper for posting/representing model data on Mechanical Turk’s
external servers.
The create_hit method takes a model,
creates a Mechanical Turk hit pointing it at
the new resource URL for the model
passed in. e.g. new_iteration_url
task = Turkee::TurkeeTask.create_hit(host, turkee_task[:hit_title], turkee_task[:
turkee_task[:hit_num_assignments], turkee_ta
The form helper directs the form action
towards the external Turk url and expands
the fields in the given block in the same
manner that Rails’s form_for helper does.
<form accept-charset="UTF-8" action="https://workersandbox.mturk.com/mturk/extern
<input type="hidden" id="assignmentId" name="assignmentId" value="2G2UC26DG67
<input type="hidden" id="hitId" name="hitId" value="29E6NG1FS3NYNMBGNC6ZVZL21
<input type="hidden" id="workerId" name="workerId" value="A3DWBLF5GT7ACT"/>
<input type="hidden" id="turkSubmitTo" name="turkSubmitTo" value="https%3A%2F
<input id="iteration_turkee_task_id" name="survey[turkee_task_id]" type="hidd
<p><textarea cols="40" id="survey_value" name="survey[value]" rows="20"></tex
<p><input name="commit" type="submit" value="Create" /></p>
</form>

14. Retrieving Data from Mechanical
Turk
The process_hits method retrieves the user posted data from
Mechanical Turk, reconstructs the data into a param hash (like a
Controller), determines the model for which the data represents,
and creates an entry for that model in the database.
Turkee::TurkeeTask.process_hits(@turkee_task)
rake turkee:get_all_results

15. Turkee Iterator (updated for Rails
3.1)
Turkee Iterator is a demo built on top of Turkee to sample some of
the capabilities of Mechanical Turk. Allows for flexible HIT requests
from Mechanical Turk.
https://github.com/aantix/turkee_iterator_rails31

16. Christmas Demos

17. Gift For Wife
Directions : Should be under $25 Should include a link to the
item on Amazon Should look like I really care.
Either vote for your favorite submission OR give me your own gift
idea (and Amazon link) for a chance to win a bonus 50 cents.
Thanks for the help!
http://turkee31.herokuapp.com/iterations?id=9

18. Difficult Dilemma
Directions: I’m having difficulty telling my kids that Santa doesn’t
really exist. On top of that, they’e been terrible this past year.
Please write a short letter to my kids from Santa explaining why he
can no longer give them gifts anymore(any reason will do). I will
transcribe the best one it and leave it in their stockings on
Christmas morning.
Merry Christmas!
http://turkee31.herokuapp.com/iterations?id=11

19. Let’s Do a Realtime Example, Say
Hello
http://turkee31.herokuapp.com/turkee/turkee_tasks/new
Please say “Hello” to my colleagues and tell us where you’re
from.
I am demo’ing Mechanical Turk to the Ruby SF meetup group.
Please tell send us a creative “hello” and where you’re from.
A one or two line introduction would be nice. Tell us how the
weather is in your city. Tell us about your family. Or just simply
say hello.
0.05
100
5
It’s mostly sunny at 65 degrees. Hello from San Mateo, CA!

20. More Content Creation

21. Mechanical Turk is an untapped
market in terms of getting content
created for your application. Start
thinking outside the context of
scientific surveys and fact finding
and start thinking of this resource
as a pool of human workers with
emotions, opinions, and stories to
tell.
E.g.
Iterative story telling where each Turker adds a piece to a
never ending story.
Have the Turker give their opinion on a topic or product and
then have other Turkers respond to that opinion.
Have the Turker describe some emotionally happy or painful
point in their life. Ask emotionally charged question to have the
Turker reveal the essence of who they are.

22. Example: Obesity Testimonials
(Design of Questions)
Upload a picture of yourself of when you
first considered yourself “fat”.
When the above picture was taken, were you aware that
you had been gaining weight? Describe the feeling you had
when you first glanced at the picture and noticed your
weight gain. Did anyone else notice or make snide
comments? How did you feel?
“As a child I was pretty average but I was lead to believe that I
was fat. I would request simple items, things that are status
symbols to the average kid, like jeans. These small requests would
be shot down, my mother telling me that I was too big to fit into a
pair of jeans. I remember her telling me that when I was eight. It
started a cycle in my life where I felt that I was too large, too fat,
for the things other children took for granted.”

23. Other Concepts

24. Micro Bonuses
The concept where the user is awarded a varying amount
depending upon how much time they take to answer a question or
how accurate it (relative to the other user answers)
E.g. Maybe there’s a trivia game where you record how long it
takes each user to answer a question and the fastest response for
the day gets 10 cents instead of 5.

25. Gambling
Allow users to gamble their potential pay.
E.g. Sentiment analysis.
Show the user all of the recent headlines for a given stock, stock
charts, etc.
You state that they can either take their payout now, OR, if they
think that the stock will close out higher than it is now at the end
of the day, they can “gamble”, and will be awarded the fee PLUS a
bonus at the end of the day. If it closes down from the point at
where it is, they receive nothing.
Does a future, larger payout increase the quality of data analysis?

26. Interesting Research Findings

27. Financial Incentives and the
Performance of Crowds
Winter Mason, Yahoo! Research
Figure 2 reveals two main findings: first, that across all difficulty
levels participants chose to complete more tasks on average when
the pay was higher (F(3,607) = 15.73, p < 0.001); and second,
that across all payment levels, the number of completed tasks
decreased with increasing difficulty.
As Figure 3 indicates, however, increasing compensation did
not improve accuracy, which we measured in two ways…

28. Toward Automatic Task Design: A
Progress Report
Eric Huang, School of Engineering and Applied Sciences
Harvard University

29. Task Search in a Human
Computation Market
Lydia B. Chilton, University of Washington **
We found strong evidence that Turkers sort by the most number of
HITs available (so they can find one task, and then do 100
instances of them in a row) and the most recently posted HITS (so
they get the latest and greatest HITs).