1. Jeff Rzeszotarski
[rez-oh-tar’-ski]
4th Year Ph.D. Student
HCII, CMU
www.jeffrz.com
jeffrz@cs.cmu.edu
Instrumenting the Crowd

2. James Cridland @ Flickr

3. James Cridland @ Flickr

4. James Cridland @ Flickr
Amazon Mechanical Turk
oDesk
CrowdFlower

8. Bernstein et al. 2010

9. Task Design
Gold Standard
Worker Agreement

10. Task Design
Cheating Resistant Design
Kittur, Chi, Suh, 2008
Von Ahn & Dabbish, 2004

11. Gold Standards
Gold Standard Worker 1 Worker 2
A A A
D B D
B C
A A
2 4 2
2 8
Callison-Burch, 2009
Downs et al. 2010

12. Worker Agreement
Callison-Burch, 2009 Ipeirotis et al. 2010
Dekel & Shamir, 2009 Snow et al. 2008
WorkerID Video Transcript
AF34DB9 Coloring comics in Photoshop is easy. First you…
QR40Q2 <blank>
TK421S9S Coloring comcs in photoshop is easy first u…
WOPR21 QOJEORJOFJDSKGNDKSNGSDSJGKDSJGLKSDJGL
APC3C55 Coloring comics in Photoshop is easy. First you…
IDKFAD1 Coloring comics in photoshop is easy. First you…

13. Disadvantages
Task Design
- Task-dependent
- Time consuming iteration
Gold Standard
- Not always available
- Not always applicable
Worker Agreement
- Limits response range
- Subject to majority effects
- Not always applicable

15. Worker A Worker B

17. Worker A

18. Worker A

19. Worker A

20. Worker A

21. Worker A

23. Worker B

24. Worker B

25. Worker B

26. Worker B

27. Worker B

28. Worker B

29. Worker B

31. Logging
ABC
SCROLL
MOUSE
MOVE
MOUSE
MOVE CLICK CLICKDELAY DELAYTYPING
120px 300px,
20px
Field1 1.1s ‘Hello’ 0.9s 100px,
220px
Submit
Example Log: Typing and submitting “Hello”

32. Features

33. Experimental MTurk Tasks
Noun Classification
“Identify which words are nouns.”
Image Tagging
“Generate 3-5 tags for each of 4 images.”
Reading Comprehension
“Read and then answer the following questions.”

34. Image Tagging
Provide at least 3 tags for
the following image…
ocean mountains
pretty
x4

35. Image Tagging

36. Image Tagging
52 participants, 114 submissions
Human Labeled Cheating
17 cheats, 97 regular
Human Labeled Quality
AVG 3.5 / 5, SD 1.13

37. Model Features
# Unique Letters
Total Time Elap.
Image Tagging
17 cheats, 97 regular
decision tree, 10-fold CV
93.0% accuracy

38. Image Tagging
Regressing on ‘quality’
SMOreg, 10-fold CV
r=0.5874 (generally ±0.5 on Likert)
Model Features
# Unique Letters
Total Time Elapsed
# of Clicks
Fields Accessed

39. Generalizing Models
Training
Reading
Comprehension
63 points
Testing
Noun
Identification
185 Points
Model predicts for a similar task
r=0.4948

40. Other Findings
Clustering workers by
behavioral patterns
De-anonymizing
worker submissions

41. Hanging Questions
Complex & creative work?
Human interpretability
Organizer cognitive models

42. CrowdScape

45. ABC
SCROLL
MOUSE
MOVE
MOUSE
MOVE CLICK CLICKDELAY DELAYTYPING
120px 300px,
20px
Field1 1.1s ‘Hello’ 0.9s 100px,
220px
Submit

48. Interac(vely
visualizing
user
behavior
and
output

50. Thanks, d3 toolkit

62. Series and series of works starting with “Godzilla"
monster movie Godzilla was released in special
effects is the name of the fictional monster of
their work.
The film named “Godzilla" is a special effects
monster movie, and Godzilla is the monster.
ゴジラはに公開した特撮怪獣映画『ゴジラ』に始まる一連のシリーズ
作品及び、それらの作品に登場する架空の怪獣の名称である。

66. I like to visit Alappuzha in Kerala, the God's own country.
This place is blessed with heavenly lakes. The view of lake
shores from the boats is very beautiful. We can enjoy cool
wind standing in the boats. We can see a complete
picture of a beautiful village by a trip through the lakes of
Alappuzha.
My favorite place in the world to be is right at home!
Ideally my daughter and husband and I just spending
time together watching television, sharing a meal, or
discussing our day. I know it might sound boring to some
but I find joy and security in the familiar surroundings. Sure,
there are other places I enjoy but given the choice my
home will always be the place I am the happiest.

70. Have you ever wondered what causes swing sets to move
with no one on them? Or why bridges twist and tear
themselves apart? The answer is aeroelastic tension. But the
science behind it is the most interesting part. Watch this and
you'll find out how it works in less than two minutes.
collapse bridge physics tension aeroelastic
Tacoma Narrows Bridge Collapse

71. Limitations
Logging is not perfect
Information overload
Worker traces
Parallel coordinates
Interpreting behavior
Feature set, ML algorithms