2. "They've done studies, you
know. 60% of the time, it
works every time. ”
- Brian Fantana
03/18/12 Will Evans & Yana Kuchirko 2
3. Surveys Aren’t Objective
They are created by people who are biased and analyze their
results based on their own (mis)perceptions.
03/18/12 Will Evans & Yana Kuchirko 3
4. Epistemological Dead End?
No. Awareness that the researcher plays an integral
role in the process of measuring any given
phenomena by deciding how to measure is key.
There are ways to minimize researcher bias by
creating better questions.
03/18/12 Will Evans & Yana Kuchirko 4
5. Methods are not “just methods”
How you measure what you are studying shapes what you find.
03/18/12 Will Evans & Yana Kuchirko 5
6. Let’s say you are studying…
How often teenagers use your website
03/18/12 Will Evans & Yana Kuchirko 6
7. You might ask teenagers…
“How often do you my website?”
c.Very rarely
d.Rarely
e.Occasionally
f.Frequently
g.Very frequently
Response options source: http://www.dataguru.org
03/18/12 Will Evans & Yana Kuchirko 7
8. You might ask teenagers…
“How often do you my website?”
c.Very rarely
d.Rarely And what’s wrong
e.Occasionally with these????
f.Frequently
g.Very frequently
Response options source: http://www.dataguru.org
03/18/12 Will Evans & Yana Kuchirko 8
9. You might ask teenagers…
“How often do you my website?” ! !!
VI TY
EC TI
c.Very rarely
BJ
d.Rarely SU And what’s wrong
e.Occasionally with these????
f.Frequently
g.Very frequently
Response options source: http://www.dataguru.org
03/18/12 Will Evans & Yana Kuchirko 9
10. “Hmmm, for me “frequently”
really means….”
A few times a week A few times a day
So what are we really measuring? NOISE.
03/18/12 Will Evans & Yana Kuchirko 10
12. What is “noise”?
We WANT this!
We DON’T WANT this!
03/18/12 Will Evans & Yana Kuchirko 12
13. An imaginary scenario
Let’s really stretch our thinking a bit here to
provide a more concrete understanding of
“noise” in your data.
Imagine you have AT&T phone service and
you’re trying to make a call. The signal isn’t
clearly going through and you end up hearing
everything but the other person’s voice.
Frustrating, right? Thank God this is only a
pretend scenario.
03/18/12 Will Evans & Yana Kuchirko 13
14. How is AT&T related to “Noise”?
When you develop bad questions, you don’t
“hear” the message of your data clearly.
03/18/12 Will Evans & Yana Kuchirko 14
15. So…why is “noise” bad???
• Badly structured and poorly worded
questions that obfuscate meaning for
participants provide bad data.
• Bad data
– Doesn’t answer your research question
– Makes it difficult to interpret results
– Is pretty much useless.
03/18/12 Will Evans & Yana Kuchirko 15
16. More bad examples
of frequently used response options
a. Completely satisfied a. Totally like
b. Very satisfied b. Very much like
c. Fairly well satisfied c. Moderately like
d. Somewhat dissatisfied d. Somewhat like
e. Very dissatisfied e. Not like
What’s the difference between What’s the difference between
“fairly well” and “somewhat”? “moderately like” and “somewhat like”?
03/18/12 Will Evans & Yana Kuchirko 16
17. More bad examples
of frequently used response options
a. Completely satisfied a. Totally like
b. Very satisfied b. Very much like
c. Fairly well satisfied c. Moderately like
d. Somewhat dissatisfied d. Somewhat like
e. Very dissatisfied e. Not like
What’s the difference between What’s the difference between
“fairly well” and “somewhat”? “moderately like” and “somewhat like”?
Semantics
03/18/12 Will Evans & Yana Kuchirko 17
18. Words like….
• Sometimes
• Often
• Moderately
• Very
• Not very much
Are subjective
= they mean different things for different people
03/18/12 Will Evans & Yana Kuchirko 18
19. A better alternative?
Actually asking people how often they do certain
activities.
03/18/12 Will Evans & Yana Kuchirko 19
20. In order to gather objective data…
…your questions must mean the same thing
for everyone.
How often do you use my website?
d.Never
e.A few times a year
f.Once a month
g.2-3 times a month
h.Once a week
i.A few times a week
j.Every day
“Never” means never for everyone!
03/18/12 Will Evans & Yana Kuchirko 20
21. Importance of “Anchors”
Responses options serve as “anchors” for each question,
determining the “location” of each responses as qualitatively
distinct from the other.
03/18/12 Will Evans & Yana Kuchirko 21
22. Importance of “Anchors”
Responses options serve as “anchors” for each
question, determining the “location” of each
responses as qualitatively distinct from the other.
Strongly Strongly
disagree Disagree Neutral Agree Agree
03/18/12 Will Evans & Yana Kuchirko 22
23. (Ideal) Response Symmetry
• Good questions aim to have symmetrical
quantitative/qualitative distance between
anchors
a. Never
b. A few times a year
The distance between “never” and
c. Once a month
“a few times a year” is proportionate to
d. 2-3 times a month “every day” and “a few times a week”.
e. Once a week
f. A few times a week
g. Every day
03/18/12 Will Evans & Yana Kuchirko 23
24. Adherence to Logic & Linearity
People are accustomed to paradigms that are
intuitive, and often “linear”.
Anchors should not be the exception.
03/18/12 Will Evans & Yana Kuchirko 24
25. Intuitive Anchor Directions
Never Always
Disagree Agree
Very poor Very good
Not very Very
important Important
03/18/12 Will Evans & Yana Kuchirko 25
26. Value of Clarity
If questions are clear and concise, participants would
spend less time analyzing the questions themselves and
more time on answering them. Mystic Arts, LLC
03/18/12 Will Evans & Yana Kuchirko 26
27. So do good questions guarantee valid
results?
• No. But good questions offer more
assurance that you are listening to signal
and not the noise.
• But answering your research question can be
done in other ways
• Behavioral Observations
• Open ended qualitative questions
• Many more…
03/18/12 Will Evans & Yana Kuchirko 27
28. The End.
So was this presentation simply
oh-my-fucking-god-awesome?
a. Hell yeah!
b. Definitely!
c. Totally rocked!
03/18/12 Will Evans & Yana Kuchirko 28
