Bad Data, Bad Decisions: The Art of Asking Better Questions, Stephanie Beadell's presentation from MozCon 2014

5. Ask Questions on a
5-point or 7-point Scale

6. Are you happy with your job?
Sample Survey Question

7. No
21%
Yes
79%
People Happy at Their Jobs

8. I am happy with my job.
Sample Survey Question

9. People Happy at Their Jobs
Strongly Disagree Disagree Neutral Agree Strongly Agree
Average = 3.05

10. Break Down Big Concepts
into Tangible Pieces

11. Are you satisfied?
Sample Survey Question

14. Sample Survey Question
Please Rate Your Satisfaction with Each Feature
Very
unsatisfied
Somewhat
Unsatisfied
Neural
Somewhat
Satisfied
Very
Satisfied
Feature 1
Feature 2
Feature 3

15. Satisfaction by Feature
Feature 1 Feature 2 Feature 3

16. Compare Answers
Across Questions

17. Sample Survey Question
Please Rate Your Experience with Our Business
Very
unsatisfied
Somewhat
Unsatisfied
Neural
Somewhat
Satisfied
Very
Satisfied
Product
Quality
Price
Account
Manager

18. Business Ratings
Product Quality Price Account Manager

19. Business Ratings by Company Size
Product Quality Price Acount Manager Product Quality Price Acount Manager
Small Business Enterprise

20. Comparing Answers with SurveyMonkey
Analyze + Compare

21. Beware of Bias

22. Priming

23. Bargh, Chen, and Burrows (1996)
Rude aggressively, bold, rude, bother, disturb, intrude, annoyingly, interrupt, audaciously,
brazen, impolitely, infringe, obnoxious, aggravating, bluntly
Polite
honor, considerate, appreciate, patiently, cordially, yield, polite, cautiously,
courteous, graciously, sensitively; discreetly, behaved, unobtrusively
Neutral exercising, flawlessly; occasionally rapidly, gleefully, practiced, optimistically,
successfully, normally, send, watches, encourages. gives, clears, prepares
Sentence Scramble Task

24. Bargh, Chen, and Burrows (1996)
People Who Interrupted After Sentence Task
Polite Neutral Rude

25. Order Bias

26. How important is link building?
Question Order Bias
How much link building do you
plan to do this year?

27. Randomization in SurveyMonkey and Google Forms
Randomize Order

28. Stereotype Threat

29. “sports intelligence”
Stone, Lynch, Sjomeling, Darley (1999)
“natural athletic ability”

30. Inzlicht (2000)
2 Women 2 Men
Female Math Test Performance when Others Present

31. Double-Barreled Questions

32. Do you play games on your
smart phone or tablet?
Double-Barreled Question

33. Leading Questions

34. Do you support helping the poor?
Leading Questions
Do you support welfare?

35. Understand How
and Why People Lie

36. Sensitive Information

37. Use ranges for demographic info.
Sensitive Information

38. Boredom

39. Set expectations & show progress.
Boredom

40. Guessing is Hard

41. Be reasonable about estimations.
Guessing is Hard

42. Cool Things You
Can Do With Data

43. How often do you use OSE?
Sample Survey Question

44. Sample Survey Question 1
How often do you use Open Site Explorer for the following tasks?
Never Rarely Sometimes Often Always
Find Backlinks
Anchor Text
Analysis
Competitor
Analysis

45. Sample Survey Question 2
How would you rate Open Site Explorer for the following tasks?
Never Rarely Sometimes Often Always
Find Backlinks
Anchor Text
Analysis
Competitor
Analysis

46. Sample Survey Question 3
How important are the following Open Site Explorer features?
Never Rarely Sometimes Often Always
Find Backlinks
Anchor Text
Analysis
Competitor
Analysis

47. Feature Prioritization Chart
Frequency
0
7.5
15
22.5
30
Rating
0 4 8 12 16

48. Categorical

49. Do you drink coffee?
Categorical Response
Yes | No

50. Coffee Drinkers
Categorical Response
No
Yes
No Yes

51. Ordinal

52. Ordinal Response
How much coffee do you drink?
0 cups per day
1-3 cups per day
4-5 cups per day
6+ cups per day

53. Ordinal Response
Coffee Drinkers
6+ Cups
4-5 Cups
1-3 Cups
0 Cups
0 Cups 1-3 Cups 4-5 Cups 6+ Cups

54. Interval & Ratio

55. How often do you drink coffee?
Interval & Ratio Response
Never Rarely Sometimes Often Always

56. How many cups of coffee do you
drink each day?
Interval & Ratio Response
______________
(fill in the blank)

57. Source: FIVETHIRTYEIGHT
Mean, Median, Mode

58. Source: Pew Global
Correlation

59. Source: Wired
Cluster Analysis

60. Interval & Ratio Response
Regression
Outcome
Variable 1
Variable 2
Variable 3
(unknown)
relative influence on outcome

61. Interval & Ratio Response
Excel Add-In: Analysis Toolpak
Free Add-In