- The document describes research analyzing survey response patterns of 370 respondents who skipped a question. Through mixture modeling, they identified 3 distinct "user types": Quitters, Returners, and Completers. - A visualization tool was created mapping response patterns and identifying troublesome questions and areas. This allowed easy identification of user types and usability issues. - Key conclusions were that the tool provides an effective way to examine survey effectiveness, pinpoint problem questions and respondents, and suggest improvements to the design and administration process.

1. Why Do Respondents Skip Questions
in Surveys: A Visually Integrative
Representation of User Types
Ricardo Carvalho
Joseph Luchman
Michael Paraloglou
Vanessa Patterson
Ron Vega
1

2. Outline
• Background
• Our Research
• Our Findings
• Our Conclusions & Recommendations
• Future Research
2

3. Background
3

4. Background
• DoD Youth Poll December 2011 survey
– Mailed to 50,000 youth ages 16 to 24 with no prior or current military
experience through stratified, probability-based sampling
– Address-based sample drawn from list frame estimated to cover 92% of target
population
– Standard mailing methodology (Dillman 2007)
– Scantron survey; double-entered and verified
– Up-front and contingent monetary incentives upon completion
– Response Rate 3: 17%, Contact Rate 2: 92%; n= 7,210
• Note: although our study was specific to completing a paper survey,
much of the theory (and more importantly, the tool) can be applied
to other survey modes and experiences
4

5. Background
5

6. Background
• The Issue # Refusals to Q29 and Q30
– Amount of refusals per item was very small 1-18 Refusals (one to many but
not all items)
(<1.5%) up to Q29 19 Refusals (all items)
– But at Q30 and thereafter, it increases to 6% 6% 5%
about 5-6% 5%
5%
4%
• Behavior of certain users
changes in consistent manner 3%
2%
• Can we understand the user’s 1% 1%
experience and behavior? 0%
Q29 Q30
6

7. Our Research
7

8. Our Research
• We noticed 370 respondents whose behavior
seemed fundamentally different
– Are these different “user types”?
– Or was there a usability issue with the survey (“troublesome areas”)?
– How can we identify the “user type” or a “troublesome area”?
Does this kind of
information tell us what to
change in the experience?
No…
8

9. Our Research
• The behavior we noticed is characteristic of “satisficing” (Simon 1957)
– Economic phenomenon: “satisfying” and “sacrificing”
– We exercise an acceptable level of effort to achieve a satisfactory but less than optimal
outcome
– Example: driving around for the cheapest gas price
– There is substantial literature written on this topic and how it applies to surveys
(Krosnick 1991)
– Behavior points to this phenomenon, but very difficult to be certain
• The focus of our presentation is NOT on exploring this behavior but on
understanding and visualizing different user types
– How does the survey experience impact users?
– Are there usability issues we can notice or isolate?
– Can we build a tool to help improve the overall user experience and hence obtain more
complete and accurate information?
9

10. Our Findings
10

11. Our Findings
• What we did:
– Examined only the 370 respondents who refused all of Q30
– Determined if unique user types existed through mixture modeling
– Wrote code to visually map these user types and their refusals for the
remainder of the survey
– Marked page breaks and “grid” questions in this map
• What we found:
– 3 distinct user types
• The Quitters
• The Returners
• The Completers
– Map allows us to easily identify these user types
– Map also allows us to easily identify “troublesome areas”
11

12. Our Findings
Black line = pages The 370 Respondents
Orange line = grid question
Colored squares = question was ANSWERED

13. Our Findings: The Quitters
Black line = pages
Orange line = grid question Last page of survey
Colored squares = question was ANSWERED
Demographics
• Engagement clearly breaks off and
users flip to back of survey
• Paper survey immediately presents
users with workload
• Completes Demographic questions
(“essential” and easy items) for token
of appreciation

14. Our Findings: The Returners
Black line = pages
Orange line = grid question
Colored squares = question was ANSWERED
Grid questions
• Engagement terminates after long
second grid question (Q30) but returns
• Selectively respond to “taskful”
questions (i.e., grid questions) to
minimize effort

15. Our Findings: The Completers
Black line = pages
Orange line = grid question
Colored squares = question was ANSWERED
• Most conscientious and engaged group
• Engagement terminates for only Q30
• Occasional refusals
Only a few questions are left
unanswered by this user type

16. Our Findings: Profiling the User Types
Other
Asian
Hispanic Asians show more Completers
Black
Whites show almost all
of the Returners
White
16

17. Our Findings: Profiling the User Types
• Mixture model: seeks homogenous distributions within data based on number of
questions refused after Q30
• Predictive model based on census block sociodemographic data linked to
respondent scores
– Exploratory predictive model (i.e., empirically driven)
Hispanic, Native Hawaiian or Pacific Islander Civilian Population in Labor Force Employed; Age 16
Overall Population Median Household Income
Population and up
Population with less than 9th Grade Education; Age Population in Labor Force Unemployed; Age 16 and
Population aged 16-17 Hispanic, Other Population
25 and up up
Population with some High School Education; Age
Population aged 18-20 Hispanic Population Population not in Labor Force; Age 16 and up
25 and up
Population with High School Education; Age 25 and Percent of Population in Labor Force Unemployed;
Population aged 21-24 Population in Nursing Home
up Age 16 and up
Population in other Institutionalized Group Population with some College Education; Age 25 Population employed in Private, for Profit; Age 16 and
Median Age
Quarters and up up
Population Employed in Private, not-for Profit; Age 16
Non-Hispanic, White Population Population in College Dorms Population with Associates Degree; Age 25 and up
and up
Population Employed in Local Government; Age 16
Non-Hispanic, Black Population Population in Military Barracks Population with Bachelors Degree; Age 25 and up
and up
Population in Non-Institutionalized Group Population Employed in State Government; Age 16
Non-Hispanic, American Indian Population Population with Masters Degree; Age 25 and up
Quarters and up
Population Employed in Federal Government; Age 16
Non-Hispanic, Asian Population Average Household Size Population with Professional Degree; Age 25 and up
and up
Non-Hispanic, Native Hawaiian or Pacific Islander
Average Household Size – Non-Family Household Population with Doctorate Degree; Age 25 and up Population Self-Employed; Age 16 and up
Population
Non-Hispanic, Other Population Average Household Size – Family Household Families at Poverty Level Population Unpaid Family Work; Age 16 and up
Population Speaking only English at Home; Age 5 Population Employed Blue Collar Work; Age 16 and
Hispanic, White Population Families at Poverty Level with Children
and Older up
Population Speaking Spanish at Home; Age 5 and Population Employed White Collar Work; Age 16 and
Hispanic, Black Population Families above Poverty Level
Older up
Population Employed Service and Farm Work; Age 16
Hispanic, American Indian Population Housing Units Owned by Occupant Families above Poverty Level with Children
and up
Population in Labor Force Employed by Armed
Hispanic, Asian Population Housing Units Rented by Occupant Population Male
Forces; Age 16 and up
Average Length of Residence Population Female
17

18. Our Findings: Profiling the User Types
Unemploy % with
Median tired of being surveyed!
“I’m Summary of other socio-economic
User Type -mentwasting our
Bachelor’s
Income
The government is variables
Rate
time/money!” Degree
Quitters $63,000 7.1% 13.4% Government and private, not-for
(n=111) profit employment with large
“I’m doing the best I can, but you’re
asking a lot” household size conditions
Returners $58,000 8.5% 10.7% More transient, socioeconomically
(n=180) “I should do a good job at this, my disadvantageous conditions
opinions are helping”
Completers $61,000 8.1% 11% Less transient, socioeconomically
(n=79) advantageous conditions
18

19. Our Conclusions & Recommendations
19

20. Our Conclusions
• The tool gives us immediately visual, easy to interpret results
that clearly bring out patterns
– “Knew” the user types before we modeled it
– Very easy to explain to clients or share across professionals
• With visual mapping, we can:
– Easily see the entire user experience
– Determine if unique user types exist
– See if usability problems exist with certain questions and people
• Length interactions
• Placement issues
• Factual vs attitudinal questions
20

21. Our Conclusions
• Provides great alternative for complex statistical investigation
– May not give you anything useful or helpful
– Usually empirically driven, so results can change frequently
– Hard to determine what to ACTUALLY do!
• A simple and effective way to communicate and examine a
survey’s effectiveness to clients and other researchers
– Can overlay respondent behavior with survey design
– Natural extension of pilot-testing and cognitive testing
21

22. Our Conclusions
• Most concerns are with total non-response. But this suggests
specific item non-response patterns
– Allows us to pinpoint the characteristics of those items
– As well as the people behind that non-response
• This suggests that item-level non-response adjustments may
be necessary if variable is of key interest
– Weigh option against client interests
– Complexity can grow exponentially
22

23. Our Recommendations
1. Everyone knows the value of pre-testing a survey. This emphasizes it and
the need for “true” test conditions.
2. Avoid areas where respondents are forced to engage for long periods
3. The design of a survey are critical and should not be left just to the
statisticians! Example: paper surveys and interaction of questions and
page location
4. Different users may required different persuasions techniques
– Incentive levels – Customized invitations
– Survey instructions – Different layout
5. Remember a reassessment of your key variables is always a good idea
and can uncover significant issues (try this new tool!)
23