Three studies examined scale usage across different data collection modes. Study 1 found differences in extreme, midpoint, and acquiescence response styles between mobile and web surveys, with mobile sliders producing higher extreme responding than grids. Study 2 found response styles varied by age and country but not gender, and using more scale points reduced midpoint responding. Study 3 found on social media that women, older people, and Australians responded more positively than others.
3. Battle Of The Scales
Scales are a cornerstone of market research
49%
42%
4. Measurement Scales & Response Styles
A person’s tendency to systematically respond to
questionnaire items in a certain way regardless of content
Paulhus
1991
ARS
MRS
ERS
Substance meaning of
the question
Factors unrelated
to content
5. Why Response Styles Matter
Lack of measurement equivalence in data across
countries implies that there is no common basis to
compare data across countries: In such cases,
observed mean differences on relevant
constructs…might result from measurement
artifacts…rather than from true differences across
countries
Alain De Beuckelaer
et al., 2007
6. Three studies
to examine
scale usage
across three
modes of data
collection
Scope Of This Paper
Social
media
Mobile
surveys
Web
surveys
7. Scope Of This Paper
What is the effect of using 3 alternative data collection options
on response styles in mobile surveys?
Compare differences in responses to 5-point Likert scales
across 3 data collection options:
Standard grid questions
on mobile web browser
Slider scales on mobile
web browser
Slider scales on
native app
8. Three studies
to examine
scale usage
across three
modes of data
collection
Scope Of This Paper
Social
media
Mobile
surveys
Web
surveys
9. 10-point
scales
Web
Surveys
Scope Of This Paper
4-point
scales
5-point
unlabeled
scales
5-point
labeled
scales
7-point
scales
10. Three studies
to examine
scale usage
across three
modes of data
collection
Scope Of This Paper
Social
media
Mobile
surveys
Web
surveys
12. Overall Research Plan
For Web & Mobile Surveys (Studies 1 & 2)
Globally
relevant topics
Include behavioral
statements that
should correlate to
attitudinal questions
7-minute
attitudinal survey
Mix of positive &
negative wording
Scales with published
measures of reliability
Fielded in multiple
countries to analyze
cross-cultural
differences
13. Overall Research Plan
For Social Media (Study 3)
Collect verbatim responses
from thousands of sites
Data collected only in
English-speaking countries
Score responses into
a 4-point scale
14. Overall Research Plan
For Web & Mobile Surveys (Studies 1 & 2)
4-point
scales
5-point
unlabeled
scales
10-point
scales
5-point
labeled
scales
7-point
scales
Used only 5-point Likert
scales for all constructs
Standard grid questions on
mobile web browser
Slider scales on mobile web
browser
Slider scales on native app
Used only 5-point Likert
scales for all constructs
Fielded in 10 countries
15. Overall Research Plan
Research Plan
Globally
relevant topics
Include behavioral
statements that
should correlate to
attitudinal questions
7-minute
attitudinal survey
Mix of positive &
negative wording
Scales with published
measures of reliability
Gender
Country Age
Simultaneously field
across 10 countries
VOP
Research Now’s
proprietary Valued
Opinions Panel
Simultaneously compare the effect of multiple response
options (4-, 5-, 7- & 10-point scales)
Using a large, census-balanced, multi-country sample
16. Battle of the Scales
Analysis Plan for Studies 1 & 2
1 1 1
Extreme
Response
Style Index
(ERSI)
Acquiescence
Response
Style Index
(ARSI)
Medium
Response
Style Index
(MRSI)
Indices were
averaged for
respondents
Value of each
index ranges
from a
minimum of 0
to a maximum
of 1.0.
17. Battle of the Scales
Research Questions: Study 1
RQ1 RQ2
Grid
Slider
Native APP
Questions?
18. Results 1: Study 1
Differences in Extreme Response Style Across Modes
Extreme response style
(ERS) index
Health
environment
sensitivity
Personal
health
responsibility
Attitude
toward
helping others
Material
values
scale
70
60
40
30
20
0
Attitude towards
advertising in
general (AAG)
Online
piracy
concern
Lie
tolerance
scale
50
Online web labeled Online web unlabeled Mobile native app
Mobile web grid Mobile web slider
19. Results 2: Study 1
Midpoint Response Style
(MRS) Index
Health
environment
sensitivity
Personal
health
responsibility
Attitude
toward
helping others
Material
values
scale
50
30
20
10
0
Differences in Midpoint Response Style Across Modes
Attitude towards
advertising in
general (AAG)
Online
piracy
concern
Lie
tolerance
scale
40
Online web labeled Online web unlabeled Mobile native app
Mobile web grid Mobile web slider
20. Results 3: Study 1
Acquiescence Response
Style (ARS) Index
Health
environment
sensitivity
Personal
health
responsibility
Online web labeled Online web unlabeled Mobile native app
Attitude
toward
helping others
Mobile web grid Mobile web slider
Material
values
scale
80
60
40
20
0
Differences in Acquiescence Response Style Across Modes
Attitude towards
advertising in
general (AAG)
Online
piracy
concern
Lie
tolerance
scale
21. Research Questions & Hypotheses: Study 2
Differences in Response Styles Across Countries, Genders & Ages
Acquiescent Midpoint Extreme
H1: There will be no
differences between
male & female
respondents with
respect to
ERS & MRS
H2: ERS & MRS will
differ across age
categories, but the
relationship between
age & ERS or MRS
will not be monotonic
H3: There will be
significant differences
across countries with
respect to ERS, MRS
& ARS
22. Results 1: Study 2
Impact of Gender on Response Styles
Hypothesis 1: There will be no meaningful differences in ERS, MRS or ARS indices between
male & female respondents
ERSI ARSI MRSI
.43 .32 .28
.43 .33 .28
23. Results 2: Study 2
Impact of Age on Response Styles
MRS (Medium) ARS (Acquiescent) ERS (Extreme)
15-17 18-24 25-34 35-44
50
45
30
25
20
0
45-54 55-64 65-74
35
Hypothesis 2: ERS & MRS will differ across age categories,
but the relationship between age & ERS or MRS will not be monotonic
40
75-84 85+
24. Results 3: Study 2
Differences in Response Styles Across Countries
H3: There will be significant differences across countries with respect to ERS, MRS & ARS
60
50
40
30
20
10
0
ERS (Extreme) ARS (Acquiescent)
25. Results 4: Study 2
Impact of Response Scales on Response Styles
RQ3: Does the use of 4-point, 5-point, 7-point & 10-point scales make any difference to
4 Point 5 Point Labelled
50
45
30
25
20
0
5 Point Unlabelled 7 Point
35
the level of extreme, acquiescent or midpoint responding?
40
10 Point
MRS (Medium) ARS (Acquiescent) ERS (Extreme)
26. Battle Of The Scales
Scales in Social Media
Awesome
Wicked
Bomb
Good
Nice
Cool
Huh
Dunno
Whatevs
Crap
Yuck
Dumb
Hideous
Disgusting
Abhor
Scales are a cornerstone of market research
27. Research Questions: Study 3
Impact of Gender, Age & Region on Social Media Scaling
RQ4: How do gender
differences affect scaling in
social media? For example,
do men/women make word
choices that are more/less
extreme in sentiment
resulting in more/less
extreme responding?
RQ5: How does age
of a social media
participant affect
the scaling of social
media sentiment?
RQ6: Are there
systematic differences
across geographic
regions or
nationalities in terms
of scaling social
media sentiment?
28. Results 1: Study 3
Impact of Gender on Response Styles
RQ6: How do gender differences affect scaling in social media? For example, do men (women) make
word choices that are more (less) extreme in sentiment resulting in more (less) extreme responding?
5.0
=
Strong/Negative Moderate/Negative Moderate/Positive Strong/Positive
34.1 51.0 9.9
3.8 29.2 53.7 13.2
-1.1 -4.9 2.7 3.3
%
29. Results 2: Study 3
Impact of Age on Response Styles
RQ7: How does age of a social media participant affect the scaling of social media sentiment?
1930
Decade of Birth
Sample Size
Strong/Negative
Moderate/Negative
Moderate/Positive
Strong/Positive
1940 1950 1960 1970 1980 1990
488 2676 4337 6492 8307 21644 20156
2.7% 1.1% 1.2% 1.4% 1.1% 1.5% 1.7%
34.4% 33.0% 30.0% 31.6% 33.0% 33.6% 35.1%
58.0% 58.7% 61.5% 60.0% 59.9% 59.6% 58.8%
4.9% 7.1% 7.3% 7.0% 6.0% 5.3% 4.4%
30. Results 3: Study 3
Impact of Country on Response Styles
RQ7: Are there differences in the scaling of social media sentiment across different
(English-speaking) countries?
AUSTRALLIeAss CnAeNgAaDtivAity aIRmEoLnAgN DAustralUiaKns US
Sample Size 29,372 54,505 155,068 962,270
Strong/Negative
Moderate/Negative
More extreme responding among United Kingdom authors
Moderate/Positive
Strong/Positive
9,323
More negativity among Americans
6.6% 7.7% 7.9% 9.5% 9.5%
29.5% 31.0% 29.6% 28.3% 30.7%
50.0% 47.1% 47.2% 45.0% 45.9%
13.9% 14.1% 15.3% 17.1% 13.9%
31. Mobile
Surveys
There are significant differences in ERS, ARS
and MRS between mobile and computer
responses, but differences are not consistent.
Not enough evidence to conclude if mobile is
better/worse than computer-based data
collection.
Compared to mobile grids, mobile sliders
seem to produce slightly higher levels of ERS.
Compared to mobile grids, mobile sliders
seem to produce lower levels of MRS.
There are no significant differences between
mobile sliders and grids in ARS.
Conclusions
32. Web
Surveys
ERS, MRS and ARS do not differ by gender,
but do differ as age increases.
Response styles vary from one country to
another.
Varying the number of response options does
help to reduce MRS.
Scale labeling did not impact scale reliability,
but did impact ERS, MRS and ARS.
Conclusions
33. Social
Media
Women are more positive.
Older people are more positive.
Australians are more positive.
Conclusions
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Lighten up bullet points – way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
Although there are significant differences in ERSI between the different measurement modes, the differences are not systematically attributable to a single mode of measurement. The mobile native app produced the highest levels of extreme responding for 4 of the scales but the ERSI was only significantly and meaningfully higher for 2 of the scales. The only observable systematic pattern is that the labeled version of the online web survey generally produced lower extreme responding than the unlabeled version.
The results of the multi-mode comparison for midpoint responding style (MRS) are presented in this graph. There is at least one systematic trend evident here, namely, that the mobile web slider consistently produces the lowest MRSI for all of the scales – although the difference is statistically significant only for 3 out of 6 scales. This trend is presented visually in Exhibit D.
When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
Convert bullet points into something more graphic (i.e. a flow chart illustrating: 1) survey requirements, 2) plan, 3) objective)
When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
See notes for slide 8
See notes for slide 8
See notes for slide 8
Include a more fitting image than what is provided to the right.
Convert bullet points into something more graphic (i.e. a flow chart illustrating: 1) survey requirements, 2) plan, 3) objective)
When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).