Perhaps one day we will figure out how to ask perfect survey questions. In the meantime, survey analyses are biased by random and correlated measurement errors, and evaluating the extent of such errors is therefore essential, both to remove the bias and to improve our question design. When there is no gold standard, these errors are often estimated using multitrait-multimethod (MTMM) experiments or longitudinal data by applying linear or ordinal factor models, which assume that (latent) measurement is linear and that the only type of method bias is one that pushes the answers monotonely in a particular direction—that of acquiescence, for example. However, not all measurement is linear and not all method bias is monotone. Extreme response tendencies, for example, are nonmonotone, as are primacy and recency effects, which act on just one category. Just as the monotone kind, such method effects will also lead to spurious dependencies among different survey questions, distorting their true relationships. Diagnosing, preventing, or correcting for such distortions therefore calls for a model that can account for them. For this purpose I will discuss the latent class MTMM model (Oberski 2011). In it, a latent loglinear modeling approach is combined with the MTMM design to yield a model that provides detailed information about the measurement quality of survey questions while also dealing with nonmonotone method biases. I will discuss the method's assumptions and demonstrate it on a few often-used survey questions. Standard software for latent class analysis can be used to estimate this model, so that evaluating the extent of nonlinear random and correlated measurement errors is now a reasonably user-friendly experience for survey researchers.

1. Multidirectional errors Latent class MTMM Conclusions
Multidirectional survey measurement errors:
the latent class MTMM model
Daniel Oberski / doberski@uvt.nl
Department of methodology & statistics
AAPOR 2015
Latent class MTMM Daniel Oberski / doberski@uvt.nl

2. Multidirectional errors Latent class MTMM Conclusions
..1 Multidirectional survey measurement errors
..2 Latent class multitrait-multimethod model
..3 Conclusions
Latent class MTMM Daniel Oberski / doberski@uvt.nl

3. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
Two respondents who:
• Went to the doctor the same number of
times
• Have the same opinion about the role of
women in society
give different answers to these questions.
This will bias estimates of relationships
between the variables.
Latent class MTMM Daniel Oberski / doberski@uvt.nl

4. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
• Latent variables can never be recovered
Latent class MTMM Daniel Oberski / doberski@uvt.nl

5. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
• Latent variables can never be recovered
• But latent variable models can recover the
amount of influence they exert
Latent class MTMM Daniel Oberski / doberski@uvt.nl

6. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
• Latent variables can never be recovered
• But latent variable models can recover the
amount of influence they exert
• This is useful to remove bias
Latent class MTMM Daniel Oberski / doberski@uvt.nl

7. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
• Latent variables can never be recovered
• But latent variable models can recover the
amount of influence they exert
• This is useful to remove bias
Latent class MTMM Daniel Oberski / doberski@uvt.nl

8. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
• Latent variables can never be recovered
• But latent variable models can recover the
amount of influence they exert
• This is useful to remove bias
• Multitrait-multimethod
(MTMM)
approach to estimating this influence
(Andrews 1984; Saris & Andrews 1991; Saris &
Gallhofer 2007);
Latent class MTMM Daniel Oberski / doberski@uvt.nl

9. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
• Latent variables can never be recovered
• But latent variable models can recover the
amount of influence they exert
• This is useful to remove bias
• Multitrait-multimethod
(MTMM)
approach to estimating this influence
(Andrews 1984; Saris & Andrews 1991; Saris &
Gallhofer 2007);
• Quasi-simplex approach (Wiley & Wiley 1970;
Alwin 2007).
Latent class MTMM Daniel Oberski / doberski@uvt.nl

10. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
• Latent variables can never be recovered
• But latent variable models can recover the
amount of influence they exert
• This is useful to remove bias
• Multitrait-multimethod
(MTMM)
approach to estimating this influence
(Andrews 1984; Saris & Andrews 1991; Saris &
Gallhofer 2007);
• Quasi-simplex approach (Wiley & Wiley 1970;
Alwin 2007).
Latent class MTMM Daniel Oberski / doberski@uvt.nl

11. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors
• Latent variables can never be recovered
• But latent variable models can recover the
amount of influence they exert
• This is useful to remove bias
• Multitrait-multimethod
(MTMM)
approach to estimating this influence
(Andrews 1984; Saris & Andrews 1991; Saris &
Gallhofer 2007);
• Quasi-simplex approach (Wiley & Wiley 1970;
Alwin 2007).
↑ Linear
models
Latent class MTMM Daniel Oberski / doberski@uvt.nl

12. Multidirectional errors Latent class MTMM Conclusions
Stochastic errors in the literature
• Random mistakes
• Acquiescence
• Answering in the socially desirable direction
• Tending to choose the first/last of several categories
• Extreme response (outer categories)
• Avoiding some particular category for whatever reason
• Preferring the midpoint
• Heaping (rounding)
• ...
Problem: Not all of these are linear
Stochastic errors are multidirectional
Latent class MTMM Daniel Oberski / doberski@uvt.nl

13. Multidirectional errors Latent class MTMM Conclusions
Example of the effect of a nonlinear error on estimate of
relationships
True relationship
1 2 3 4
1 136 112 91 75
2 112 75 50 34
3 91 50 28 15
4 75 34 15 7
Polychoric correlation = -0.25
Relationship with ERS
1 2 3 4
1 374 41 34 205
2 41 4 3 12
3 34 3 1 6
4 205 12 6 19
Polychoric correlation = -0.43
Latent class MTMM Daniel Oberski / doberski@uvt.nl

14. Multidirectional errors Latent class MTMM Conclusions
Problem: Stochastic errors can strongly bias relationship
estimates;
Problem: Linear latent variable models assume errors are all
one-way, so bias is not appropriately removed.
Solution: Latent class models to allow for multidirectional errors.
Here MTMM but could also be quasi-simplex
Latent class MTMM Daniel Oberski / doberski@uvt.nl

15. Multidirectional errors Latent class MTMM Conclusions
The latent class MTMM model
Oberski, Hagenaars & Saris, to appear in Psychological Methods.
Latent class MTMM Daniel Oberski / doberski@uvt.nl

16. Multidirectional errors Latent class MTMM Conclusions
M1 M2 M3
T1 T2 T3
y11 y21 y31 y12 y22 y32 y13 y23 y33
• Latent variables are discrete (categorical)
• Observed may be continuous or discrete
• Relationships (can be) nonparametric
Latent class MTMM Daniel Oberski / doberski@uvt.nl

17. Multidirectional errors Latent class MTMM Conclusions
Experimental design: split-ballot MTMM
Method 1 Method 2 Method 3
Random group 1 . .
Random group 2 . .
Latent class MTMM Daniel Oberski / doberski@uvt.nl

18. Multidirectional errors Latent class MTMM Conclusions
Opinion about the role of women experiment: Main
questionnaire (first method)
Latent class MTMM Daniel Oberski / doberski@uvt.nl

19. Multidirectional errors Latent class MTMM Conclusions
Opinion about the role of women experiment:
Supplementary group 1 (second method)
Latent class MTMM Daniel Oberski / doberski@uvt.nl

20. Multidirectional errors Latent class MTMM Conclusions
Opinion about the role of women experiment:
Supplementary group 2 (third method)
Latent class MTMM Daniel Oberski / doberski@uvt.nl

21. Multidirectional errors Latent class MTMM Conclusions
Effect of trait on item distribution, Greece
Men more right, positive agree-disgree
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
Latent class MTMM Daniel Oberski / doberski@uvt.nl
1 2 3 4 5
Factor score: 0.00
Category
Proportion
0.00.20.40.60.81.0

22. Multidirectional errors Latent class MTMM Conclusions
Effect of trait on item distribution, Slovenia
Men more right, negative agree-disgree
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
Latent class MTMM Daniel Oberski / doberski@uvt.nl
1 2 3 4 5
Factor score: 0.00
Category
Proportion
0.00.20.40.60.81.0

23. Multidirectional errors Latent class MTMM Conclusions
Method factors have a nonlinear influence on the items
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
Cut down, method 1
Trait
Method
1
2
34
5
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
Cut down, method 2
Trait
Method
1
2 3
4
5
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.0
Cut down, method 3
Trait
Method
1
2
3
4
5
Latent class MTMM Daniel Oberski / doberski@uvt.nl

24. Multidirectional errors Latent class MTMM Conclusions
Prevalence of method behavior
Latent class MTMM Daniel Oberski / doberski@uvt.nl

25. Multidirectional errors Latent class MTMM Conclusions
Conclusion
Latent class MTMM Daniel Oberski / doberski@uvt.nl

26. Multidirectional errors Latent class MTMM Conclusions
Conclusion
• Stochastic errors are important when you are interested in
relationships;
• Need latent variable models to estimate their extent so their
effects can be removed statistically;
• In traditional MTMM and quasi-simplex models, these effects
are all one-way;
• Latent class models allow for multidirectional errors.
• Example: the latent
class
MTMM model
(Oberski et al., to appear - see http://daob.nl/publications)
Latent class MTMM Daniel Oberski / doberski@uvt.nl

27. Multidirectional errors Latent class MTMM Conclusions
Thank you for your attention!
doberski@uvt.nl
@DanielOberski
See http://daob.nl/publications for preprints
Supported by Veni grant number 451-14-017 from the Netherlands
Organization for Scientific Research (NWO).
Latent class MTMM Daniel Oberski / doberski@uvt.nl