As the market research industry is gradually outsourcing more and more of its online data collection to specialized online panel providers and with research budgets still shifting towards online data collection, the topic of managing the quality of online access panels has become extremely important. With the statement “Garbage in, garbage out” more than ever relevant, we are witnessing a necessary evolution towards defining relevant norms for the industry. As in any debate there are supporters and opponents who each make a case for their own right, while the simple truth is we do not know (yet). One of the key issues is whether duplication, i.e. respondents belonging to multiple panels, really makes a difference in terms of the quality of the research data. Another issue is whether online versus offline recruitment of access panel members leads to quality differences. In order to shed a better light on both of these issues, we have set up two different studies in cooperation with the XL Online Panels online research panel.

1. DUPLICATION & MULTI-SOURCE ONLINE PANEL RECRUITMENT Real quality differences or idle rumours? Kristof De Wulf Sam Berteloot

3. Some major concerns ...

4. Some major concerns ...

5. Some major concerns ... “ 1% of panel members accounts for 34% of questionnaires completed” (CASRO 2006) “ It's been difficult to duplicate the results from one panel "so that people in the business can have confidence that the results they're getting are valid and predictable” ( ClickZ Network 2007) “ The market research industry faces a crisis of ‘over-fishing’ and poor respondent quality” that demands solutions to “improve the quality of online samples and associated responses” (comScore 2004) “ In the past year, marketers and researchers alike have focused unprecedented attention on the accuracy of research – specifically, data from online panels, which can produce wildly varying results for the same questions within and across suppliers ” (American Marketing Association 2007) “ We need to stop this blind trust at the client side. And the industry needs to help clients understand what quality means, and why data quality needs to be a key differentiator ” (Research World April 2007)

10. Who are people being subscribed to multiple panels? People subscribed to multiple panels do not differ from people subscribed to one panel only in terms of education, professional situation, and monthly net family income. Significant difference between groups (95% confidence interval) Professional situation Monthly net family income Highest education

11. Who are people being subscribed to multiple panels? People subscribed to multiple panels are more internet savvy : they have been using the medium longer and use it more often and intensively. Significant difference between groups (95% confidence interval) Daily usage of internet (hours) Number of days using internet Usage history internet

13. Effects of multiple membership No difference in response status : while the overall click rate is higher among subscribers to multiple panels, drop-out rates are higher, resulting in a comparable participation rate. Significant difference between groups (95% confidence interval)

15. Effects of multiple membership Response quality is slightly better among subscribers to multiple panels : they indicate to fill out surveys more truthfully, to be better concentrated when answering questions, and to try harder while filling out surveys. Significant difference between groups (95% confidence interval) Self-reported quality of responses (5-pt scale)

16. Effects of multiple membership Response quality is better among subscribers to multiple panels : item response consistency between surveys is higher. Item response consistency Significant difference between groups (95% confidence interval)

18. Effects of multiple membership Panel member feedback is more positive among subscribers to multiple panels : overall satisfaction, intention to participate, positive word-of-mouth and recommendation to others are higher. Significant difference between groups (95% confidence interval) Panel member satisfaction & intentions

19. Effects of multiple membership Intrinsic motivations to learn new things are higher among subscribers to multiple panels . Extrinsic motivations (receiving rewards) are less important and not different between both groups. Significant difference between groups (95% confidence interval) Reasons to subscribe

22. Effects of recruitment Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept . There are no significant differences in fill-out time between modes of recruitment. Significant difference between groups (95% confidence interval) Average across 5 data collection waves

23. Effects of recruitment Participation rate is highest among people recruited via e-mail and lowest among people recruited via intercept . E-mail Online ad Telephone Pop-up survey Intercept Participation rate per data collection wave Significant difference between groups (95% confidence interval)

25. Effects of recruitment Response styles are relatively more present among people recruited via online ads . Non-response to open ended questions is larger among people being recruited via e-mail . Significant difference between groups (95% confidence interval) Response style & open answer detail based on wave 3

26. Effects of recruitment Item response consistency is comparable between modes of recruitment. Significant difference between groups (95% confidence interval) Item response consistency based on wave 3

27. Effects of recruitment Self-reported response quality is comparable between modes of recruitment. Significant difference between groups (95% confidence interval) Self-reported quality of responses (5-pt scale) based on wave 5

29. Effects of recruitment People recruited via online ads are most driven by intrinsic motivations : providing their opinion and liking to fill out surveys. People recruited via offline methods are most driven by ‘ helping others ’ (researchers and charity). Significant difference between groups (95% confidence interval) Reasons to subscribe based on wave 5 (5-pt scale)

30. Effects of recruitment Panel member satisfaction among people recruited via online ads is significantly higher . Significant difference between groups (95% confidence interval) Project satisfaction based on waves 4 and 5 (5-pt scale) Panel member satisfaction based on wave 5 (5-pt scale)

34. THANK YOU More information? Kristof De Wulf: [email_address] Sam Berteloot: [email_address]