At the Advertising Research Foundation’s (ARF) 2011 annual re:think convention, a key issues forum presentation was held entitled Mixing the Right Sample Ingredients. The presentation was given by Jackie Lorch, VP of Global Knowledge Management for Survey Sampling International. The presentation discussed which factors to blend and emphasized the importance of implementing multi-source testing.
How to Get Started in Social Media for Art League City
Mixing the Right Sample Ingredients
1.
2. Mixing the Right Sample
Ingredients
A New Source Recipe
Jackie Lorch
VP, Global Knowledge Management
Survey Sampling International
3. What is the Problem
We Are Trying to Solve?
1. A systematic problem with online access panels
4. What is the Problem
We Are Trying to Solve?
2. A scarce resource
5. Many Reasons People Are Not in Panels
Why have you not joined an Why do you no longer belong to an
online research panel?1 online research panel?2
I don’t want any more emails 7% Too many surveys 4%
I think they are scams 6% Not enough surveys 27%
It’s a waste of time 3% I never qualified for a survey 44%
Not enough time to take surveys 11% Poor respondent experience 4%
Never heard of an online research panel 20% Not enough time to take surveys 17%
Don’t want to commit to a panel 22% Didn’t want to commit to a panel anymore 8%
Never had any opportunities to join a panel 37% Other 8%
Other 6%
1. Asked of people online who have never joined a research panel
2. Asked of people online, on a panel previously, not currently
Multiple answers allowed
10. Why Not Use Socio-Demographics?
• Demographics may not be the most helpful
or relevant stabilizers
• Demographic quotas work if the stratification
chosen is relevant to questionnaire topic
• A product may be liked across all demographics
11. Which are the right quotas?
Sample 1 Sample 2
Balanced on: Balanced on:
Age Ethnicity
Gender
Region
12. From Sources to People
• Using thousands of sources
• Metric must be at “person” level
14. Factors for the Blend
Broad set of factors defining groups of people,
spanning multiple disciplines
Personality traits Need for cognition (Cacioppo et al)
Music preferences (Rentfrow) Neurographics measures
Cognitive ability Propensity factors e.g., to participate,
(Kahneman and Frederick) share, risk averse, attitude to privacy
Geographic / personality alignment Chronotypes, “lark” or “owl”
(Gosling and Rentfrow) (Roenneberg)
Social Values. Schwarz has international Disruption/orienting reflex measure,
benchmarks habituation disruption
15. Factors for the Blend
• Questions created around 162 variables
• Tested factor performance against questions on
technology, hobbies, interests, brand preference,
loyalty, ad awareness
• Variables tested iteratively on power to
“move the needle” on dependent variables
• 14 variables identified as explaining
more variance than socio-demographics
alone; from these created 8 clusters to control
16. Multi-Source Testing
• Broad set of factors defining groups of people,
spanning multiple disciplines
• Factors included such topics as attitudes to trust,
gender roles, online behavior, personality traits,
ideas and beliefs
• Results varied by source
17. Results Closer to “Truth”
On measures where benchmarks are available,
blend comes closer to it
Social Media
iPod/MP3 Smartphone (Facebook or
Ownership Ownership MySpace)
SSI SurveySpot panel 37 16 60
Benchmark 43 c.20-22 69
SSI Blend 43 20 68
Benchmark sources: Pew Research; comScore; Harris Interactive
23. Practical Steps When Blending
• Know your sources
• Use a consistent blend for the entire project
• Use benchmarks and calibrate
• Understand blending techniques being used
24. In Summary
• Multi-sourcing is the future, offering superior sample
• We can keep multi-sourced samples consistent by
pre-profiling participants
• Standard socio-demographics aren’t enough
• A concise list of variables can
be used for a broad range of
research subjects
• Participant experience is key