Presented by Ray Poynter (NewMR & Potentiate) Access the recording of this presentation via NewMR.org/Play-Again Presentation Description Ray Poynter presents a 2021 State of the Art review of the issues surrounding the collection of better data. Ray outlines the key challenges, new initiatives, the impact of quality on decisions, and pointers to what is likely to happen in the near future.

1. What does collecting ‘better data’
mean, and how to achieve it?
14 October 2021
Ray Poynter

2. Sponsors
Communication

3. Agenda
• The rise of Evidence-based Decision Making
• The link between Better Data and ‘Good Enough’
• The link between Better Data and ‘Errors’
• Total Survey Error
• Mapping Problems to Solutions
• Mapping Solutions to Problems

4. “Without data, you're just
another person with an
opinion.”
W. Edwards Deming

5. Photo by Hugo Rocha on Unsplash
Photo by Nihal Demirci on Unsplash
Photo by NeONBRAND on Unsplash
Photo by Erik Mclean on Unsplash
Good Enough

6. Improving these does not make this
wagon better as a form of transport

7. Compound Errors
3 steps, A, B and C with the following errors
o A with 10% error, B with 20% error, C with 50% error
o Confidence = 90% * 80% * 50% = 36%
Reduce the error in A by half
o 95% * 80* 50% = 38%
Reduce the error in C by half
o 90% * 80% * 75% = 54%

8. Groves, R.; Fowler, F.; Couper, M.; Lepkowski, J.; Singer, E.;
Tourangeau, R. (2009). Survey Methodology (2nd Edition).
John Wiley & Sons
Let’s think about TSE in terms of
predicting the USA Presidential Elections
2016 and 2020
Typical model is based on
1) Who do people say they are going to
vote for
2) How likely they say they are to vote
3) Weighting by demographics and the
way they say they voted last time
Total Survey Error

9. Total Survey Error
Validity
Is asking people to say which way they
are going to vote a valid way of
predicting the result?
If I ask you to predict what you will eat
on Saturday, what is the chance that it
will be right?
Is weighting by previous election going
to work with an atypical campaign?

10. Total Survey Error
Measurement Error
Did people make a mistake when
entering their answers?
Is the scale capable of collecting
the data accurately enough?
Did the survey correctly display
on their device, in the right
language, and capture
everything it should?

11. Total Survey Error
Processing Error
Did we spot all the bogus or
flawed responses?
Was the ‘likely voter’ adjustment
correct?
Was the weighting correct?

12. Total Survey Error
Coverage Error
We want a sample frame that
reflects everybody who votes
Online surveys reflect people who
use the internet.
Panel surveys reflect people who
have signed up to panels.
Telephone surveys reflect people
with a phone who answer it.

13. Total Survey Error
Sampling Error
What is the risk that just by
bad luck we have a sample that
does not reflect the
population?

14. Total Survey Error
Nonresponse Error
What about people who
decline to take part? Busy
people? Sceptical people?
Evidently, many Trump
supporters decline to speak to
pollsters (and to other people
who ring/email them).

15. Total Survey Error
Adjustment Error
In 2016 the weighting did not take the
importance of a) not having a college
education and b) being a white
Christian as being important enough –
both are key drivers of being pro-Trump
In 2020 it looks as though one
weighting error was to assume
Hispanics were one group – e.g. ex-
Cubans seem to be more pro-Trump

16. Applying TSE
Mapping Problem to Solutions

17. Total Survey Error
Validity
Is asking people a direct question going
to work?
If not:
1. Derived answers (e.g. conjoint)
2. Projective qual
3. Observations

18. Total Survey Error
Measurement Error
Did people make a mistake
when entering their answers?
Consider
1. Avoid typing numbers, and
assuming people
understand percentages
2. Build redundancy or checks
in to the survey
3. Probe qual answers
4. Get examples, e.g. photos
or videos

19. Total Survey Error
Processing Error
Check for errors and bad
responses in the data
Recode the data to increase
robustness
Apply qualitative analysis methods

20. Total Survey Error
Coverage Error
Define the population
The market?
Customers?
Regular customers?
If you are using a panel – who are you
missing?
The over 70s
Nat Rep? (disability, ethnicity, etc)
If you are using online – who are you
missing
Consider multi-mode

21. Total Survey Error
Sampling Error
If we have a random
probability sample
100 people = +/- 10%
1000 people = +/-3%

22. Total Survey Error
Nonresponse Error
2 key groups
1) People who are asked take
part but who decline to
take part
2) People who start but do
not finish
This is where engagement
comes into play

23. Total Survey Error
Adjustment Error
Dealing with errors – don’t just report
what you have found
Weighting the data – especially to
match samples
Code non-numeric data (e.g. text and
images)
Transcribe qual to text (enhancing the
analysis)

24. Solutions Mapped to Problems
Qual – when the question can’t be asked in a way that can generate numbers
Gamification – reduce non-response, in some cases improve validity
Multi-mode – improve coverage and reduce non-response
Chatbots - reduce non-response, in some cases improve validity
Video and images – reduce measurement error, increase validity
Conjoint – increase validity
Observational data – increase validity, reduce measurement error

25. Q & A
Ray Poynter

26. Sponsors
Communication